## -CHANGE REQUEST COVER SHEET

## Change Request Number: 06-25 

## 

## Date Received: 3/7/06

**Title:** GAO Findings Response and Standard FAA Program Milestones

# **Name:** David Woodson

**Routing Symbol:** ATO-A, Acquisition and Procurement Policy Branch

**Phone:** 267-7601

**Policy OR Guidance:** Guidance

**Section/Text Location Affected:** Multiple

**Summary of Change:** This change proposal includes the following:

Guidance

1.  Establishes guidance for evolutionary product development.

2.  Establishes guidance for the FAA's knowledge-based product
    development process for systems.

3.  Establishes guidance that identifies and defines standard program
    milestones for systems (sheet 1) and facilities (sheet 2).

4.  Adds to the introduction of the contracting module in the
    developmental hardware and/or software process flowcharts the
    requirement to include in any prime contract for system development
    those deliverables necessary for an investment program to satisfy
    decision criteria for transition from product integration to product
    demonstration and from product demonstration to full production.

5.  Modifies the development process flowcharts in FAST for
    developmental hardware and/or software to add the product
    demonstration decision.

6.  Modifies the implementation process flowcharts in FAST for
    developmental hardware and/or software to add the production
    decision.

**Reason for Change:** GAO Report 05-23, FAA's Acquisition Management
Has Improved, but Policies and Oversight Need Strengthening to Help
Ensure Results, made the following recommendations for improving the FAA
Acquisition Management System:

1.  Develop explicit written criteria for key decision points called for
    under best practices ^1^, including the capture of specific design
    and manufacturing knowledge

2.  Require corporate executive-level decisions at these key decision
    points (before an acquisition moves from integration to
    demonstration and again before it moves to production)

^1^ GAO Report 02-701, Best Practices, Capturing Design and
Manufacturing Knowledge Early Improves Acquisition Outcomes.

The FAA Acquisition Executive agreed to implement the decision points
and associated criteria, but stated that the decision-maker would be the
Vice President or Director of the implementing service organization, as
appropriate.

The change request also includes standard FAA program milestones
requested by the ASAG Secretariat.

**Development, Review, and/or Concurrence:** Policy and guidance was
developed by the GAO Response Team, established by the ASAG, with
representatives from ATO-A, ATO-F, ATO-P, ATO-E, ATO-T, ATO-W, and ATO
Quality Assurance.

**Target Audience:** AMS workforce

**Potential Links within FAST for the Change:** Multiple links between
the new policy and guidance.

#### Attachments: KBDM -- EPD Policy 02-06, KBDM -- Final Investment Decision, KBDM -- Final Investment Analysis, KBDM -- Solution Implementation, KBDM Guidance R6, Evolutionary Product Development Guidance, KBDM -- Contracting Activity Description, KBDM -- Product Demonstration Decision, KBDM - System Integration Activity Descriptions, Program Milestones R3.

### ASAG Briefing Conducted: 3/7/06

**ASAG Responsibilities:** Review, comment, and endorse.

**System Process Flowcharts / System Development Work Activities**
~(Revised\ 10/2005)~

 **Industry / Government Partnership**

System development work activities span the period from Investment
Analysis to the completion of the Design Qualification Test when the
government accepts the developmental product from the contractor and
authorizes field implementation. The most important event in this entire
process is selection of the developmental contractor and formation of
the government / industry partnership. Both the government and
contractor are stakeholders in the process. It is vitally important to
establish and maintain a positive working relationship. Neither can
succeed without the other, and each has its own challenges and
constraints.

The government is the customer and senior partner in the enterprise. The
government brings all team members to the table and integrates the
diverse activities associated with system development, implementation,
and lifecycle support. The focus of its agent, the Product Team, is
technical leadership, the government / industry partnership, and overall
program integration and management. Contractors and subcontractors, on
the other hand, focus on the very detailed engineering aspects of the
system and its interfaces.

Development contractors are faced with certain very real constraints
that the Product Team must allow for. Companies cannot afford to have
large teams of engineers sit idle waiting for a contract they may not
receive. A significant amount of time is needed after contract award to
form the developmental team. This effort is especially important during
the initial phases of a system development, and is seldom taken in to
account by the government when planning contract schedules. The Product
Team needs to recognize this condition and allow time for the contractor
to \'gear up\'.

Government planners must also allow adequate time for a thorough
analysis of requirements. In an effort to meet unrealistic early
milestone dates, contractors often short change this analysis and fail
to achieve a clear, mutual understanding of requirements with the
government. Studies have shown conclusively that omissions occurring in
the requirements analysis phase have a disproportionately greater effect
on program cost and schedule than errors occurring in later phases.

**Management and Assessment of the Engineering Development Process**

Programs rarely fail because of deficiencies in technology. Most
failures stem directly from inadequate management. Everyone involved in
program management must take a \"proactive\" role to ensure the
deployment of a successful system. Each government employee must think
beyond his or her individual piece of the system and take action to
identify and meet challenges before they enlarge into serious problems.
It is tempting to get down into the details and sometimes it is
necessary. But Product Team members must take a broader view, and should
perform the critical higher level tasks of managing, assessing,
integrating, and taking action to solve problems when they arise.

It is *critically* important for the Product Team to insist on
successful completion of early development activities, such as design of
the system architecture and completion of critical design, and not allow
the program to progress for the sake of \"maintaining schedule\".
Proceeding to such activities as source code development without a sound
overall system / software architecture usually results in a great deal
of fruitless work, and has resulted in complete failure of entire
acquisition programs.

The key to a successful system acquisition program is the integration of
all associated engineering disciplines into an efficient, coherent
process that achieves the development of equipment that meet user
requirements. Even when there is an \"integrating contractor\", the
government must work very hard at achieving harmonious orchestration.
The engineering management process depicted in this section governs and
controls product development. It addresses all key aspects of total
system performance (such as security, logistics support, and
configuration management), and provides a technical baseline for the
Integrated Program Plan. Its primary function is to provide a management
and engineering framework with which the Product Team can structure
developmental activity that will result in a product that meets cost,
schedule, and performance objectives over the entire product lifecycle.
The following are examples of contractor activities that Product Team
members will find themselves overseeing and guiding during this
intricate and detailed process:

-   Trade-off studies among requirements (operational, functional,
    performance), design alternatives (and their related manufacturing,
    testing, and support processes), schedule, and lifecycle cost.

-   Risk management to identify, evaluate, assess, and mitigate
    potential sources of risk associated with the contractor's design,
    manufacture, test, and support approach.

-   Configuration management of system products, processes, and related
    documentation. This effort includes identifying, documenting, and
    verifying the functional and physical characteristics of an item;
    recording its configuration; and controlling changes to the item and
    its documentation.

-   Integrated data management to capture and control the technical
    baseline (configuration documentation, technical data, and technical
    manuals); provide traceability among requirements, designs,
    decisions, rationale, and other related program planning, reporting,
    and support configuration procedures; and serve as a ready reference
    for the systems engineering effort.

-   The determination of metrics for use as measures of technical
    progress relative to what was planned in the areas of performance,
    risk mitigation, cost, and schedule. Performance metrics must be
    traceable to performance parameters in the Requirements Document and
    System Specification

-   The establishment of interface controls to ensure changes to all
    internal and external interface requirements are properly recorded
    and communicated to all affected configuration items.

-   The development of a structured review process that demonstrates and
    confirms completion of required accomplishments and their exit
    criteria, as defined in program planning.

**Assessing and Reporting Progress**

Rigorous internal program control systems are essential to effective and
accountable program management. They enable acquisition functions to be
performed efficiently and effectively by maximizing the utilization of
resources. The focus should be on results, not process. One key is to
integrate cost, schedule, and technical performance is the application
of Earned Value management. Another is to ensure that the individuals
responsible for day-to-day management of the acquisition contract feel
they have ownership of and are responsible for their elements of the
Work Breakdown Structure. When this approach is taken, the Product Team
is more likely to achieve full integration of cost, schedule, and
technical performance within the quantifiable terms of earned-value --
and not just rely on the completion of technical or schedule events or
milestones.

One of the most important outputs of Earned Value Management is
reliable, timely, and quantifiable data that enables the Product Team
Lead to accurately determine the Estimate at Complete (EAC).
Historically, a lack of understanding of information in Cost Performance
Reports and Cost/Schedule Status Reports or the processes and
methodology used to generate them have made it difficult for the Product
Lead to make critical program decisions. Knowing how the contractor
earns value for a particular element of work allows the Product Team to
manage their areas of responsibility with confidence.

**Providing Direction**

Sound direction from the government comes only from accurate assessments
of where the program is compared to where it is supposed to be. Periodic
program management reviews using Earned Value techniques provide a
mechanism for determining where you are. With this information, the
Product Lead can provide confident direction to each contractor and team
member involved in the development program.

+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Approve     |             |             |                           |
| Contractor  |             |             |                           |
| Management  |             |             |                           |
| Plans\      |             |             |                           |
| D-1**       |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Approved    | Service     |                           |
| develops    | contractor  | Team        |                           |
|             | management  |             |                           |
| Service     | plans       |             |                           |
| Team        |             |             |                           |
| specialists |             |             |                           |
| review and  |             |             |                           |
| comment     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descr    |             |             |                           |
| iption]{.un |             |             |                           |
| derline}**: |             |             |                           |
|             |             |             |                           |
| The         |             |             |                           |
| quickest,   |             |             |                           |
| cheapest    |             |             |                           |
| way to      |             |             |                           |
| build       |             |             |                           |
| quality     |             |             |                           |
| software is |             |             |                           |
| to avoid    |             |             |                           |
| mistakes    |             |             |                           |
| during      |             |             |                           |
| development |             |             |                           |
| and to      |             |             |                           |
| remove      |             |             |                           |
| errors      |             |             |                           |
| early in    |             |             |                           |
| design      |             |             |                           |
| before      |             |             |                           |
| beginning   |             |             |                           |
| the coding. |             |             |                           |
| The driving |             |             |                           |
| factor      |             |             |                           |
| behind      |             |             |                           |
| successful  |             |             |                           |
| softwar     |             |             |                           |
| e-intensive |             |             |                           |
| systems is  |             |             |                           |
| software    |             |             |                           |
| quality     |             |             |                           |
| rather than |             |             |                           |
| simply cost |             |             |                           |
| or          |             |             |                           |
| schedule.   |             |             |                           |
| Quality     |             |             |                           |
| software    |             |             |                           |
| development |             |             |                           |
| depends on  |             |             |                           |
| the         |             |             |                           |
| management  |             |             |                           |
| processes   |             |             |                           |
| and         |             |             |                           |
| procedures  |             |             |                           |
| used by the |             |             |                           |
| contractor. |             |             |                           |
| As the      |             |             |                           |
| first       |             |             |                           |
| crucial     |             |             |                           |
| activity    |             |             |                           |
| after       |             |             |                           |
| contract    |             |             |                           |
| award, the  |             |             |                           |
| Service     |             |             |                           |
| Team        |             |             |                           |
| reviews and |             |             |                           |
| comments on |             |             |                           |
| contractor  |             |             |                           |
| management  |             |             |                           |
| plans,      |             |             |                           |
| which       |             |             |                           |
| become part |             |             |                           |
| of the      |             |             |                           |
| c           |             |             |                           |
| ontractor's |             |             |                           |
| management  |             |             |                           |
| baseline    |             |             |                           |
| after       |             |             |                           |
| approval.   |             |             |                           |
| Changes to  |             |             |                           |
| these       |             |             |                           |
| documents   |             |             |                           |
| are         |             |             |                           |
| coordinated |             |             |                           |
| with the    |             |             |                           |
| Service     |             |             |                           |
| Team before |             |             |                           |
| being       |             |             |                           |
| implemented |             |             |                           |
| by the      |             |             |                           |
| contractor. |             |             |                           |
| Examples of |             |             |                           |
| contractor  |             |             |                           |
| planning    |             |             |                           |
| documents   |             |             |                           |
| include:    |             |             |                           |
| Project     |             |             |                           |
| Management  |             |             |                           |
| Plan,       |             |             |                           |
| System      |             |             |                           |
| Engineering |             |             |                           |
| Management  |             |             |                           |
| Plan,       |             |             |                           |
| Software    |             |             |                           |
| Development |             |             |                           |
| Plan        |             |             |                           |
| (including  |             |             |                           |
| quality     |             |             |                           |
| assurance   |             |             |                           |
| processes), |             |             |                           |
| Co          |             |             |                           |
| nfiguration |             |             |                           |
| Management  |             |             |                           |
| Plan, and   |             |             |                           |
| Su          |             |             |                           |
| bcontractor |             |             |                           |
| Management  |             |             |                           |
| Plan.       |             |             |                           |
|             |             |             |                           |
| Point your  |             |             |                           |
| browser at  |             |             |                           |
| <           |             |             |                           |
| http://sepo |             |             |                           |
| .nosc.mil/> |             |             |                           |
| and select  |             |             |                           |
| the \"SQA\" |             |             |                           |
| menu item   |             |             |                           |
| for some    |             |             |                           |
| good tools  |             |             |                           |
| on SW       |             |             |                           |
| Quality     |             |             |                           |
| Assurance   |             |             |                           |
| plans,      |             |             |                           |
| processes,  |             |             |                           |
| and         |             |             |                           |
| procedures. |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee     |             |             |                           |
| System      |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| Analysis\   |             |             |                           |
| D-2**       |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Revised     | Service     |                           |
| performs    | System      | Team lead   |                           |
|             | Sp          | engineer    |                           |
| Service     | ecification |             |                           |
| Team        |             |             |                           |
| specialists | Preliminary |             |                           |
| oversee     | System,     |             |                           |
|             | Segment     |             |                           |
|             | Sp          |             |                           |
|             | ecification |             |                           |
|             |             |             |                           |
|             | Interface   |             |                           |
|             | Control     |             |                           |
|             | Documents   |             |                           |
|             | (Segment to |             |                           |
|             | Segment)    |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| oversees    |             |             |                           |
| and         |             |             |                           |
| generally   |             |             |                           |
| supports    |             |             |                           |
| contractor  |             |             |                           |
| assessment  |             |             |                           |
| of system   |             |             |                           |
| sp          |             |             |                           |
| ecification |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| and         |             |             |                           |
| ide         |             |             |                           |
| ntification |             |             |                           |
| of          |             |             |                           |
| technical   |             |             |                           |
| risks       |             |             |                           |
| associated  |             |             |                           |
| with        |             |             |                           |
| satisfying  |             |             |                           |
| those       |             |             |                           |
| re          |             |             |                           |
| quirements. |             |             |                           |
| This        |             |             |                           |
| activity    |             |             |                           |
| determines  |             |             |                           |
| whether     |             |             |                           |
| user        |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| can         |             |             |                           |
| ultimately  |             |             |                           |
| be          |             |             |                           |
| achieved.   |             |             |                           |
| As much as  |             |             |                           |
| half of all |             |             |                           |
| software    |             |             |                           |
| errors      |             |             |                           |
| emanate     |             |             |                           |
| from errors |             |             |                           |
| in          |             |             |                           |
| re          |             |             |                           |
| quirements, |             |             |                           |
| and the     |             |             |                           |
| main reason |             |             |                           |
| is lack of  |             |             |                           |
| co          |             |             |                           |
| mmunication |             |             |                           |
| with the    |             |             |                           |
| user.       |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| may be      |             |             |                           |
| fleshed out |             |             |                           |
| in greater  |             |             |                           |
| detail      |             |             |                           |
| during this |             |             |                           |
| analysis    |             |             |                           |
| through     |             |             |                           |
| user        |             |             |                           |
| interviews, |             |             |                           |
| s           |             |             |                           |
| imulations, |             |             |                           |
| p           |             |             |                           |
| rototyping, |             |             |                           |
| demo        |             |             |                           |
| nstrations, |             |             |                           |
| or any      |             |             |                           |
| activity    |             |             |                           |
| that gives  |             |             |                           |
| confidence  |             |             |                           |
| the         |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| are well    |             |             |                           |
| understood  |             |             |                           |
| and well    |             |             |                           |
| defined.    |             |             |                           |
| This        |             |             |                           |
| activity    |             |             |                           |
| provides    |             |             |                           |
| the basis   |             |             |                           |
| for         |             |             |                           |
| defining    |             |             |                           |
| the entire  |             |             |                           |
| system, its |             |             |                           |
| components, |             |             |                           |
| and their   |             |             |                           |
| interrel    |             |             |                           |
| ationships, |             |             |                           |
| as well as  |             |             |                           |
| for         |             |             |                           |
| providing a |             |             |                           |
| foundation  |             |             |                           |
| for         |             |             |                           |
| measuring   |             |             |                           |
| the extent  |             |             |                           |
| to which    |             |             |                           |
| program     |             |             |                           |
| re          |             |             |                           |
| quirements, |             |             |                           |
| objectives, |             |             |                           |
| and         |             |             |                           |
| priorities  |             |             |                           |
| are         |             |             |                           |
| satisfied   |             |             |                           |
| by system   |             |             |                           |
| design.     |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|   Assessing |             |             |                           |
|     /       |             |             |                           |
|             |             |             |                           |
|    refining |             |             |                           |
|     system  |             |             |                           |
|     re      |             |             |                           |
| quirements; |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
| Identifying |             |             |                           |
|             |             |             |                           |
|    segments |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|  allocating |             |             |                           |
|     r       |             |             |                           |
| equirements |             |             |                           |
|     to      |             |             |                           |
|             |             |             |                           |
|    segments |             |             |                           |
|     (HWCIs, |             |             |                           |
|     CSCIs); |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Defining |             |             |                           |
|     design  |             |             |                           |
|     c       |             |             |                           |
| onstraints; |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Defining |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|  supporting |             |             |                           |
|             |             |             |                           |
|    external |             |             |                           |
|             |             |             |                           |
| interfaces. |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Conduct     |             |             |                           |
| System      |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| Review      |             |             |                           |
| (SRR)\      |             |             |                           |
| D-3**       |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Refined     | Service     |                           |
| conducts    | System      | Team        |                           |
| review      | Sp          |             |                           |
|             | ecification |             |                           |
| Service     |             |             |                           |
| Team        | System,     |             |                           |
| specialists | Segment     |             |                           |
| assess,     | Sp          |             |                           |
| evaluate,   | ecification |             |                           |
| and comment |             |             |                           |
|             | Interface   |             |                           |
|             | Control     |             |                           |
|             | Documents   |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The System  |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| Review      |             |             |                           |
| (SRR) forms |             |             |                           |
| the basis   |             |             |                           |
| for         |             |             |                           |
| determining |             |             |                           |
| whether (1) |             |             |                           |
| system      |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| are         |             |             |                           |
| consistent, |             |             |                           |
| achievable, |             |             |                           |
| and         |             |             |                           |
| complete,   |             |             |                           |
| and (2) the |             |             |                           |
| government  |             |             |                           |
| and         |             |             |                           |
| contractor  |             |             |                           |
| have a      |             |             |                           |
| clear and   |             |             |                           |
| mutual      |             |             |                           |
| un          |             |             |                           |
| derstanding |             |             |                           |
| of them.    |             |             |                           |
| While the   |             |             |                           |
| contractor  |             |             |                           |
| conducts    |             |             |                           |
| this        |             |             |                           |
| review, the |             |             |                           |
| government  |             |             |                           |
| takes an    |             |             |                           |
| active part |             |             |                           |
| by clearly  |             |             |                           |
| defining    |             |             |                           |
| e           |             |             |                           |
| xpectations |             |             |                           |
| and         |             |             |                           |
| standards   |             |             |                           |
| before the  |             |             |                           |
| review and  |             |             |                           |
| by          |             |             |                           |
| pa          |             |             |                           |
| rticipating |             |             |                           |
| actively in |             |             |                           |
| the event.  |             |             |                           |
|             |             |             |                           |
| The review  |             |             |                           |
| typically   |             |             |                           |
| covers the  |             |             |                           |
| System      |             |             |                           |
| Spe         |             |             |                           |
| cification, |             |             |                           |
| System /    |             |             |                           |
| Segment     |             |             |                           |
| Spe         |             |             |                           |
| cification, |             |             |                           |
| and         |             |             |                           |
| Interface   |             |             |                           |
| Control     |             |             |                           |
| Documents.  |             |             |                           |
| The         |             |             |                           |
| products of |             |             |                           |
| functional  |             |             |                           |
| analysis,   |             |             |                           |
| syntheses,  |             |             |                           |
| and trade   |             |             |                           |
| studies may |             |             |                           |
| also be     |             |             |                           |
| reviewed.   |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee     |             |             |                           |
| Design of   |             |             |                           |
| System      |             |             |                           |
| Ar          |             |             |                           |
| chitecture\ |             |             |                           |
| D-4**       |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Revisions   | Responsible |                           |
| designs     | to System   | Service     |                           |
|             | Sp          | Team        |                           |
| Service     | ecification | specialists |                           |
| Team        |             |             |                           |
| specialists | Revisions   |             |                           |
| oversee     | to System / |             |                           |
|             | Segment     |             |                           |
|             | Sp          |             |                           |
|             | ecification |             |                           |
|             |             |             |                           |
|             | Preliminary |             |                           |
|             | Software    |             |                           |
|             | R           |             |                           |
|             | equirements |             |                           |
|             | Sp          |             |                           |
|             | ecification |             |                           |
|             |             |             |                           |
|             | Preliminary |             |                           |
|             | Interface   |             |                           |
|             | R           |             |                           |
|             | equirements |             |                           |
|             | Sp          |             |                           |
|             | ecification |             |                           |
|             |             |             |                           |
|             | Preliminary |             |                           |
|             | Systems /   |             |                           |
|             | Segment     |             |                           |
|             | Design      |             |                           |
|             | Document    |             |                           |
|             |             |             |                           |
|             | R           |             |                           |
|             | equirements |             |                           |
|             | T           |             |                           |
|             | raceability |             |                           |
|             | Matrix      |             |                           |
|             |             |             |                           |
|             | Update to   |             |                           |
|             | Software    |             |                           |
|             | Development |             |                           |
|             | Plan        |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| oversees    |             |             |                           |
| and         |             |             |                           |
| generally   |             |             |                           |
| supports    |             |             |                           |
| contractor  |             |             |                           |
| activity to |             |             |                           |
| define,     |             |             |                           |
| develop,    |             |             |                           |
| and         |             |             |                           |
| baseline    |             |             |                           |
| the system  |             |             |                           |
| ar          |             |             |                           |
| chitecture. |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| partitions  |             |             |                           |
| system      |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| to software |             |             |                           |
| and         |             |             |                           |
| hardware    |             |             |                           |
| co          |             |             |                           |
| nfiguration |             |             |                           |
| items and   |             |             |                           |
| evaluates   |             |             |                           |
| alternative |             |             |                           |
| design      |             |             |                           |
| approaches, |             |             |                           |
| leading to  |             |             |                           |
| preliminary |             |             |                           |
| system /    |             |             |                           |
| subsystem   |             |             |                           |
| designs and |             |             |                           |
| the         |             |             |                           |
| ide         |             |             |                           |
| ntification |             |             |                           |
| of system   |             |             |                           |
| components. |             |             |                           |
| Each        |             |             |                           |
| subsystem   |             |             |                           |
| design is   |             |             |                           |
| integrated  |             |             |                           |
| into the    |             |             |                           |
| total       |             |             |                           |
| system and  |             |             |                           |
| evaluated   |             |             |                           |
| against     |             |             |                           |
| program     |             |             |                           |
| constraints |             |             |                           |
| of time and |             |             |                           |
| cost. The   |             |             |                           |
| flow of     |             |             |                           |
| data and    |             |             |                           |
| the flow of |             |             |                           |
| control are |             |             |                           |
| also        |             |             |                           |
| determined. |             |             |                           |
| Trade       |             |             |                           |
| studies and |             |             |                           |
| system      |             |             |                           |
| modeling    |             |             |                           |
| may also be |             |             |                           |
| performed.  |             |             |                           |
|             |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| also        |             |             |                           |
| develops a  |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| T           |             |             |                           |
| raceability |             |             |                           |
| Matrix that |             |             |                           |
| identifies  |             |             |                           |
| which       |             |             |                           |
| co          |             |             |                           |
| nfiguration |             |             |                           |
| item will   |             |             |                           |
| achieve     |             |             |                           |
| each        |             |             |                           |
| requirement |             |             |                           |
| in the      |             |             |                           |
| System      |             |             |                           |
| Spe         |             |             |                           |
| cification. |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| are         |             |             |                           |
| allocated   |             |             |                           |
| in the      |             |             |                           |
| System      |             |             |                           |
| A           |             |             |                           |
| rchitecture |             |             |                           |
| Baseline    |             |             |                           |
| and         |             |             |                           |
| Allocated   |             |             |                           |
| Baseline.   |             |             |                           |
|             |             |             |                           |
| If an open  |             |             |                           |
| a           |             |             |                           |
| rchitecture |             |             |                           |
| has been    |             |             |                           |
| chosen,     |             |             |                           |
| point your  |             |             |                           |
| browser to  |             |             |                           |
| <ht         |             |             |                           |
| tp://www.it |             |             |                           |
| l.nist.gov/ |             |             |                           |
| div897/pubs |             |             |                           |
| /index.htm> |             |             |                           |
| and see     |             |             |                           |
| Special     |             |             |                           |
| Publication |             |             |                           |
| 500-187,    |             |             |                           |
| Application |             |             |                           |
| Portability |             |             |                           |
| Profile     |             |             |                           |
| (APP). The  |             |             |                           |
| U.S.        |             |             |                           |
| Go          |             |             |                           |
| vernment\'s |             |             |                           |
| Open System |             |             |                           |
| Environment |             |             |                           |
| Profile,    |             |             |                           |
| OSE/1,      |             |             |                           |
| Version     |             |             |                           |
| 1.0. is     |             |             |                           |
| based on an |             |             |                           |
| open system |             |             |                           |
| environment |             |             |                           |
| defined by  |             |             |                           |
| non-        |             |             |                           |
| proprietary |             |             |                           |
| spec        |             |             |                           |
| ifications. |             |             |                           |
| Components  |             |             |                           |
| may be      |             |             |                           |
| added or    |             |             |                           |
| deleted as  |             |             |                           |
| technology  |             |             |                           |
| changes and |             |             |                           |
| as Federal  |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| change.     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Conduct     |             |             |                           |
| System      |             |             |                           |
| Design      |             |             |                           |
| Review\     |             |             |                           |
| D-5**       |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Comments on | Appropriate |                           |
| conducts    | documents   | Service     |                           |
| SDR         |             | Team        |                           |
|             |             | specialist  |                           |
| Service     |             |             |                           |
| Team        |             |             |                           |
| specialists |             |             |                           |
| review      |             |             |                           |
| contractor  |             |             |                           |
| design      |             |             |                           |
| documents   |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| conducts    |             |             |                           |
| the System  |             |             |                           |
| Design      |             |             |                           |
| Review      |             |             |                           |
| (SDR) with  |             |             |                           |
| the Service |             |             |                           |
| Team        |             |             |                           |
| reviewing   |             |             |                           |
| and         |             |             |                           |
| evaluating  |             |             |                           |
| the status  |             |             |                           |
| and         |             |             |                           |
| maturity of |             |             |                           |
| system      |             |             |                           |
| design.     |             |             |                           |
| This        |             |             |                           |
| activity is |             |             |                           |
| intended to |             |             |                           |
| confirm     |             |             |                           |
| that (1)    |             |             |                           |
| system      |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| can be met, |             |             |                           |
| and (2) the |             |             |                           |
| contractor  |             |             |                           |
| has a sound |             |             |                           |
| top-level   |             |             |                           |
| design with |             |             |                           |
| system      |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| segmented   |             |             |                           |
| to          |             |             |                           |
| appropriate |             |             |                           |
| co          |             |             |                           |
| nfiguration |             |             |                           |
| items.      |             |             |                           |
|             |             |             |                           |
| The         |             |             |                           |
| following   |             |             |                           |
| should be   |             |             |                           |
| complete    |             |             |                           |
| prior to    |             |             |                           |
| the System  |             |             |                           |
| Design      |             |             |                           |
| Review:     |             |             |                           |
|             |             |             |                           |
| -   All     |             |             |                           |
|             |             |             |                           |
|  subsystem, |             |             |                           |
|             |             |             |                           |
| functional, |             |             |                           |
|     and     |             |             |                           |
|     interim |             |             |                           |
|     system  |             |             |                           |
|     reviews |             |             |                           |
|     with    |             |             |                           |
|             |             |             |                           |
|  unresolved |             |             |                           |
|     issues  |             |             |                           |
|             |             |             |                           |
| documented; |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
| Definitions |             |             |                           |
|     of      |             |             |                           |
|             |             |             |                           |
|  functional |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| performance |             |             |                           |
|     r       |             |             |                           |
| equirements |             |             |                           |
|     for     |             |             |                           |
|     system  |             |             |                           |
|     d       |             |             |                           |
| evelopment, |             |             |                           |
|     man     |             |             |                           |
| ufacturing, |             |             |                           |
|     ve      |             |             |                           |
| rification, |             |             |                           |
|             |             |             |                           |
| deployment, |             |             |                           |
|             |             |             |                           |
| operations, |             |             |                           |
|             |             |             |                           |
|    support, |             |             |                           |
|             |             |             |                           |
|   training, |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|   disposal; |             |             |                           |
|             |             |             |                           |
| -   System  |             |             |                           |
|     trade   |             |             |                           |
|             |             |             |                           |
|    studies, |             |             |                           |
|     as well |             |             |                           |
|     as      |             |             |                           |
|     trade   |             |             |                           |
|     studies |             |             |                           |
|             |             |             |                           |
|  supporting |             |             |                           |
|     the     |             |             |                           |
|             |             |             |                           |
| preparation |             |             |                           |
|     of      |             |             |                           |
|     draft   |             |             |                           |
|     Co      |             |             |                           |
| nfiguration |             |             |                           |
|     Item    |             |             |                           |
|             |             |             |                           |
| development |             |             |                           |
|     spec    |             |             |                           |
| ifications; |             |             |                           |
|             |             |             |                           |
| -   System  |             |             |                           |
|     a       |             |             |                           |
| rchitecture |             |             |                           |
|     to the  |             |             |                           |
|             |             |             |                           |
|   specified |             |             |                           |
|     level;  |             |             |                           |
|             |             |             |                           |
| -   Draft   |             |             |                           |
|     Co      |             |             |                           |
| nfiguration |             |             |                           |
|     Item    |             |             |                           |
|     a       |             |             |                           |
| rchitecture |             |             |                           |
|     to      |             |             |                           |
|             |             |             |                           |
|   specified |             |             |                           |
|     level;  |             |             |                           |
|             |             |             |                           |
| -   Draft   |             |             |                           |
|     Co      |             |             |                           |
| nfiguration |             |             |                           |
|     Item    |             |             |                           |
|             |             |             |                           |
|  (including |             |             |                           |
|     CSCI)   |             |             |                           |
|             |             |             |                           |
| development |             |             |                           |
|     spe     |             |             |                           |
| cifications |             |             |                           |
|     to the  |             |             |                           |
|     desired |             |             |                           |
|     level   |             |             |                           |
|     below   |             |             |                           |
|     the     |             |             |                           |
|     system  |             |             |                           |
|     spe     |             |             |                           |
| cification. |             |             |                           |
|             |             |             |                           |
| The         |             |             |                           |
| following   |             |             |                           |
| design      |             |             |                           |
| documents   |             |             |                           |
| are         |             |             |                           |
| typically   |             |             |                           |
| reviewed    |             |             |                           |
| during the  |             |             |                           |
| SDR:        |             |             |                           |
|             |             |             |                           |
| -   System  |             |             |                           |
|     /       |             |             |                           |
|     Segment |             |             |                           |
|     Spe     |             |             |                           |
| cification; |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
| Preliminary |             |             |                           |
|             |             |             |                           |
|    Software |             |             |                           |
|     R       |             |             |                           |
| equirements |             |             |                           |
|     Spe     |             |             |                           |
| cification; |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
| Preliminary |             |             |                           |
|             |             |             |                           |
|   Interface |             |             |                           |
|             |             |             |                           |
| Requirement |             |             |                           |
|     Spe     |             |             |                           |
| cification; |             |             |                           |
|             |             |             |                           |
| -   System  |             |             |                           |
|     /       |             |             |                           |
|     Segment |             |             |                           |
|     Design  |             |             |                           |
|             |             |             |                           |
|   Document; |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| -   R       |             |             |                           |
| equirements |             |             |                           |
|     T       |             |             |                           |
| raceability |             |             |                           |
|     Matrix. |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee     |             |             |                           |
| Software    |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| Analysis\   |             |             |                           |
| D-6**       |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Final       | Responsible |                           |
| develops    | System /    | Service     |                           |
|             | Segment     | Team        |                           |
| Service     | Sp          | specialist  |                           |
| Team        | ecification |             |                           |
| specialists |             |             |                           |
| and users   | Final       |             |                           |
| evaluate    | Software    |             |                           |
|             | R           |             |                           |
|             | equirements |             |                           |
|             | Sp          |             |                           |
|             | ecification |             |                           |
|             |             |             |                           |
|             | Initial     |             |                           |
|             | CSCI        |             |                           |
|             | Preliminary |             |                           |
|             | Design      |             |                           |
|             |             |             |                           |
|             | Final       |             |                           |
|             | Interface   |             |                           |
|             | R           |             |                           |
|             | equirements |             |                           |
|             | Sp          |             |                           |
|             | ecification |             |                           |
|             |             |             |                           |
|             | Updated     |             |                           |
|             | System /    |             |                           |
|             | Segment     |             |                           |
|             | Design      |             |                           |
|             | Document    |             |                           |
|             |             |             |                           |
|             | Updated     |             |                           |
|             | R           |             |                           |
|             | equirements |             |                           |
|             | T           |             |                           |
|             | raceability |             |                           |
|             | Matrix      |             |                           |
|             |             |             |                           |
|             | Data from   |             |                           |
|             | Feasibility |             |                           |
|             | Prototyping |             |                           |
|             | (if used)   |             |                           |
|             |             |             |                           |
|             | Data from   |             |                           |
|             | Modeling    |             |                           |
|             | and         |             |                           |
|             | Simulation  |             |                           |
|             | (if used)   |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| oversees,   |             |             |                           |
| tracks, and |             |             |                           |
| generally   |             |             |                           |
| supports    |             |             |                           |
| the         |             |             |                           |
| c           |             |             |                           |
| ontractor's |             |             |                           |
| analysis of |             |             |                           |
| software    |             |             |                           |
| re          |             |             |                           |
| quirements. |             |             |                           |
| This phase  |             |             |                           |
| of software |             |             |                           |
| development |             |             |                           |
| converts    |             |             |                           |
| s           |             |             |                           |
| ystem-level |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| allocated   |             |             |                           |
| to software |             |             |                           |
| in the      |             |             |                           |
| System /    |             |             |                           |
| Segment     |             |             |                           |
| Sp          |             |             |                           |
| ecification |             |             |                           |
| to          |             |             |                           |
| statements  |             |             |                           |
| of          |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| that are    |             |             |                           |
| complete,   |             |             |                           |
| consistent, |             |             |                           |
| testable,   |             |             |                           |
| and         |             |             |                           |
| feasible    |             |             |                           |
| for         |             |             |                           |
| software    |             |             |                           |
| impl        |             |             |                           |
| ementation. |             |             |                           |
| As part of  |             |             |                           |
| this        |             |             |                           |
| process,    |             |             |                           |
| the         |             |             |                           |
| contractor  |             |             |                           |
| develops    |             |             |                           |
| the overall |             |             |                           |
| software    |             |             |                           |
| ar          |             |             |                           |
| chitecture, |             |             |                           |
| identifies  |             |             |                           |
| principal   |             |             |                           |
| CSCIs and   |             |             |                           |
| CSCs, and   |             |             |                           |
| allocates   |             |             |                           |
| resources   |             |             |                           |
| to them     |             |             |                           |
| using the   |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| T           |             |             |                           |
| raceability |             |             |                           |
| Matrix      |             |             |                           |
| database.   |             |             |                           |
| Results are |             |             |                           |
| documented  |             |             |                           |
| in the      |             |             |                           |
| Software    |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| Spe         |             |             |                           |
| cification. |             |             |                           |
| This        |             |             |                           |
| process     |             |             |                           |
| also        |             |             |                           |
| updates     |             |             |                           |
| interface   |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| and         |             |             |                           |
| performance |             |             |                           |
| i           |             |             |                           |
| nformation. |             |             |                           |
|             |             |             |                           |
| Risk        |             |             |                           |
| management  |             |             |                           |
| and fault   |             |             |                           |
| prevention  |             |             |                           |
| are         |             |             |                           |
| extremely   |             |             |                           |
| important   |             |             |                           |
| at this     |             |             |                           |
| point in    |             |             |                           |
| the         |             |             |                           |
| software    |             |             |                           |
| development |             |             |                           |
| process.    |             |             |                           |
| Errors      |             |             |                           |
| discovered  |             |             |                           |
| and         |             |             |                           |
| corrected   |             |             |                           |
| here are    |             |             |                           |
| much less   |             |             |                           |
| costly to   |             |             |                           |
| correct     |             |             |                           |
| than errors |             |             |                           |
| discovered  |             |             |                           |
| during      |             |             |                           |
| imp         |             |             |                           |
| lementation |             |             |                           |
| or          |             |             |                           |
| m           |             |             |                           |
| aintenance. |             |             |                           |
| Feasibility |             |             |                           |
| prototyping |             |             |                           |
| and         |             |             |                           |
| modeling /  |             |             |                           |
| simulation  |             |             |                           |
| are the     |             |             |                           |
| most common |             |             |                           |
| methods of  |             |             |                           |
| risk        |             |             |                           |
| mitigation. |             |             |                           |
| Feasibility |             |             |                           |
| prototyping |             |             |                           |
| may consist |             |             |                           |
| of          |             |             |                           |
| conceptual  |             |             |                           |
| prototyping |             |             |                           |
| or          |             |             |                           |
| de          |             |             |                           |
| monstration |             |             |                           |
| p           |             |             |                           |
| rototyping. |             |             |                           |
|             |             |             |                           |
| Conceptual  |             |             |                           |
| prototyping |             |             |                           |
| provides a  |             |             |                           |
| tangible    |             |             |                           |
| rep         |             |             |                           |
| resentation |             |             |                           |
| of the      |             |             |                           |
| concept of  |             |             |                           |
| operation   |             |             |                           |
| and human / |             |             |                           |
| machine     |             |             |                           |
| interface   |             |             |                           |
| that can be |             |             |                           |
| critically  |             |             |                           |
| reviewed by |             |             |                           |
| the         |             |             |                           |
| customer or |             |             |                           |
| user. This  |             |             |                           |
| class of    |             |             |                           |
| prototyping |             |             |                           |
| often uses  |             |             |                           |
| rapid       |             |             |                           |
| prototyping |             |             |                           |
| techniques  |             |             |                           |
| requiring   |             |             |                           |
| the use of  |             |             |                           |
| non-target  |             |             |                           |
| hardware,   |             |             |                           |
| c           |             |             |                           |
| onstraining |             |             |                           |
| the display |             |             |                           |
| and menu    |             |             |                           |
| formats and |             |             |                           |
| medium      |             |             |                           |
| (i.e.,      |             |             |                           |
| graphics,   |             |             |                           |
| color,      |             |             |                           |
| fonts,      |             |             |                           |
| etc.) to    |             |             |                           |
| those       |             |             |                           |
| supported   |             |             |                           |
| on the      |             |             |                           |
| target      |             |             |                           |
| machines.   |             |             |                           |
|             |             |             |                           |
| De          |             |             |                           |
| monstration |             |             |                           |
| prototyping |             |             |                           |
| seeks to    |             |             |                           |
| ensure that |             |             |                           |
| at least    |             |             |                           |
| one         |             |             |                           |
| solution    |             |             |                           |
| exists for  |             |             |                           |
| a stated    |             |             |                           |
| r           |             |             |                           |
| equirement. |             |             |                           |
| De          |             |             |                           |
| monstration |             |             |                           |
| prototyping |             |             |                           |
| is          |             |             |                           |
| a           |             |             |                           |
| ccomplished |             |             |                           |
| on the      |             |             |                           |
| target      |             |             |                           |
| equipment,  |             |             |                           |
| and often   |             |             |                           |
| d           |             |             |                           |
| emonstrates |             |             |                           |
| only those  |             |             |                           |
| parts of    |             |             |                           |
| the target  |             |             |                           |
| environment |             |             |                           |
| that are    |             |             |                           |
| impacted by |             |             |                           |
| the         |             |             |                           |
| specific    |             |             |                           |
| requirement |             |             |                           |
| being       |             |             |                           |
| de          |             |             |                           |
| monstrated. |             |             |                           |
| For         |             |             |                           |
| instance, a |             |             |                           |
| de          |             |             |                           |
| monstration |             |             |                           |
| prototype   |             |             |                           |
| may show a  |             |             |                           |
| solution on |             |             |                           |
| a           |             |             |                           |
| co          |             |             |                           |
| mputational |             |             |                           |
| server      |             |             |                           |
| without     |             |             |                           |
| simulating  |             |             |                           |
| the entire  |             |             |                           |
| network.    |             |             |                           |
|             |             |             |                           |
| Simulation  |             |             |                           |
| and         |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| modeling    |             |             |                           |
| are used to |             |             |                           |
| demonstrate |             |             |                           |
| feasibility |             |             |                           |
| in those    |             |             |                           |
| cases where |             |             |                           |
| m           |             |             |                           |
| athematical |             |             |                           |
| or other    |             |             |                           |
| techniques  |             |             |                           |
| exist that  |             |             |                           |
| provide     |             |             |                           |
| insight     |             |             |                           |
| into the    |             |             |                           |
| question at |             |             |                           |
| hand, or    |             |             |                           |
| for those   |             |             |                           |
| questions   |             |             |                           |
| where       |             |             |                           |
| prototyping |             |             |                           |
| of a        |             |             |                           |
| solution    |             |             |                           |
| may be too  |             |             |                           |
| expensive   |             |             |                           |
| or          |             |             |                           |
| time        |             |             |                           |
| -consuming. |             |             |                           |
| Modeling of |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| requires an |             |             |                           |
| evaluation  |             |             |                           |
| plan that   |             |             |                           |
| documents   |             |             |                           |
| the         |             |             |                           |
| proposed    |             |             |                           |
| techniques  |             |             |                           |
| to be used, |             |             |                           |
| the type    |             |             |                           |
| and amount  |             |             |                           |
| of data to  |             |             |                           |
| be          |             |             |                           |
| collected,  |             |             |                           |
| the         |             |             |                           |
| algorithms  |             |             |                           |
| to be used, |             |             |                           |
| the         |             |             |                           |
| char        |             |             |                           |
| acteristics |             |             |                           |
| of the      |             |             |                           |
| system to   |             |             |                           |
| be          |             |             |                           |
| included,   |             |             |                           |
| and the     |             |             |                           |
| schedule to |             |             |                           |
| be          |             |             |                           |
| followed.   |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Conduct     |             |             |                           |
| Software    |             |             |                           |
| Sp          |             |             |                           |
| ecification |             |             |                           |
| Review\     |             |             |                           |
| D-7**       |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Baseline    | Service     |                           |
| hosts the   | Software    | Team        |                           |
| review      | R           | software    |                           |
|             | equirements | engineer    |                           |
| Service     | Sp          |             |                           |
| Team        | ecification |             |                           |
| specialists |             |             |                           |
| review and  |             |             |                           |
| comment     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| conducts a  |             |             |                           |
| formal      |             |             |                           |
| review of   |             |             |                           |
| the         |             |             |                           |
| Software    |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| Sp          |             |             |                           |
| ecification |             |             |                           |
| (SRS) in    |             |             |                           |
| accordance  |             |             |                           |
| with        |             |             |                           |
| MIL         |             |             |                           |
| -STD-1521B. |             |             |                           |
| The intent  |             |             |                           |
| is to       |             |             |                           |
| demonstrate |             |             |                           |
| that all    |             |             |                           |
| segment and |             |             |                           |
| Interface   |             |             |                           |
| Control     |             |             |                           |
| Document    |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| are         |             |             |                           |
| allocated   |             |             |                           |
| and traced  |             |             |                           |
| to Computer |             |             |                           |
| Software    |             |             |                           |
| Co          |             |             |                           |
| nfiguration |             |             |                           |
| Items, and  |             |             |                           |
| that the    |             |             |                           |
| methods of  |             |             |                           |
| verifying   |             |             |                           |
| and         |             |             |                           |
| validating  |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| are         |             |             |                           |
| reflected   |             |             |                           |
| in the SRS. |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| reviews the |             |             |                           |
| allocation  |             |             |                           |
| of          |             |             |                           |
| performance |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| to major    |             |             |                           |
| ar          |             |             |                           |
| chitectural |             |             |                           |
| components, |             |             |                           |
| as well as  |             |             |                           |
| external    |             |             |                           |
| and         |             |             |                           |
| internal    |             |             |                           |
| interface   |             |             |                           |
| re          |             |             |                           |
| quirements. |             |             |                           |
| Ad          |             |             |                           |
| ditionally, |             |             |                           |
| the Service |             |             |                           |
| Team        |             |             |                           |
| evaluates   |             |             |                           |
| the impact  |             |             |                           |
| of any      |             |             |                           |
| proposed    |             |             |                           |
| changes to  |             |             |                           |
| the System  |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| Spe         |             |             |                           |
| cification. |             |             |                           |
|             |             |             |                           |
| The Air     |             |             |                           |
| Force       |             |             |                           |
| Acquisition |             |             |                           |
| Model       |             |             |                           |
| (AFAM) is a |             |             |                           |
| good tool   |             |             |                           |
| to use when |             |             |                           |
| determining |             |             |                           |
| the         |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| for all     |             |             |                           |
| Systems     |             |             |                           |
| Engineering |             |             |                           |
| Reviews,    |             |             |                           |
| p           |             |             |                           |
| articularly |             |             |                           |
| for         |             |             |                           |
| software.   |             |             |                           |
| It is       |             |             |                           |
| available   |             |             |                           |
| on CD ROM   |             |             |                           |
| and         |             |             |                           |
| includes    |             |             |                           |
| Best        |             |             |                           |
| Practices   |             |             |                           |
| and Lessons |             |             |                           |
| Learned.    |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Conduct     |             |             |                           |
| Hardware    |             |             |                           |
| Sp          |             |             |                           |
| ecification |             |             |                           |
| Review\     |             |             |                           |
| D-8**       |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  |             |             |                           |
| conducts    |             |             |                           |
| review      |             |             |                           |
|             |             |             |                           |
| Service     |             |             |                           |
| Team        |             |             |                           |
| specialists |             |             |                           |
| participate |             |             |                           |
| and resolve |             |             |                           |
| problems    |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| and Service |             |             |                           |
| Team        |             |             |                           |
| jointly     |             |             |                           |
| review      |             |             |                           |
| hardware    |             |             |                           |
| spe         |             |             |                           |
| cifications |             |             |                           |
| in the      |             |             |                           |
| System      |             |             |                           |
| Sp          |             |             |                           |
| ecification |             |             |                           |
| to ensure   |             |             |                           |
| all         |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| can be      |             |             |                           |
| satisfied   |             |             |                           |
| by the      |             |             |                           |
| vendor's    |             |             |                           |
| COTS / NDI  |             |             |                           |
| hardware.   |             |             |                           |
| Any         |             |             |                           |
| di          |             |             |                           |
| screpancies |             |             |                           |
| or          |             |             |                           |
| conflicts   |             |             |                           |
| must be     |             |             |                           |
| corrected   |             |             |                           |
| now before  |             |             |                           |
| proceeding  |             |             |                           |
| further in  |             |             |                           |
| d           |             |             |                           |
| evelopment. |             |             |                           |
| When        |             |             |                           |
| problems    |             |             |                           |
| emerge, it  |             |             |                           |
| may be      |             |             |                           |
| possible to |             |             |                           |
| implement   |             |             |                           |
| fu          |             |             |                           |
| nctionality |             |             |                           |
| in software |             |             |                           |
| rather than |             |             |                           |
| hardware or |             |             |                           |
| it may      |             |             |                           |
| require a   |             |             |                           |
| change to   |             |             |                           |
| the System  |             |             |                           |
| Sp          |             |             |                           |
| ecification |             |             |                           |
| or program. |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee     |             |             |                           |
| Software    |             |             |                           |
| Preliminary |             |             |                           |
| Design\     |             |             |                           |
| D-9**       |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Service     | Software    |             |                           |
| Team        | Design      |             |                           |
| software    | Document    |             |                           |
| engineer    |             |             |                           |
|             | Interface   |             |                           |
|             | Design      |             |                           |
|             | Document    |             |                           |
|             |             |             |                           |
|             | Preliminary |             |                           |
|             | Software    |             |                           |
|             | Test Plan   |             |                           |
|             |             |             |                           |
|             | Updated RTM |             |                           |
|             |             |             |                           |
|             | Design      |             |                           |
|             | prototyping |             |                           |
|             | data        |             |                           |
|             |             |             |                           |
|             | Operational |             |                           |
|             | prototyping |             |                           |
|             | data        |             |                           |
|             |             |             |                           |
|             | Software    |             |                           |
|             | Development |             |                           |
|             | Folder(s)   |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| oversees,   |             |             |                           |
| tracks, and |             |             |                           |
| generally   |             |             |                           |
| supports    |             |             |                           |
| the         |             |             |                           |
| preliminary |             |             |                           |
| design      |             |             |                           |
| effort      |             |             |                           |
| during      |             |             |                           |
| which the   |             |             |                           |
| contractor  |             |             |                           |
| partitions  |             |             |                           |
| Computer    |             |             |                           |
| Software    |             |             |                           |
| Co          |             |             |                           |
| nfiguration |             |             |                           |
| Items       |             |             |                           |
| (CSCIs)     |             |             |                           |
| into        |             |             |                           |
| Computer    |             |             |                           |
| Software    |             |             |                           |
| Components  |             |             |                           |
| (CSCs) and  |             |             |                           |
| Computer    |             |             |                           |
| Software    |             |             |                           |
| Units       |             |             |                           |
| (CSUs), and |             |             |                           |
| allocates   |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| to them.    |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| also:       |             |             |                           |
|             |             |             |                           |
| -   Updates |             |             |                           |
|     the     |             |             |                           |
|     R       |             |             |                           |
| equirements |             |             |                           |
|     T       |             |             |                           |
| raceability |             |             |                           |
|     Matrix  |             |             |                           |
|     to      |             |             |                           |
|     reflect |             |             |                           |
|     the     |             |             |                           |
|             |             |             |                           |
|  allocation |             |             |                           |
|     of      |             |             |                           |
|             |             |             |                           |
|    software |             |             |                           |
|     r       |             |             |                           |
| equirements |             |             |                           |
|     to each |             |             |                           |
|     CSC or  |             |             |                           |
|     CSU;    |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|   Generates |             |             |                           |
|             |             |             |                           |
| preliminary |             |             |                           |
|             |             |             |                           |
|    software |             |             |                           |
|             |             |             |                           |
|  acceptance |             |             |                           |
|     test    |             |             |                           |
|     d       |             |             |                           |
| escriptions |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| procedures; |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Develops |             |             |                           |
|     a       |             |             |                           |
|             |             |             |                           |
| preliminary |             |             |                           |
|             |             |             |                           |
|    Software |             |             |                           |
|     Test    |             |             |                           |
|     Plan.   |             |             |                           |
|             |             |             |                           |
| As          |             |             |                           |
| development |             |             |                           |
| progresses, |             |             |                           |
| the         |             |             |                           |
| contractor  |             |             |                           |
| finalizes   |             |             |                           |
| the         |             |             |                           |
| top-level   |             |             |                           |
| software    |             |             |                           |
| a           |             |             |                           |
| rchitecture |             |             |                           |
| and places  |             |             |                           |
| products    |             |             |                           |
| under       |             |             |                           |
| software    |             |             |                           |
| co          |             |             |                           |
| nfiguration |             |             |                           |
| management. |             |             |                           |
|             |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| may perform |             |             |                           |
| some design |             |             |                           |
| prototyping |             |             |                           |
| to assess   |             |             |                           |
| all or      |             |             |                           |
| parts of    |             |             |                           |
| the CSCI    |             |             |                           |
| design      |             |             |                           |
| (e.g.,      |             |             |                           |
| interfaces  |             |             |                           |
| with new    |             |             |                           |
| hardware or |             |             |                           |
| new         |             |             |                           |
| operating   |             |             |                           |
| services).  |             |             |                           |
| Because     |             |             |                           |
| these       |             |             |                           |
| prototypes  |             |             |                           |
| demonstrate |             |             |                           |
| a solution, |             |             |                           |
| they often  |             |             |                           |
| provide the |             |             |                           |
| bulk (if    |             |             |                           |
| not all) of |             |             |                           |
| code        |             |             |                           |
| delivered   |             |             |                           |
| with the    |             |             |                           |
| product. As |             |             |                           |
| such,       |             |             |                           |
| prototyping |             |             |                           |
| can         |             |             |                           |
| complete    |             |             |                           |
| the design  |             |             |                           |
| and         |             |             |                           |
| accomplish  |             |             |                           |
| some        |             |             |                           |
| de          |             |             |                           |
| velopmental |             |             |                           |
| testing     |             |             |                           |
| while still |             |             |                           |
| formally    |             |             |                           |
| being part  |             |             |                           |
| of the      |             |             |                           |
| design      |             |             |                           |
| effort. The |             |             |                           |
| contractor  |             |             |                           |
| assumes the |             |             |                           |
| risk of     |             |             |                           |
| non         |             |             |                           |
| -acceptance |             |             |                           |
| of code by  |             |             |                           |
| the         |             |             |                           |
| customer,   |             |             |                           |
| and         |             |             |                           |
| mitigates   |             |             |                           |
| that risk   |             |             |                           |
| by          |             |             |                           |
| conducting  |             |             |                           |
| reviews     |             |             |                           |
| that match  |             |             |                           |
| the reviews |             |             |                           |
| conducted   |             |             |                           |
| on more     |             |             |                           |
| classically |             |             |                           |
| developed   |             |             |                           |
| software.   |             |             |                           |
|             |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| may perform |             |             |                           |
| some        |             |             |                           |
| operational |             |             |                           |
| prototyping |             |             |                           |
| during the  |             |             |                           |
| design      |             |             |                           |
| phase to    |             |             |                           |
| meet        |             |             |                           |
| hi          |             |             |                           |
| gh-priority |             |             |                           |
| user        |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| or to       |             |             |                           |
| demonstrate |             |             |                           |
| p           |             |             |                           |
| roject-wide |             |             |                           |
| ca          |             |             |                           |
| pabilities. |             |             |                           |
| Selecting a |             |             |                           |
| component   |             |             |                           |
| for         |             |             |                           |
| operational |             |             |                           |
| prototyping |             |             |                           |
| depends on  |             |             |                           |
| the risk    |             |             |                           |
| associated  |             |             |                           |
| with the    |             |             |                           |
| remaining   |             |             |                           |
| system      |             |             |                           |
| design.     |             |             |                           |
| Operational |             |             |                           |
| prototyping |             |             |                           |
| is          |             |             |                           |
| recommended |             |             |                           |
| for:        |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Critical |             |             |                           |
|             |             |             |                           |
| operational |             |             |                           |
|     con     |             |             |                           |
| siderations |             |             |                           |
|     that    |             |             |                           |
|     support |             |             |                           |
|             |             |             |                           |
|   specific, |             |             |                           |
|             |             |             |                           |
|    narrowly |             |             |                           |
|     defined |             |             |                           |
|     user    |             |             |                           |
|     needs;  |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Software |             |             |                           |
|     au      |             |             |                           |
| gmentations |             |             |                           |
|     to      |             |             |                           |
|             |             |             |                           |
|   operating |             |             |                           |
|     system  |             |             |                           |
|             |             |             |                           |
|   services, |             |             |                           |
|     data    |             |             |                           |
|             |             |             |                           |
| management, |             |             |                           |
|     or      |             |             |                           |
|     network |             |             |                           |
|             |             |             |                           |
|  functions; |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Software |             |             |                           |
|     back    |             |             |                           |
|     planes  |             |             |                           |
|             |             |             |                           |
|    required |             |             |                           |
|     to      |             |             |                           |
|     support |             |             |                           |
|             |             |             |                           |
|    multiple |             |             |                           |
|     build   |             |             |                           |
| s/releases; |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|   Libraries |             |             |                           |
|     needed  |             |             |                           |
|     early   |             |             |                           |
|     in the  |             |             |                           |
|             |             |             |                           |
| development |             |             |                           |
|     cycle;  |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
| Interfacing |             |             |                           |
|     code to |             |             |                           |
|             |             |             |                           |
|    existing |             |             |                           |
|             |             |             |                           |
|    external |             |             |                           |
|     systems |             |             |                           |
|     to      |             |             |                           |
|     allow   |             |             |                           |
|             |             |             |                           |
|   continued |             |             |                           |
|             |             |             |                           |
|   operation |             |             |                           |
|     during  |             |             |                           |
|     switch  |             |             |                           |
|     over.   |             |             |                           |
|             |             |             |                           |
| Operational |             |             |                           |
| prototypes  |             |             |                           |
| are         |             |             |                           |
| expected to |             |             |                           |
| provide     |             |             |                           |
| fully       |             |             |                           |
| operational |             |             |                           |
| code. They  |             |             |                           |
| are totally |             |             |                           |
| designed,   |             |             |                           |
| i           |             |             |                           |
| mplemented, |             |             |                           |
| and tested  |             |             |                           |
| while the   |             |             |                           |
| actual      |             |             |                           |
| design of   |             |             |                           |
| the overall |             |             |                           |
| product is  |             |             |                           |
| underway.   |             |             |                           |
| As with     |             |             |                           |
| design      |             |             |                           |
| assessment  |             |             |                           |
| p           |             |             |                           |
| rototyping, |             |             |                           |
| the         |             |             |                           |
| contractor  |             |             |                           |
| mitigates   |             |             |                           |
| risk by     |             |             |                           |
| conducting  |             |             |                           |
| reviews at  |             |             |                           |
| points      |             |             |                           |
| within the  |             |             |                           |
| prototype   |             |             |                           |
| development |             |             |                           |
| that match  |             |             |                           |
| more        |             |             |                           |
| classically |             |             |                           |
| developed   |             |             |                           |
| software.   |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Conduct     |             |             |                           |
| Preliminary |             |             |                           |
| Design      |             |             |                           |
| Review\     |             |             |                           |
| D-10**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Formal      | Service     |                           |
| conducts    | review of   | Team        |                           |
|             | contractor  | software    |                           |
| Service     | design      | engineer or |                           |
| Team        | approach    | appropriate |                           |
| specialists |             | specialist  |                           |
| review and  | Formal      |             |                           |
| comment     | comment on  |             |                           |
|             | all design  |             |                           |
|             | documents   |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| presents    |             |             |                           |
| the         |             |             |                           |
| preliminary |             |             |                           |
| system      |             |             |                           |
| ar          |             |             |                           |
| chitecture, |             |             |                           |
| overall     |             |             |                           |
| design      |             |             |                           |
| approach,   |             |             |                           |
| and         |             |             |                           |
| Co          |             |             |                           |
| nfiguration |             |             |                           |
| Item (CI)   |             |             |                           |
| structure   |             |             |                           |
| to the      |             |             |                           |
| Service     |             |             |                           |
| Team at the |             |             |                           |
| Preliminary |             |             |                           |
| Design      |             |             |                           |
| Review      |             |             |                           |
| (PDR) in    |             |             |                           |
| accordance  |             |             |                           |
| with        |             |             |                           |
| MIL         |             |             |                           |
| -STD-1521B. |             |             |                           |
| The         |             |             |                           |
| objective   |             |             |                           |
| is to       |             |             |                           |
| demonstrate |             |             |                           |
| formally    |             |             |                           |
| that they   |             |             |                           |
| are         |             |             |                           |
| s           |             |             |                           |
| ufficiently |             |             |                           |
| mature to   |             |             |                           |
| begin       |             |             |                           |
| detailed    |             |             |                           |
| design.     |             |             |                           |
| Sometimes   |             |             |                           |
| the         |             |             |                           |
| contractor  |             |             |                           |
| conducts a  |             |             |                           |
| series of   |             |             |                           |
| PDRs, one   |             |             |                           |
| for each    |             |             |                           |
| Co          |             |             |                           |
| nfiguration |             |             |                           |
| Item or     |             |             |                           |
| aggregate   |             |             |                           |
| of CIs      |             |             |                           |
| leading to  |             |             |                           |
| a system    |             |             |                           |
| PDR at      |             |             |                           |
| completion. |             |             |                           |
| Each PDR    |             |             |                           |
| must (1)    |             |             |                           |
| ensure the  |             |             |                           |
| process     |             |             |                           |
| used to     |             |             |                           |
| arrive at   |             |             |                           |
| functional  |             |             |                           |
| and         |             |             |                           |
| performance |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| for each    |             |             |                           |
| Co          |             |             |                           |
| nfiguration |             |             |                           |
| Item is     |             |             |                           |
| rigorous    |             |             |                           |
| and         |             |             |                           |
| complete,   |             |             |                           |
| including   |             |             |                           |
| trades and  |             |             |                           |
| a           |             |             |                           |
| llocations; |             |             |                           |
| and (2)     |             |             |                           |
| establish   |             |             |                           |
| an audit    |             |             |                           |
| trail from  |             |             |                           |
| user        |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| to the      |             |             |                           |
| functional  |             |             |                           |
| baseline,   |             |             |                           |
| sub         |             |             |                           |
| stantiating |             |             |                           |
| changes as  |             |             |                           |
| necessary.  |             |             |                           |
|             |             |             |                           |
| The PDR is  |             |             |                           |
| intended to |             |             |                           |
| demonstrate |             |             |                           |
| the         |             |             |                           |
| contractor  |             |             |                           |
| has:        |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|   Developed |             |             |                           |
|     a       |             |             |                           |
|     s       |             |             |                           |
| atisfactory |             |             |                           |
|             |             |             |                           |
| preliminary |             |             |                           |
|     design  |             |             |                           |
|     for     |             |             |                           |
|     i       |             |             |                           |
| mplementing |             |             |                           |
|     segment |             |             |                           |
|     re      |             |             |                           |
| quirements; |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|   Allocated |             |             |                           |
|     and     |             |             |                           |
|     traced  |             |             |                           |
|     all     |             |             |                           |
|     r       |             |             |                           |
| equirements |             |             |                           |
|     to the  |             |             |                           |
|     CSC     |             |             |                           |
|     level;  |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|  Identified |             |             |                           |
|     all     |             |             |                           |
|     open    |             |             |                           |
|     issues  |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|  identified |             |             |                           |
|     a plan  |             |             |                           |
|     for     |             |             |                           |
|     closing |             |             |                           |
|     them;   |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|  Identified |             |             |                           |
|     all     |             |             |                           |
|             |             |             |                           |
|   technical |             |             |                           |
|     and     |             |             |                           |
|     p       |             |             |                           |
| rogrammatic |             |             |                           |
|             |             |             |                           |
|   high-risk |             |             |                           |
|     areas;  |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|   Developed |             |             |                           |
|     a       |             |             |                           |
|     viable  |             |             |                           |
|             |             |             |                           |
|    database |             |             |                           |
|     design; |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|   Developed |             |             |                           |
|     a       |             |             |                           |
|             |             |             |                           |
|    complete |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|    adequate |             |             |                           |
|     plan    |             |             |                           |
|     for     |             |             |                           |
|             |             |             |                           |
|    software |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|  operations |             |             |                           |
|             |             |             |                           |
|    testing. |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee     |             |             |                           |
| Software    |             |             |                           |
| Detailed    |             |             |                           |
| Design\     |             |             |                           |
| D-11**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Preliminary | Service     |                           |
| designs     | Software    | Team        |                           |
|             | Design      | software    |                           |
| Service     | Document    | specialist  |                           |
| Team        |             |             |                           |
| oversees    | Preliminary |             |                           |
| and         | Interface   |             |                           |
| supports    | Design      |             |                           |
|             | Document    |             |                           |
|             |             |             |                           |
|             | Updated     |             |                           |
|             | Software    |             |                           |
|             | Test Plan   |             |                           |
|             |             |             |                           |
|             | Preliminary |             |                           |
|             | Software    |             |                           |
|             | Test        |             |                           |
|             | Description |             |                           |
|             |             |             |                           |
|             | Updated     |             |                           |
|             | R           |             |                           |
|             | equirements |             |                           |
|             | T           |             |                           |
|             | raceability |             |                           |
|             | Matrix      |             |                           |
|             |             |             |                           |
|             | Test        |             |                           |
|             | R           |             |                           |
|             | equirements |             |                           |
|             | Matrix      |             |                           |
|             |             |             |                           |
|             | Updated     |             |                           |
|             | Software    |             |                           |
|             | Development |             |                           |
|             | Folder(s)   |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descr    |             |             |                           |
| iption]{.un |             |             |                           |
| derline}**: |             |             |                           |
|             |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| oversees,   |             |             |                           |
| tracks, and |             |             |                           |
| generally   |             |             |                           |
| supports    |             |             |                           |
| the         |             |             |                           |
| c           |             |             |                           |
| ontractor's |             |             |                           |
| detailed    |             |             |                           |
| software    |             |             |                           |
| design      |             |             |                           |
| effort      |             |             |                           |
| during      |             |             |                           |
| which each  |             |             |                           |
| Computer    |             |             |                           |
| Software    |             |             |                           |
| Component   |             |             |                           |
| (CSC) is    |             |             |                           |
| decomposed  |             |             |                           |
| into a      |             |             |                           |
| complete    |             |             |                           |
| structure   |             |             |                           |
| of          |             |             |                           |
| lower-level |             |             |                           |
| CSCs and    |             |             |                           |
| Computer    |             |             |                           |
| Software    |             |             |                           |
| Units       |             |             |                           |
| (CSUs).     |             |             |                           |
| Interfaces  |             |             |                           |
| are         |             |             |                           |
| described   |             |             |                           |
| to the      |             |             |                           |
| lowest      |             |             |                           |
| component   |             |             |                           |
| level; data |             |             |                           |
| structures  |             |             |                           |
| and         |             |             |                           |
| contents    |             |             |                           |
| are         |             |             |                           |
| defined;    |             |             |                           |
| input and   |             |             |                           |
| output      |             |             |                           |
| details are |             |             |                           |
| designed;   |             |             |                           |
| and         |             |             |                           |
| decisions   |             |             |                           |
| are made on |             |             |                           |
| handling    |             |             |                           |
| errors. As  |             |             |                           |
| design      |             |             |                           |
| activities  |             |             |                           |
| proceed,    |             |             |                           |
| the         |             |             |                           |
| contractor  |             |             |                           |
| finalizes   |             |             |                           |
| the         |             |             |                           |
| Software    |             |             |                           |
| Design      |             |             |                           |
| Document    |             |             |                           |
| and         |             |             |                           |
| Interface   |             |             |                           |
| Design      |             |             |                           |
| Document,   |             |             |                           |
| begins      |             |             |                           |
| development |             |             |                           |
| on the      |             |             |                           |
| Software    |             |             |                           |
| Test        |             |             |                           |
| D           |             |             |                           |
| escription, |             |             |                           |
| and updates |             |             |                           |
| the         |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| T           |             |             |                           |
| raceability |             |             |                           |
| Matrix.     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Conduct     |             |             |                           |
| Critical    |             |             |                           |
| Design      |             |             |                           |
| Review\     |             |             |                           |
| D-12**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Detailed    | Appropriate |                           |
| conducts    | CSCI Design | Service     |                           |
|             | Documents   | Team        |                           |
| Service     |             | specialist  |                           |
| Team        | Detailed    |             |                           |
| specialists | CSCI        |             |                           |
| review and  | Interface   |             |                           |
| comment     | Design      |             |                           |
|             | Documents   |             |                           |
|             |             |             |                           |
|             | Preliminary |             |                           |
|             | Software    |             |                           |
|             | Test        |             |                           |
|             | D           |             |                           |
|             | escriptions |             |                           |
|             |             |             |                           |
|             | Draft CSCI  |             |                           |
|             | test        |             |                           |
|             | procedures  |             |                           |
|             |             |             |                           |
|             | Updated     |             |                           |
|             | R           |             |                           |
|             | equirements |             |                           |
|             | T           |             |                           |
|             | raceability |             |                           |
|             | Matrix      |             |                           |
|             |             |             |                           |
|             | Undated     |             |                           |
|             | CSCI        |             |                           |
|             | development |             |                           |
|             | metrics     |             |                           |
|             |             |             |                           |
|             | Undated     |             |                           |
|             | cost and    |             |                           |
|             | schedule    |             |                           |
|             | estimates   |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| Critical    |             |             |                           |
| Design      |             |             |                           |
| Review      |             |             |                           |
| determines  |             |             |                           |
| (1) whether |             |             |                           |
| the         |             |             |                           |
| detailed    |             |             |                           |
| design      |             |             |                           |
| meets the   |             |             |                           |
| specified   |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| in the      |             |             |                           |
| appropriate |             |             |                           |
| de          |             |             |                           |
| velopmental |             |             |                           |
| baseline,   |             |             |                           |
| and (2)     |             |             |                           |
| whether the |             |             |                           |
| design is   |             |             |                           |
| complete    |             |             |                           |
| and ready   |             |             |                           |
| to be       |             |             |                           |
| implemented |             |             |                           |
| via         |             |             |                           |
| detailed    |             |             |                           |
| software    |             |             |                           |
| code and    |             |             |                           |
| test. The   |             |             |                           |
| contractor  |             |             |                           |
| conducts    |             |             |                           |
| the         |             |             |                           |
| Critical    |             |             |                           |
| Design      |             |             |                           |
| Review in   |             |             |                           |
| accordance  |             |             |                           |
| with        |             |             |                           |
| MIL         |             |             |                           |
| -STD-1521B, |             |             |                           |
| with the    |             |             |                           |
| Service     |             |             |                           |
| Team        |             |             |                           |
| reviewing   |             |             |                           |
| and         |             |             |                           |
| evaluating  |             |             |                           |
| contractor  |             |             |                           |
| activity    |             |             |                           |
| and         |             |             |                           |
| progress.   |             |             |                           |
| Often, a    |             |             |                           |
| series of   |             |             |                           |
| Critical    |             |             |                           |
| Design      |             |             |                           |
| Reviews are |             |             |                           |
| conducted,  |             |             |                           |
| one for     |             |             |                           |
| each        |             |             |                           |
| Co          |             |             |                           |
| nfiguration |             |             |                           |
| Item in the |             |             |                           |
| co          |             |             |                           |
| nfiguration |             |             |                           |
| tree,       |             |             |                           |
| rather than |             |             |                           |
| a single    |             |             |                           |
| \"system\"  |             |             |                           |
| CDR.        |             |             |                           |
|             |             |             |                           |
| Sp          |             |             |                           |
| ecifically, |             |             |                           |
| the CDR     |             |             |                           |
| reviews     |             |             |                           |
| formally    |             |             |                           |
| the design  |             |             |                           |
| of each     |             |             |                           |
| hardware    |             |             |                           |
| and         |             |             |                           |
| software    |             |             |                           |
| co          |             |             |                           |
| nfiguration |             |             |                           |
| item and    |             |             |                           |
| determines  |             |             |                           |
| whether:    |             |             |                           |
|             |             |             |                           |
| -   All     |             |             |                           |
|             |             |             |                           |
|    software |             |             |                           |
|     r       |             |             |                           |
| equirements |             |             |                           |
|     are     |             |             |                           |
|             |             |             |                           |
|   allocated |             |             |                           |
|     and     |             |             |                           |
|     traced  |             |             |                           |
|     to the  |             |             |                           |
|     CSU     |             |             |                           |
|     level;  |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Detailed |             |             |                           |
|     designs |             |             |                           |
|     reflect |             |             |                           |
|     an      |             |             |                           |
|     a       |             |             |                           |
| rchitecture |             |             |                           |
|     that    |             |             |                           |
|     sat     |             |             |                           |
| isfactorily |             |             |                           |
|             |             |             |                           |
|   minimizes |             |             |                           |
|     risk;   |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Database |             |             |                           |
|     design  |             |             |                           |
|     is      |             |             |                           |
|             |             |             |                           |
|    complete |             |             |                           |
|     and     |             |             |                           |
|     meets   |             |             |                           |
|     all     |             |             |                           |
|     re      |             |             |                           |
| quirements; |             |             |                           |
|             |             |             |                           |
| -   A       |             |             |                           |
|             |             |             |                           |
|    complete |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|    adequate |             |             |                           |
|     plan is |             |             |                           |
|     in      |             |             |                           |
|     place   |             |             |                           |
|     for     |             |             |                           |
|             |             |             |                           |
|    software |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|  operations |             |             |                           |
|             |             |             |                           |
|    testing; |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| -   Changes |             |             |                           |
|     to      |             |             |                           |
|     r       |             |             |                           |
| equirements |             |             |                           |
|     have    |             |             |                           |
|     been    |             |             |                           |
|             |             |             |                           |
|    analyzed |             |             |                           |
|     for     |             |             |                           |
|     impact  |             |             |                           |
|     on      |             |             |                           |
|             |             |             |                           |
|    detailed |             |             |                           |
|             |             |             |                           |
|    software |             |             |                           |
|     design, |             |             |                           |
|             |             |             |                           |
|   including |             |             |                           |
|             |             |             |                           |
|    internal |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|    external |             |             |                           |
|             |             |             |                           |
|  interfaces |             |             |                           |
|     (p      |             |             |                           |
| articularly |             |             |                           |
|             |             |             |                           |
|  associated |             |             |                           |
|     COTS /  |             |             |                           |
|     NDI     |             |             |                           |
|             |             |             |                           |
|  hardware). |             |             |                           |
|             |             |             |                           |
| As part of  |             |             |                           |
| this        |             |             |                           |
| activity,   |             |             |                           |
| the         |             |             |                           |
| contractor  |             |             |                           |
| should      |             |             |                           |
| prototype   |             |             |                           |
| both CSCI   |             |             |                           |
| external    |             |             |                           |
| interfaces  |             |             |                           |
| to other    |             |             |                           |
| NAS systems |             |             |                           |
| and         |             |             |                           |
| internal    |             |             |                           |
| interfaces  |             |             |                           |
| to COTS /   |             |             |                           |
| NDI         |             |             |                           |
| hardware.   |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| should also |             |             |                           |
| identify    |             |             |                           |
| those       |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| that have   |             |             |                           |
| already     |             |             |                           |
| been tested |             |             |                           |
| and for     |             |             |                           |
| which test  |             |             |                           |
| data is     |             |             |                           |
| available   |             |             |                           |
| (typically  |             |             |                           |
| NDI / COTS  |             |             |                           |
| c           |             |             |                           |
| omponents). |             |             |                           |
|             |             |             |                           |
| <table>     |             |             |                           |
| <colgroup>  |             |             |                           |
| <col        |             |             |                           |
| style="wid  |             |             |                           |
| th: 25%" /> |             |             |                           |
| <col        |             |             |                           |
| style="wid  |             |             |                           |
| th: 25%" /> |             |             |                           |
| <col        |             |             |                           |
| style="wid  |             |             |                           |
| th: 25%" /> |             |             |                           |
| <col        |             |             |                           |
| style="wid  |             |             |                           |
| th: 23%" /> |             |             |                           |
| </colgroup> |             |             |                           |
| <tbody>     |             |             |                           |
| <tr         |             |             |                           |
| c           |             |             |                           |
| lass="odd"> |             |             |                           |
| <td         |             |             |                           |
| colspan=    |             |             |                           |
| "4"><p><str |             |             |                           |
| ong>Activit |             |             |                           |
| y:</strong> |             |             |                           |
| <           |             |             |                           |
| strong>Make |             |             |                           |
| Product     |             |             |                           |
| De          |             |             |                           |
| monstration |             |             |                           |
| Decision</  |             |             |                           |
| strong></p> |             |             |                           |
| <p><strong> |             |             |                           |
| D-13</stron |             |             |                           |
| g></p></td> |             |             |                           |
| </tr>       |             |             |                           |
| <tr         |             |             |                           |
| cl          |             |             |                           |
| ass="even"> |             |             |                           |
| <           |             |             |                           |
| td><strong> |             |             |                           |
| Responsible |             |             |                           |
| Agent</s    |             |             |                           |
| trong></td> |             |             |                           |
| <td><strong |             |             |                           |
| >Product</s |             |             |                           |
| trong></td> |             |             |                           |
| <td><stro   |             |             |                           |
| ng>Approval |             |             |                           |
| A           |             |             |                           |
| uthority</s |             |             |                           |
| trong></td> |             |             |                           |
| <td><s      |             |             |                           |
| trong>Tools |             |             |                           |
| and         |             |             |                           |
| Aids</s     |             |             |                           |
| trong></td> |             |             |                           |
| </tr>       |             |             |                           |
| <tr         |             |             |                           |
| c           |             |             |                           |
| lass="odd"> |             |             |                           |
| <td>Service |             |             |                           |
| team        |             |             |                           |
| leader      |             |             |                           |
| provides    |             |             |                           |
| information |             |             |                           |
| that        |             |             |                           |
| d           |             |             |                           |
| emonstrates |             |             |                           |
| achievement |             |             |                           |
| of          |             |             |                           |
| product     |             |             |                           |
| integration |             |             |                           |
| decision    |             |             |                           |
| cr          |             |             |                           |
| iteria</td> |             |             |                           |
| <td>Product |             |             |                           |
| De          |             |             |                           |
| monstration |             |             |                           |
| Decision    |             |             |                           |
| Memo        |             |             |                           |
| randum</td> |             |             |                           |
| <td>Vice    |             |             |                           |
| President   |             |             |                           |
| (ATO) or    |             |             |                           |
| Director    |             |             |                           |
| (non-ATO)   |             |             |                           |
| of the      |             |             |                           |
| i           |             |             |                           |
| mplementing |             |             |                           |
| service     |             |             |                           |
| organi      |             |             |                           |
| zation</td> |             |             |                           |
| <td></td>   |             |             |                           |
| </tr>       |             |             |                           |
| <tr         |             |             |                           |
| cl          |             |             |                           |
| ass="even"> |             |             |                           |
| <td         |             |             |                           |
| colspan="4  |             |             |                           |
| "><p><stron |             |             |                           |
| g>Discussio |             |             |                           |
| n:</strong> |             |             |                           |
| The         |             |             |                           |
| service     |             |             |                           |
| team        |             |             |                           |
| leader      |             |             |                           |
| presents    |             |             |                           |
| the         |             |             |                           |
| results     |             |             |                           |
| of          |             |             |                           |
| product     |             |             |                           |
| integration |             |             |                           |
| to the      |             |             |                           |
| Vice        |             |             |                           |
| President   |             |             |                           |
| (ATO) or    |             |             |                           |
| Director    |             |             |                           |
| (non-ATO)   |             |             |                           |
| of the      |             |             |                           |
| i           |             |             |                           |
| mplementing |             |             |                           |
| service     |             |             |                           |
| or          |             |             |                           |
| ganization. |             |             |                           |
| The Vice    |             |             |                           |
| President   |             |             |                           |
| or          |             |             |                           |
| Director,   |             |             |                           |
| as          |             |             |                           |
| a           |             |             |                           |
| ppropriate, |             |             |                           |
| reviews     |             |             |                           |
| product     |             |             |                           |
| knowledge   |             |             |                           |
| to          |             |             |                           |
| determine   |             |             |                           |
| if the      |             |             |                           |
| following   |             |             |                           |
| decision    |             |             |                           |
| criteria    |             |             |                           |
| have been   |             |             |                           |
| sat         |             |             |                           |
| isfied:</p> |             |             |                           |
| <ul>        |             |             |                           |
| <li><p>Key  |             |             |                           |
| product     |             |             |                           |
| char        |             |             |                           |
| acteristics |             |             |                           |
| are         |             |             |                           |
| define      |             |             |                           |
| d;</p></li> |             |             |                           |
| <li><p>S    |             |             |                           |
| takeholders |             |             |                           |
| agree       |             |             |                           |
| that        |             |             |                           |
| product     |             |             |                           |
| design      |             |             |                           |
| and         |             |             |                           |
| fu          |             |             |                           |
| nctionality |             |             |                           |
| satisfy     |             |             |                           |
| customer    |             |             |                           |
| requirement |             |             |                           |
| s;</p></li> |             |             |                           |
| <l          |             |             |                           |
| i><p>System |             |             |                           |
| design      |             |             |                           |
| reviews     |             |             |                           |
| are         |             |             |                           |
| complet     |             |             |                           |
| e;</p></li> |             |             |                           |
| <li><p>     |             |             |                           |
| Engineering |             |             |                           |
| drawings    |             |             |                           |
| are         |             |             |                           |
| complet     |             |             |                           |
| e;</p></li> |             |             |                           |
| <li>        |             |             |                           |
| <p>Detailed |             |             |                           |
| softwa      |             |             |                           |
| re/firmware |             |             |                           |
| design is   |             |             |                           |
| complete,   |             |             |                           |
| including   |             |             |                           |
| critical    |             |             |                           |
| software    |             |             |                           |
| processes   |             |             |                           |
| and         |             |             |                           |
| thread      |             |             |                           |
| s;</p></li> |             |             |                           |
| <li><p>RMA  |             |             |                           |
| goals are   |             |             |                           |
| defined     |             |             |                           |
| and         |             |             |                           |
| planning    |             |             |                           |
| is          |             |             |                           |
| complet     |             |             |                           |
| e;</p></li> |             |             |                           |
| <li         |             |             |                           |
| ><p>Failure |             |             |                           |
| modes and   |             |             |                           |
| effects     |             |             |                           |
| analysis    |             |             |                           |
| is          |             |             |                           |
| complete;   |             |             |                           |
| a           |             |             |                           |
| nd</p></li> |             |             |                           |
| <li>        |             |             |                           |
| <p>Critical |             |             |                           |
| ma          |             |             |                           |
| nufacturing |             |             |                           |
| processes   |             |             |                           |
| are         |             |             |                           |
| identifi    |             |             |                           |
| ed</p></li> |             |             |                           |
| </ul>       |             |             |                           |
| <p>If so,   |             |             |                           |
| the         |             |             |                           |
| decision    |             |             |                           |
| authority   |             |             |                           |
| issues a    |             |             |                           |
| decision    |             |             |                           |
| memorandum  |             |             |                           |
| authorizing |             |             |                           |
| the         |             |             |                           |
| investment  |             |             |                           |
| program     |             |             |                           |
| to          |             |             |                           |
| proceed     |             |             |                           |
| to          |             |             |                           |
| initial     |             |             |                           |
| ma          |             |             |                           |
| nufacturing |             |             |                           |
| of          |             |             |                           |
| fi          |             |             |                           |
| rst-article |             |             |                           |
| units for   |             |             |                           |
| use in      |             |             |                           |
| product     |             |             |                           |
| dem         |             |             |                           |
| onstration. |             |             |                           |
| Findings    |             |             |                           |
| are         |             |             |                           |
| recorded    |             |             |                           |
| in the      |             |             |                           |
| decision    |             |             |                           |
| memorandum  |             |             |                           |
| along       |             |             |                           |
| with any    |             |             |                           |
| required    |             |             |                           |
| action      |             |             |                           |
| s.</p></td> |             |             |                           |
| </tr>       |             |             |                           |
| </tbody>    |             |             |                           |
| </table>    |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee     |             |             |                           |
| Software    |             |             |                           |
| Coding and  |             |             |                           |
| CSU         |             |             |                           |
| Testing\    |             |             |                           |
| D-14**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | CSU code    | Service     |                           |
| develops    |             | Team        |                           |
|             | Updated     | software    |                           |
| Service     | Software    | specialist  |                           |
| Team        | Development |             |                           |
| oversees,   | Folder(s),  |             |                           |
| tracks,     | including   |             |                           |
| supports    | CSU Test    |             |                           |
|             | Procedures  |             |                           |
|             |             |             |                           |
|             | Updated RTM |             |                           |
|             |             |             |                           |
|             | Test        |             |                           |
|             | R           |             |                           |
|             | equirements |             |                           |
|             | Matrix      |             |                           |
|             |             |             |                           |
|             | Test        |             |                           |
|             | Development |             |                           |
|             | Folder(s),  |             |                           |
|             | including   |             |                           |
|             | Software    |             |                           |
|             | Turnover    |             |                           |
|             | Plan        |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| oversees,   |             |             |                           |
| tracks, and |             |             |                           |
| generally   |             |             |                           |
| supports    |             |             |                           |
| contractor  |             |             |                           |
| software    |             |             |                           |
| CSU coding  |             |             |                           |
| and         |             |             |                           |
| testing.    |             |             |                           |
| Coding and  |             |             |                           |
| testing     |             |             |                           |
| generally   |             |             |                           |
| begins      |             |             |                           |
| after       |             |             |                           |
| completion  |             |             |                           |
| of detailed |             |             |                           |
| design and  |             |             |                           |
| a           |             |             |                           |
| successful  |             |             |                           |
| CDR,        |             |             |                           |
| although    |             |             |                           |
| some        |             |             |                           |
| prototype   |             |             |                           |
| code may    |             |             |                           |
| have been   |             |             |                           |
| produced    |             |             |                           |
| during      |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| analysis or |             |             |                           |
| preliminary |             |             |                           |
| design.     |             |             |                           |
| Coding and  |             |             |                           |
| CSU testing |             |             |                           |
| implements  |             |             |                           |
| the design  |             |             |                           |
| validated   |             |             |                           |
| by the CDR. |             |             |                           |
| This        |             |             |                           |
| activity    |             |             |                           |
| produces a  |             |             |                           |
| set of      |             |             |                           |
| program     |             |             |                           |
| units that  |             |             |                           |
| completely  |             |             |                           |
| implement   |             |             |                           |
| the         |             |             |                           |
| functional  |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| specified   |             |             |                           |
| in the      |             |             |                           |
| Software    |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| Spe         |             |             |                           |
| cification. |             |             |                           |
|             |             |             |                           |
| Go to       |             |             |                           |
| [htt        |             |             |                           |
| p://www.sts |             |             |                           |
| c.hill.af.m |             |             |                           |
| il](http:// |             |             |                           |
| www.stsc.hi |             |             |                           |
| ll.af.mil/) |             |             |                           |
| and click   |             |             |                           |
| on the      |             |             |                           |
| subject     |             |             |                           |
| \"SW        |             |             |                           |
| Testing\"   |             |             |                           |
| in          |             |             |                           |
| \           |             |             |                           |
| "Technology |             |             |                           |
| Areas\" for |             |             |                           |
| information |             |             |                           |
| on ATPS, a  |             |             |                           |
| set of      |             |             |                           |
| expert      |             |             |                           |
| s           |             |             |                           |
| ystem-based |             |             |                           |
| tools       |             |             |                           |
| designed to |             |             |                           |
| aid         |             |             |                           |
| acquisition |             |             |                           |
| staffs in   |             |             |                           |
| the         |             |             |                           |
| oversight   |             |             |                           |
| of test and |             |             |                           |
| evaluation. |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee     |             |             |                           |
| Hardware    |             |             |                           |
| A           |             |             |                           |
| cquisition\ |             |             |                           |
| D-15**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Fabricated  | Quality     |                           |
| obtains     | hardware    | Reliability |                           |
| hardware    | components  | Officer or  |                           |
|             |             | designated  |                           |
| Service     |             | official    |                           |
| Team        |             |             |                           |
| Quality     |             |             |                           |
| Reliability |             |             |                           |
| Officer     |             |             |                           |
| enforces    |             |             |                           |
| contract    |             |             |                           |
| provisions  |             |             |                           |
| and accepts |             |             |                           |
| qualified   |             |             |                           |
| units       |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| fabricates  |             |             |                           |
| or procures |             |             |                           |
| the COTS /  |             |             |                           |
| NDI         |             |             |                           |
| hardware    |             |             |                           |
| components  |             |             |                           |
| of the      |             |             |                           |
| system. The |             |             |                           |
| Service     |             |             |                           |
| Team        |             |             |                           |
| ensures     |             |             |                           |
| hardware    |             |             |                           |
| components  |             |             |                           |
| are         |             |             |                           |
| fabricated  |             |             |                           |
| in          |             |             |                           |
| accordance  |             |             |                           |
| with        |             |             |                           |
| provisions  |             |             |                           |
| of the      |             |             |                           |
| contract    |             |             |                           |
| and accepts |             |             |                           |
| delivery of |             |             |                           |
| hardware    |             |             |                           |
| units upon  |             |             |                           |
| completion  |             |             |                           |
| of all      |             |             |                           |
| required    |             |             |                           |
| testing and |             |             |                           |
| quality     |             |             |                           |
| assurance   |             |             |                           |
| checks.     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Conduct     |             |             |                           |
| Integration |             |             |                           |
| Test        |             |             |                           |
| Review\     |             |             |                           |
| D-16**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Comments on | Service     |                           |
| conducts    | code        | Team        |                           |
| review      | quality     | software    |                           |
|             |             | specialist  |                           |
| Service     | Permission  |             |                           |
| Team        | to proceed  |             |                           |
| reviews and | to          |             |                           |
| evaluates   | integration |             |                           |
| contractor  | testing     |             |                           |
| readiness   |             |             |                           |
| to          |             |             |                           |
| integrate   |             |             |                           |
| and test    |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| conducts    |             |             |                           |
| the CSC /   |             |             |                           |
| CSCI        |             |             |                           |
| Integration |             |             |                           |
| Test Review |             |             |                           |
| (ITR) at    |             |             |                           |
| the         |             |             |                           |
| completion  |             |             |                           |
| of CSC      |             |             |                           |
| testing and |             |             |                           |
| before      |             |             |                           |
| release of  |             |             |                           |
| code for    |             |             |                           |
| i           |             |             |                           |
| ntegration. |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| evaluates   |             |             |                           |
| the quality |             |             |                           |
| of CSC code |             |             |                           |
| and reviews |             |             |                           |
| contractor  |             |             |                           |
| readiness   |             |             |                           |
| to enter    |             |             |                           |
| integration |             |             |                           |
| testing.    |             |             |                           |
| This review |             |             |                           |
| examines:   |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
| Correctness |             |             |                           |
|     of code |             |             |                           |
|     i       |             |             |                           |
| nteractions |             |             |                           |
|     as      |             |             |                           |
|             |             |             |                           |
|   specified |             |             |                           |
|     in ICDs |             |             |                           |
|     and     |             |             |                           |
|     design  |             |             |                           |
|             |             |             |                           |
|   Interface |             |             |                           |
|             |             |             |                           |
| Agreements; |             |             |                           |
|             |             |             |                           |
| -   C       |             |             |                           |
| ompleteness |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| consistency |             |             |                           |
|     of CSC  |             |             |                           |
|     test    |             |             |                           |
|     results |             |             |                           |
|     with    |             |             |                           |
|     the     |             |             |                           |
|             |             |             |                           |
|   predicted |             |             |                           |
|     output; |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
| Correctness |             |             |                           |
|     of data |             |             |                           |
|     tra     |             |             |                           |
| nsformation |             |             |                           |
|     within  |             |             |                           |
|     the     |             |             |                           |
|     code;   |             |             |                           |
|             |             |             |                           |
| -   Ability |             |             |                           |
|     of code |             |             |                           |
|     to meet |             |             |                           |
|     timing, |             |             |                           |
|             |             |             |                           |
|   resource, |             |             |                           |
|     or      |             |             |                           |
|             |             |             |                           |
| performance |             |             |                           |
|     re      |             |             |                           |
| strictions; |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| -   Ability |             |             |                           |
|     of code |             |             |                           |
|     to deal |             |             |                           |
|     with    |             |             |                           |
|     error   |             |             |                           |
|             |             |             |                           |
| conditions, |             |             |                           |
|     out-o   |             |             |                           |
| f-tolerance |             |             |                           |
|             |             |             |                           |
|  processing |             |             |                           |
|             |             |             |                           |
| conditions, |             |             |                           |
|     and     |             |             |                           |
|     o       |             |             |                           |
| ut-of-limit |             |             |                           |
|     data    |             |             |                           |
|     values. |             |             |                           |
|             |             |             |                           |
| Following   |             |             |                           |
| the         |             |             |                           |
| Integration |             |             |                           |
| Test        |             |             |                           |
| Review,     |             |             |                           |
| software is |             |             |                           |
| presented   |             |             |                           |
| to the      |             |             |                           |
| integration |             |             |                           |
| test        |             |             |                           |
| o           |             |             |                           |
| rganization |             |             |                           |
| for review, |             |             |                           |
| evaluation, |             |             |                           |
| and         |             |             |                           |
| eventual    |             |             |                           |
| turnover.   |             |             |                           |
| Upon        |             |             |                           |
| acceptance, |             |             |                           |
| CSCI code   |             |             |                           |
| and         |             |             |                           |
| supporting  |             |             |                           |
| test data   |             |             |                           |
| are placed  |             |             |                           |
| under       |             |             |                           |
| formal      |             |             |                           |
| co          |             |             |                           |
| nfiguration |             |             |                           |
| control and |             |             |                           |
| the change  |             |             |                           |
| authority   |             |             |                           |
| is          |             |             |                           |
| transferred |             |             |                           |
| from the    |             |             |                           |
| developing  |             |             |                           |
| o           |             |             |                           |
| rganization |             |             |                           |
| to the      |             |             |                           |
| project     |             |             |                           |
| change      |             |             |                           |
| control     |             |             |                           |
| board.      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee CSC |             |             |                           |
| / CSCI      |             |             |                           |
| Integration |             |             |                           |
| and         |             |             |                           |
| Testing\    |             |             |                           |
| D-17**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Integrated  | Service     |                           |
| performs    | software    | Team        |                           |
|             | baseline    | software    |                           |
| Service     |             | specialist  |                           |
| Team        | Updated     |             |                           |
| oversees,   | Test        |             |                           |
| tracks, and | Development |             |                           |
| supports    | Folder(s),  |             |                           |
|             | including   |             |                           |
|             | integration |             |                           |
|             | test        |             |                           |
|             | results     |             |                           |
|             |             |             |                           |
|             | Updated     |             |                           |
|             | Software    |             |                           |
|             | Development |             |                           |
|             | Folder(s)   |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descr    |             |             |                           |
| iption]{.un |             |             |                           |
| derline}**: |             |             |                           |
|             |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| oversees,   |             |             |                           |
| tracks, and |             |             |                           |
| generally   |             |             |                           |
| supports    |             |             |                           |
| contractor  |             |             |                           |
| CSC / CSCI  |             |             |                           |
| integration |             |             |                           |
| and         |             |             |                           |
| testing.    |             |             |                           |
| CSC / CSCI  |             |             |                           |
| integration |             |             |                           |
| testing     |             |             |                           |
| assembles   |             |             |                           |
| partitioned |             |             |                           |
| CSU         |             |             |                           |
| components  |             |             |                           |
| into the    |             |             |                           |
| respective  |             |             |                           |
| CSC, and    |             |             |                           |
| those CSCs  |             |             |                           |
| into the    |             |             |                           |
| whole CSCI  |             |             |                           |
| that        |             |             |                           |
| represents  |             |             |                           |
| part or all |             |             |                           |
| of the      |             |             |                           |
| product     |             |             |                           |
| under       |             |             |                           |
| d           |             |             |                           |
| evelopment. |             |             |                           |
| The         |             |             |                           |
| integration |             |             |                           |
| of multiple |             |             |                           |
| CSUs into   |             |             |                           |
| the parent  |             |             |                           |
| CSC is an   |             |             |                           |
| iterative   |             |             |                           |
| process of  |             |             |                           |
| accepting   |             |             |                           |
| CSUs from   |             |             |                           |
| unit test,  |             |             |                           |
| testing     |             |             |                           |
| them        |             |             |                           |
| together,   |             |             |                           |
| correcting  |             |             |                           |
| any         |             |             |                           |
| de          |             |             |                           |
| ficiencies, |             |             |                           |
| and         |             |             |                           |
| repeating   |             |             |                           |
| the process |             |             |                           |
| with the    |             |             |                           |
| next CSU.   |             |             |                           |
| This        |             |             |                           |
| continues   |             |             |                           |
| until all   |             |             |                           |
| CSUs have   |             |             |                           |
| been        |             |             |                           |
| integrated  |             |             |                           |
| into the    |             |             |                           |
| parent CSC. |             |             |                           |
| Likewise,   |             |             |                           |
| CSCs are    |             |             |                           |
| then        |             |             |                           |
| integrated  |             |             |                           |
| and tested  |             |             |                           |
| s           |             |             |                           |
| equentially |             |             |                           |
| to form the |             |             |                           |
| CSCI. After |             |             |                           |
| completion  |             |             |                           |
| of CSC /    |             |             |                           |
| CSCI        |             |             |                           |
| integration |             |             |                           |
| testing,    |             |             |                           |
| software is |             |             |                           |
| turned over |             |             |                           |
| for CSCI    |             |             |                           |
| integration |             |             |                           |
| and         |             |             |                           |
| testing.    |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee     |             |             |                           |
| CSCI        |             |             |                           |
| Integration |             |             |                           |
| & Testing\  |             |             |                           |
| D-18**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Final       | Service     |                           |
| performs    | Software    | Team        |                           |
| integration | Design      | software    |                           |
| and testing | Document    | specialist  |                           |
|             |             |             |                           |
| Service     | Final       |             |                           |
| Team        | System Test |             |                           |
| oversees,   | Procedures  |             |                           |
| tracks, and |             |             |                           |
| supports    | Integrated  |             |                           |
|             | software    |             |                           |
|             | baseline    |             |                           |
|             | (tested)    |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| oversees,   |             |             |                           |
| tracks, and |             |             |                           |
| generally   |             |             |                           |
| supports    |             |             |                           |
| CSCI        |             |             |                           |
| integration |             |             |                           |
| and testing |             |             |                           |
| by the      |             |             |                           |
| contractor. |             |             |                           |
| Also known  |             |             |                           |
| as software |             |             |                           |
| build /     |             |             |                           |
| release     |             |             |                           |
| testing,    |             |             |                           |
| CSCI        |             |             |                           |
| integration |             |             |                           |
| and testing |             |             |                           |
| identifies  |             |             |                           |
| and         |             |             |                           |
| corrects    |             |             |                           |
| software    |             |             |                           |
| problems    |             |             |                           |
| through the |             |             |                           |
| execution   |             |             |                           |
| of system   |             |             |                           |
| test        |             |             |                           |
| procedures  |             |             |                           |
| before the  |             |             |                           |
| start of    |             |             |                           |
| Formal      |             |             |                           |
| Qu          |             |             |                           |
| alification |             |             |                           |
| Testing.    |             |             |                           |
| During this |             |             |                           |
| activity,   |             |             |                           |
| the         |             |             |                           |
| contractor: |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Executes |             |             |                           |
|     system  |             |             |                           |
|     test    |             |             |                           |
|             |             |             |                           |
|  procedures |             |             |                           |
|     to      |             |             |                           |
|     verify  |             |             |                           |
|     that    |             |             |                           |
|             |             |             |                           |
|   baselined |             |             |                           |
|             |             |             |                           |
|    software |             |             |                           |
|     meets   |             |             |                           |
|     system  |             |             |                           |
|     re      |             |             |                           |
| quirements; |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Executes |             |             |                           |
|     an      |             |             |                           |
|             |             |             |                           |
|   endurance |             |             |                           |
|     test to |             |             |                           |
|     monitor |             |             |                           |
|     system  |             |             |                           |
|             |             |             |                           |
| performance |             |             |                           |
|     under   |             |             |                           |
|             |             |             |                           |
|   sustained |             |             |                           |
|             |             |             |                           |
|    loading; |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Resolves |             |             |                           |
|     any     |             |             |                           |
|             |             |             |                           |
|    problems |             |             |                           |
|             |             |             |                           |
| encountered |             |             |                           |
|     during  |             |             |                           |
|             |             |             |                           |
|    testing; |             |             |                           |
|             |             |             |                           |
| -   Freezes |             |             |                           |
|     the     |             |             |                           |
|             |             |             |                           |
|   software; |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
|    Conducts |             |             |                           |
|     a dry   |             |             |                           |
|     run of  |             |             |                           |
|     all     |             |             |                           |
|     system  |             |             |                           |
|     test    |             |             |                           |
|             |             |             |                           |
|  procedures |             |             |                           |
|     prior   |             |             |                           |
|     to the  |             |             |                           |
|     Test    |             |             |                           |
|             |             |             |                           |
|   Readiness |             |             |                           |
|     Review. |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee     |             |             |                           |
| Hardware    |             |             |                           |
| Co          |             |             |                           |
| nfiguration |             |             |                           |
| Item        |             |             |                           |
| Integration |             |             |                           |
| and         |             |             |                           |
| Testing\    |             |             |                           |
| D-19**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Integrated  | Quality     |                           |
| integrates  | and tested  | Reliability |                           |
| and tests   | hardware    | Officer or  |                           |
|             | units       | designated  |                           |
| Service     |             | official    |                           |
| Team        |             |             |                           |
| oversees    |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descr    |             |             |                           |
| iption]{.un |             |             |                           |
| derline}**: |             |             |                           |
|             |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| tests each  |             |             |                           |
| Hardware    |             |             |                           |
| Co          |             |             |                           |
| nfiguration |             |             |                           |
| Item (HWCI) |             |             |                           |
| to confirm  |             |             |                           |
| each        |             |             |                           |
| satisfies   |             |             |                           |
| System      |             |             |                           |
| Sp          |             |             |                           |
| ecification |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| allocated   |             |             |                           |
| by the      |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| T           |             |             |                           |
| raceability |             |             |                           |
| Matrix. In  |             |             |                           |
| a similar   |             |             |                           |
| manner as   |             |             |                           |
| with CSCI   |             |             |                           |
| integration |             |             |                           |
| and         |             |             |                           |
| testing,    |             |             |                           |
| the         |             |             |                           |
| contractor  |             |             |                           |
| s           |             |             |                           |
| equentially |             |             |                           |
| links and   |             |             |                           |
| tests HWCIs |             |             |                           |
| to the      |             |             |                           |
| degree      |             |             |                           |
| practical.  |             |             |                           |
| When HWCI   |             |             |                           |
| interfaces  |             |             |                           |
| are         |             |             |                           |
| provided by |             |             |                           |
| software,   |             |             |                           |
| the         |             |             |                           |
| integration |             |             |                           |
| and testing |             |             |                           |
| of HWCIs    |             |             |                           |
| occurs      |             |             |                           |
| during      |             |             |                           |
| System      |             |             |                           |
| Integration |             |             |                           |
| and         |             |             |                           |
| Testing.    |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Oversee     |             |             |                           |
| System      |             |             |                           |
| Integration |             |             |                           |
| & Testing\  |             |             |                           |
| D-20**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Updated     | Service     |                           |
| performs    | Software    | Team        |                           |
| integration | Development | software    |                           |
| and testing | Folders     | specialist  |                           |
|             |             |             |                           |
| Service     |             |             |                           |
| Team        |             |             |                           |
| oversees,   |             |             |                           |
| tracks, and |             |             |                           |
| supports    |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descr    |             |             |                           |
| iption]{.un |             |             |                           |
| derline}**: |             |             |                           |
|             |             |             |                           |
| The Service |             |             |                           |
| Team        |             |             |                           |
| oversees,   |             |             |                           |
| tracks, and |             |             |                           |
| generally   |             |             |                           |
| supports    |             |             |                           |
| system      |             |             |                           |
| integration |             |             |                           |
| and testing |             |             |                           |
| by the      |             |             |                           |
| contractor. |             |             |                           |
| This        |             |             |                           |
| activity    |             |             |                           |
| integrates  |             |             |                           |
| CSCIs with  |             |             |                           |
| interfacing |             |             |                           |
| HWCIs and   |             |             |                           |
| CSCIs       |             |             |                           |
| (including  |             |             |                           |
| COTS /      |             |             |                           |
| NDI), tests |             |             |                           |
| the         |             |             |                           |
| groupings   |             |             |                           |
| to          |             |             |                           |
| determine   |             |             |                           |
| whether     |             |             |                           |
| they work   |             |             |                           |
| together as |             |             |                           |
| intended,   |             |             |                           |
| and         |             |             |                           |
| continues   |             |             |                           |
| the process |             |             |                           |
| until all   |             |             |                           |
| CSCIs and   |             |             |                           |
| HWCIs are   |             |             |                           |
| integrated  |             |             |                           |
| and tested. |             |             |                           |
| Softw       |             |             |                           |
| are-related |             |             |                           |
| information |             |             |                           |
| such as     |             |             |                           |
| analysis    |             |             |                           |
| and test    |             |             |                           |
| results is  |             |             |                           |
| recorded in |             |             |                           |
| appropriate |             |             |                           |
| Software    |             |             |                           |
| Development |             |             |                           |
| Folders.    |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Conduct     |             |             |                           |
| Test        |             |             |                           |
| Readiness   |             |             |                           |
| Review\     |             |             |                           |
| D-21**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Final       | Service     |                           |
| conducts    | System Test | Team test   |                           |
| review      | Procedures  | specialist  |                           |
|             |             |             |                           |
| Service     | Agreement   |             |                           |
| Team        | to proceed  |             |                           |
| evaluates   | with Formal |             |                           |
| results and | Qu          |             |                           |
| determines  | alification |             |                           |
| system      | Testing     |             |                           |
| readiness   |             |             |                           |
| for FQT     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descri   |             |             |                           |
| ption]{.und |             |             |                           |
| erline}**:\ |             |             |                           |
| \           |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| conducts a  |             |             |                           |
| System Test |             |             |                           |
| Readiness   |             |             |                           |
| Review      |             |             |                           |
| (TRR) in    |             |             |                           |
| accordance  |             |             |                           |
| with        |             |             |                           |
| MI          |             |             |                           |
| L-STD-1521B |             |             |                           |
| to provide  |             |             |                           |
| the Service |             |             |                           |
| Team with   |             |             |                           |
| documented  |             |             |                           |
| proof that  |             |             |                           |
| all         |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| necessary   |             |             |                           |
| to start    |             |             |                           |
| Formal      |             |             |                           |
| Qu          |             |             |                           |
| alification |             |             |                           |
| Testing     |             |             |                           |
| (FQT) have  |             |             |                           |
| been met.   |             |             |                           |
| The review  |             |             |                           |
| is          |             |             |                           |
| conducted   |             |             |                           |
| at the      |             |             |                           |
| software    |             |             |                           |
| unit and    |             |             |                           |
| component   |             |             |                           |
| level to    |             |             |                           |
| evaluate    |             |             |                           |
| the         |             |             |                           |
| c           |             |             |                           |
| ompleteness |             |             |                           |
| of test     |             |             |                           |
| procedures, |             |             |                           |
| and to      |             |             |                           |
| ensure the  |             |             |                           |
| contractor  |             |             |                           |
| is ready    |             |             |                           |
| for formal  |             |             |                           |
| testing.    |             |             |                           |
| Test        |             |             |                           |
| procedures  |             |             |                           |
| are         |             |             |                           |
| evaluated   |             |             |                           |
| for         |             |             |                           |
| compliance  |             |             |                           |
| with the    |             |             |                           |
| Master Test |             |             |                           |
| Plan. Test  |             |             |                           |
| plan        |             |             |                           |
| d           |             |             |                           |
| escriptions |             |             |                           |
| are         |             |             |                           |
| evaluated   |             |             |                           |
| for         |             |             |                           |
| adequacy in |             |             |                           |
| ac          |             |             |                           |
| complishing |             |             |                           |
| test        |             |             |                           |
| re          |             |             |                           |
| quirements. |             |             |                           |
| Preliminary |             |             |                           |
| testing is  |             |             |                           |
| reviewed to |             |             |                           |
| compare     |             |             |                           |
| results     |             |             |                           |
| against     |             |             |                           |
| baseline,   |             |             |                           |
| and to      |             |             |                           |
| ensure      |             |             |                           |
| necessary   |             |             |                           |
| coding      |             |             |                           |
| changes     |             |             |                           |
| have been   |             |             |                           |
| ac          |             |             |                           |
| complished. |             |             |                           |
| Informal    |             |             |                           |
| testing and |             |             |                           |
| any updates |             |             |                           |
| to          |             |             |                           |
| operational |             |             |                           |
| and support |             |             |                           |
| documents   |             |             |                           |
| are also    |             |             |                           |
| reviewed.   |             |             |                           |
|             |             |             |                           |
| At a        |             |             |                           |
| minimum,    |             |             |                           |
| the Service |             |             |                           |
| Team        |             |             |                           |
| evaluates   |             |             |                           |
| the         |             |             |                           |
| following   |             |             |                           |
| types of    |             |             |                           |
| data and    |             |             |                           |
| information |             |             |                           |
| to assess   |             |             |                           |
| the         |             |             |                           |
| readiness   |             |             |                           |
| of the      |             |             |                           |
| developed   |             |             |                           |
| system for  |             |             |                           |
| FQT:        |             |             |                           |
|             |             |             |                           |
| -   Results |             |             |                           |
|             |             |             |                           |
|    obtained |             |             |                           |
|     from    |             |             |                           |
|     the     |             |             |                           |
|     dry-run |             |             |                           |
|     of      |             |             |                           |
|     system  |             |             |                           |
|     test    |             |             |                           |
|             |             |             |                           |
| procedures; |             |             |                           |
|             |             |             |                           |
| -   Results |             |             |                           |
|     of      |             |             |                           |
|             |             |             |                           |
| lower-level |             |             |                           |
|             |             |             |                           |
|    hardware |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|    software |             |             |                           |
|     tests;  |             |             |                           |
|             |             |             |                           |
| -           |             |             |                           |
| Outstanding |             |             |                           |
|     Program |             |             |                           |
|     Trouble |             |             |                           |
|     Reports |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
|    waivers, |             |             |                           |
|     along   |             |             |                           |
|     with    |             |             |                           |
|             |             |             |                           |
|  assessment |             |             |                           |
|     of      |             |             |                           |
|     impact  |             |             |                           |
|     upon    |             |             |                           |
|     test    |             |             |                           |
|             |             |             |                           |
|  readiness; |             |             |                           |
|     and     |             |             |                           |
|             |             |             |                           |
| -   Formal  |             |             |                           |
|     reco    |             |             |                           |
| mmendations |             |             |                           |
|     by the  |             |             |                           |
|             |             |             |                           |
|    software |             |             |                           |
|     quality |             |             |                           |
|             |             |             |                           |
|   assurance |             |             |                           |
|     or      |             |             |                           |
| ganization. |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Witness     |             |             |                           |
| Design      |             |             |                           |
| Qu          |             |             |                           |
| alification |             |             |                           |
| Test\       |             |             |                           |
| D-22**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Program     | Service     |                           |
| conducts    | Trouble     | Team        |                           |
| DQT         | Reports     | Quality     |                           |
|             | (Resolved)  | Reliability |                           |
| Service     |             | Officer or  |                           |
| Team        | Software    | designated  |                           |
| witnesses   | Test Report | official    |                           |
| DQT and     |             |             |                           |
| accepts     | Updated     |             |                           |
| system on   | Software    |             |                           |
| completion  | Development |             |                           |
|             | Folder(s)   |             |                           |
|             |             |             |                           |
|             | Fully       |             |                           |
|             | qualified   |             |                           |
|             | system      |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descr    |             |             |                           |
| iption]{.un |             |             |                           |
| derline}**: |             |             |                           |
|             |             |             |                           |
| The Service |             |             |                           |
| Team and    |             |             |                           |
| quality     |             |             |                           |
| assurance   |             |             |                           |
| repr        |             |             |                           |
| esentatives |             |             |                           |
| witness the |             |             |                           |
| Design      |             |             |                           |
| Qu          |             |             |                           |
| alification |             |             |                           |
| Test (DQT)  |             |             |                           |
| conducted   |             |             |                           |
| by the      |             |             |                           |
| contractor. |             |             |                           |
| The DQT     |             |             |                           |
| d           |             |             |                           |
| emonstrates |             |             |                           |
| the         |             |             |                           |
| system's    |             |             |                           |
| ability to  |             |             |                           |
| satisfy all |             |             |                           |
| r           |             |             |                           |
| equirements |             |             |                           |
| of the      |             |             |                           |
| System /    |             |             |                           |
| Segment     |             |             |                           |
| Sp          |             |             |                           |
| ecification |             |             |                           |
| and         |             |             |                           |
| Interface   |             |             |                           |
| R           |             |             |                           |
| equirements |             |             |                           |
| Spe         |             |             |                           |
| cification. |             |             |                           |
| This        |             |             |                           |
| testing     |             |             |                           |
| involves    |             |             |                           |
| fully       |             |             |                           |
| integrated  |             |             |                           |
| CSCIs and   |             |             |                           |
| HWCIs, with |             |             |                           |
| the CSCIs   |             |             |                           |
| installed   |             |             |                           |
| on the      |             |             |                           |
| approved    |             |             |                           |
| hardware    |             |             |                           |
| con         |             |             |                           |
| figuration. |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| executes    |             |             |                           |
| all test    |             |             |                           |
| scripts     |             |             |                           |
| from        |             |             |                           |
| approved    |             |             |                           |
| System Test |             |             |                           |
| Procedures, |             |             |                           |
| and         |             |             |                           |
| evaluates   |             |             |                           |
| results     |             |             |                           |
| against the |             |             |                           |
| acceptance  |             |             |                           |
| criteria    |             |             |                           |
| included in |             |             |                           |
| test        |             |             |                           |
| doc         |             |             |                           |
| umentation. |             |             |                           |
| All Program |             |             |                           |
| Trouble     |             |             |                           |
| Reports     |             |             |                           |
| opened      |             |             |                           |
| during DQT  |             |             |                           |
| must be     |             |             |                           |
| resolved    |             |             |                           |
| before the  |             |             |                           |
| conclusion  |             |             |                           |
| of testing. |             |             |                           |
| Successful  |             |             |                           |
| completion  |             |             |                           |
| of the DQT  |             |             |                           |
| marks the   |             |             |                           |
| end of      |             |             |                           |
| de          |             |             |                           |
| velopmental |             |             |                           |
| testing and |             |             |                           |
| transfer of |             |             |                           |
| ownership   |             |             |                           |
| of the      |             |             |                           |
| system to   |             |             |                           |
| the FAA.    |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **Activity: |             |             |                           |
| Conduct     |             |             |                           |
| Production  |             |             |                           |
| Acceptance  |             |             |                           |
| Test\       |             |             |                           |
| Each Unit\  |             |             |                           |
| D-23**      |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **R         | **Product** | **Approval\ | **Tools and Aids**        |
| esponsible\ |             | Authority** |                           |
| Agent**     |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| Contractor  | Accepted    | Quality     |                           |
| conducts    | production  | Reliability |                           |
| testing     | units       | Officer or  |                           |
|             |             | designated  |                           |
| Service     |             | official    |                           |
| Team        |             |             |                           |
| monitors    |             |             |                           |
|             |             |             |                           |
|             |             |             |                           |
+-------------+-------------+-------------+---------------------------+
| **[Descr    |             |             |                           |
| iption]{.un |             |             |                           |
| derline}**: |             |             |                           |
|             |             |             |                           |
| The         |             |             |                           |
| contractor  |             |             |                           |
| tests each  |             |             |                           |
| production  |             |             |                           |
| unit using  |             |             |                           |
| test        |             |             |                           |
| procedures  |             |             |                           |
| approved by |             |             |                           |
| the FAA to  |             |             |                           |
| verify the  |             |             |                           |
| assembly    |             |             |                           |
| line is     |             |             |                           |
| producing   |             |             |                           |
| units with  |             |             |                           |
| the same    |             |             |                           |
| quality and |             |             |                           |
| performance |             |             |                           |
| as the      |             |             |                           |
| first       |             |             |                           |
| article.    |             |             |                           |
| The Quality |             |             |                           |
| Reliability |             |             |                           |
| Officer     |             |             |                           |
| (QRO)       |             |             |                           |
| monitors    |             |             |                           |
| production  |             |             |                           |
| acceptance  |             |             |                           |
| testing,    |             |             |                           |
| and is the  |             |             |                           |
| approving   |             |             |                           |
| official    |             |             |                           |
| for         |             |             |                           |
| successful  |             |             |                           |
| compliance  |             |             |                           |
| with test   |             |             |                           |
| procedures  |             |             |                           |
| and test    |             |             |                           |
| criteria.   |             |             |                           |
+-------------+-------------+-------------+---------------------------+