# Presentation: 432239

## Supplier Assessments

- NASA / GSFC

- Charlie Robinson / NCAS
- Quality Assurance and Risk Management Services, Inc.
- 28-30 October 2008

- Goddard Space Flight Center - Greenbelt, MD

## AGENDA

**Preparing for an Assessment**
**RESULTS**
**Using Historical Data**
**LESSONS-LEARNED**
**Document Control**
**Contracts**
**Software**
**Internal Audits**
**SUMMARY**

**AGENDA**

## Preparing for an Assessment

- General
- Information

- AS9100
- Assignments

- Assessment
- Plan

- Blank
- Assessment Card

- Assessors Input
- Template

- E-mails to
- Team & NASA Rep.

- Evaluation
- Form

- In-Briefing

- Attendance
- Sheet

**Preparing for an Assessment**

**The Assessment Team Takes Pre-Assessment Planning Seriously**

- Team Conference Call

## Using Historical Data

- NCAS –  55% (18 out of 33) have Software problems (NOV 06 – SEP 08)
- SAC   –  89%  (32 out of 36) had   Software problems (FEB  02 - JUN 05)
- 10 Suppliers have been assessed 2 to 4 times since FEB 2002

**Using Historical Data**

**Notes:**
Item Codes
AI = Internal Audits
AT = Acceptance Testing
C = Management & Contracts  
CA = Calibration
CM = Configuration Management 
CP = Corrective/Prevent Action
CR = Clean Room
DA = Data Analysis 
DC = Document Control 
DP = DPAS
ED = Engineering Design (not software)
ES = Electrostatic Discharge
FA = First Article
GF = Government Furnished Property
GP = GIDEP
MC = Material Control
NC = Nonconforming material
O = Operations
P = Purchasing & Supplier Control
Q = Quality
R = Records
RM = Risk Management
S = Specifications, old
SI = Safety, Industrial
SP = Processes, Special
SR = System Reliability
SS = System Safety
ST = Security
T = Training
WI = Work Instructions

## Using Historical Data (continued)

**HIGH**

**LOW**

- R
- I
- S
- K

- Items produced & delivered before identifying the Environment & Contamination  Requirements
- Contractually required Stress Relieve Procedure not developed.
- Unaware of DPAS rating
- Wiring ‘Crimped” – Contract requires “Welding”

**Identify Requirements**

**Notes:**
Large > 1,000
Medium 200-750
Small < 200

This presentation is based on the results of assessments of suppliers that support NASA/GSFC.

## Using Historical Data (continued)

**LOW**

- R
- I
- S
- K

- No Configuration Management Process
- Amendment to Instrument Coding Standard made without Peer-Review
- Life-cycle metrics not distributed as required by contract
- No Software Safety Program
- Coding Standards in SDP not used.
- A “Draft” Satellite Simulator Software is being used.

**Software**

**Using Historical Data ****(continued)**

**Notes:**
Large > 1,000
Medium 200-750
Small < 200

This presentation is based on the results of assessments of suppliers that support NASA/GSFC.

## Lessons-Learned

**Document Control**

- 47 Non-compliances & 37 Observations

- Internal Conflicts between procedures
- Matrix of procedure inter-relationships is a way to avoid these conflicts
- Industry Specifications not current
- A consolidated listing of all industry specifications that are maintained
- Procedures not maintained to reflect current practices
- Supervisor responsibility & Internal Audits should provide direction and assurance that procedures are current

## Lessons-Learned

**Contracts**

- 46 Non-compliances & 45 Observations

- There is not a reliable internal flow-down of requirements
- Clean-room, DPAS, Calibration, QMS, Record Control, etc.
- The review is flawed as suppliers agree to requirements they do not meet.
- Suppliers often do not know their basic system and therefore are not able to identify those contractual requirements that they do not satisfy.
- CDRL Items are either late or revised without updating the customer.
- Assure that a single person is responsible for tracking the status of all CDRLs

## Lessons-Learned

**Software**

- 38 Non-compliances & 24 Observations

- The software controls are not all encompassing
- Control of software includes deliverable, design, test, and operational software.
- There is not a compliant Software Safety Program (SSP)
- Evaluate the SSP against NASA-Std-2100
- The individual who developed the software integration are developing the integration test.
- The need for objective independence is compromised by having the individual who developed the software integration also develop the integration test.

**Notes:**
NASA-STD-8719.13A – Software Safety, NASA Technical Standard

NASA-STD-8739.8 – Software Assurance Standard

NASA-STD-2100-91 - NASA Software Documentation Standard

MIL-STD-882C - System Safety Program Requirements

NASA-STD-2201 – Software Assurance Standard
Software safety program shall ensure that: 
Safety-related deficiencies in specifications and design are identified and corrected. 
Software design incorporates positive measures to enhance the safety of the system.
Software safety is included as an agenda item for formal reviews. 
The software safety process shall include the following activities: 
Determining the safety criticality for each software component. 
Analyzing consistency, completeness, correctness, and testability of safety requirements. 
Analysis of design and code to ensure that they correctly implement safety-critical requirements. 

## Lessons-Learned

**Internal Audits**

- 18 Non-compliances & 14 Observations

- Contractual compliance and procedural support are not always part of the internal audit process.
- Without auditing the contractual requirements the management will believe the Quality Management System is appropriate when in fact it does not support contractual requirements.
- Internal Audits are behind schedule.
- This is a discipline that is necessary to avoid requirement creep or slippage.  As Dwight D Eisenhower said “The uninspected inevitably deteriorate.”
- The internal auditors do not audit or are not qualified to audit special processes.
- Assure that subject mater experts are used when auditing software, soldering, heat treat, plating, etc.

## Summary

- Individual commitment to a group effort - that is what makes a
- team work,
**company work**,
- society work,
- civilization work.
- _[Vince Lombardi](http://www.brainyquote.com/quotes/quotes/v/vincelomba129818.html)_