# Presentation: 794725

## NASA/GSFC Safety & Mission Assurance Directorate -A Value-Added Organization 

**NASA/GSFC Safety & Mission Assurance Directorate -****A Value-Added Organization **

**February 5, 2009**

**Presented by Mike Kelly, Institutional Support Office, Chief**

- 8

- February 3, 2009

## AGENDA

**Safety and Mission Assurance Directorate Organization Charts**** **
**How Code 300 Organization Interacts with a GSFC Projects**
- Code 323, Mission Assurance Branch
- Code 324, Institutional Assurance Branch
- NASA/GSFC Mission Assurance Approach
- Standard MAR
- Support at Suppliers
- Software Assurance

- February 3, 2009

## Agenda (Con’t)

**Code 301, System Review Office**** **
**Code 302, Institutional Support Office**
- Supply Chain Management
- Introduction to AS9100 Class at GSFC
- Lead Auditor Class at GSFC
- Code 300 Orientation Program
- Code 300 Education Series
- GOLD Rules
- Supplier Conference at GSFC
- Internal Audit Training Program
- Mission Operations Assurance

- February 3, 2009

## Agenda (Con’t)

**Code 300 EEE Parts/Workmanship Group**
- GIDEP Program
- Workmanship Standards/Training
- ESD Certification Program
**Code 321, System Safety Branch – Safety Program**
- Typical Safety Deliverables
**Code 322, Reliability and Risk Analysis Branch - Reliability and Risk Management  Program**
- Typical Reliability Deliverables
**Code 305, Resource Analysis Office**** **
**Presenter’s Lessons Learned**

- February 3, 2009

## GSFC Organization Chart *November 2008

- GSFC Organization Chart *November 2008

- Safety & Mission Assurance Directorate

- * Safety & Mission Assurance Directorate, Deputy was added.

**M. So**

- February 3, 2009

## Safety and Mission Assurance Directorate (Code 300) *(Draft, 02/02/09)

- * Principal Engineer changed to “Vacant”

**(Vacant)**

**(Vacant)**

- February 3, 2009

## Actual Manpower Numbers

- GSFC Greenbelt:  2957 civil servants
- GSFC: Greenbelt  5706 contractors supporting the civil servants (FY05)
- GSFC Safety & Mission Assurance Directorate:  207 total distributed as follows:
- 100 contractors
- 107 civil servants
- 87 permanent
- 18 term
- 2 co-op
- Approximately 100 contractors total from Mantech/SRS and Honeywell

- February 3, 2009

## Codes 323/324: Mission Assurance/Institutional Assurance Branches

- Name:		Bob Savage/Rob Sticka
- Title: 		Branch Head
- Office 		W120/W126D
- Tel: 		68840/69900
- Email:   _[Robert.M.Savage@nasa.gov](mailto:Robert.M.Savage@nasa.gov)_ / Rob.Sticka@nasa.gov

- February 3, 2009

**Notes:**
I am the CKO here at Goddard, but my task is to help Goddard be a learning organization.

## How Code 300 Organization Interacts With GSFC Projects

- QUALITY

- ENGINEERING

- TEAM (HW/SW)

- Code 320

- SYSTEM SAFETY

- ENGINEERING

- Code 321

- RELIABILITY

- ENGINEERING

- Code 322

- Code 320

- MATERIALS

- ENGINEERING

- Code 541

- EEE PARTS/

- RADIATION

- ENGINEERING

- Code 562

- Code 561

- CONTAMINATION

- Code

- 546

- RESIDENT OFFICE

- SUPPORT AT

- SUPPLIERS’

- FACILITIES

- PROGRAM

- MANAGER

- PROJECT

- MANAGER

**Chief  Safety & Mission Assurance Officer **

** **

**(CSO)**

** **

** **

** **

- February 3, 2009

## Chief Safety and Mission Assurance Officers (CSO, Code 323, Mission Assurance Branch) 

- Robert Savage, (Branch Chief)
- Projects:
- POES, Sam Archer-Davies
- GOES NOP, Dave Bogart
- SAM, Roger Counts
- GLORY, Jack Ellis
- LDCM, RSDO, Patty Huber
- GLAST, James Lohr
- NPP, Tim Bowser
- TDRSS K, David Smalts

- February 3, 2009

## Chief Safety and Mission Assurance Officers (CSO, Code 324, Institutional Assurance Branch) 

- Rob Sticka, (Branch Chief)
- Projects:
- JWST ISIM, Sue Aleman
- MMS, John Blackwood
- IBEX/RBSP, Robert Calvo
- SDO, Oscar Cheatom
- ELC, Joseph Hall
- LRO, Ronald Kolecki
- LRO/SMAP, Lydia Lee
- HST, Lynette Marbley
- GPM, John Rauscher
- HST, Renee Robinson
- JWST, Joe Radich
- S/GN/SNE/NIMO/SCiP, Thomas Toutsi

- February 3, 2009

## Organization Functions

- Chief Safety and Mission Assurance OfficerCSO

- CSOs assigned to Projects (formerly known as SAMs and FAMs before that)
- Co-located with Projects
- Reports to Project Manager (dotted line)
- Assurance program includes Quality Assurance, Safety, Reliability, Workmanship, Risk Management, Parts, Materials
- Reports independently back to Code 300
- Works Project full life-cycle from Concept through Launch
- Manages assurance program for both in-house and out-of-house Projects
- Generates and implements Mission Assurance Requirements (MAR)

- February 3, 2009

## Organization Functions

- CSOs cont’d

- Lead for Problem Report/Problem Failure Report (PR/PFR) System
- Lead for Work Order Authorization (WOA) implementation (IAB)
- Typically the Project’s ISO 9001 Implementation Manager
- Responsible for manufacturing and QA oversight of Project contractors by utilizing:
- Defense Contracts Management Agency (DCMA)
- NASA Contractor Assurance Services (NCAS)
- Code 300 Support Contractors
- Works with Systems Safety Engineers to implement project safety program
- Works with Reliability engineering to implement project reliability program

- February 3, 2009

## Organization Functions

- CSOs cont’d

- Member of Parts Control Board. Works closely with Code 562 Parts Engineers.
- Implements Government-Industry Data Exchange Program (GIDEP) compliance and dispositions
- Works with Code 541 Materials to determine acceptability of printed wiring boards by coupon evaluation
- Ensures parts and materials lists are thoroughly reviewed and acceptable for use.
- Coordinates radiation requirements and implementation with Code 561 (Radiation Effects)
- Implements Workmanship Standards such as soldering, cabling, harnessing, conformal coating

- February 3, 2009

## NASA/GSFC Mission Assurance Approach

- NASA Chief Safety and Mission Assurance Officer (CSO) is the program/project focal point and is responsible for supporting the Goddard missions from an End-to-End Perspective which includes Procurement Activities through On-Orbit Operations.
- CSO has an independent reporting chain to the GSFC Center Director.
- The Mission Assurance Team supports the Program and Project Offices in their daily operations.  However, if there are conflicting opinions it is the CSO’s responsibility to report those disagreements to NASA management.

- (CONTINUED)

- February 3, 2009

## NASA/GSFC Mission Assurance Approach

- Generally the CSO is co-located with the project office, to provide the most efficient access to the project manager and his staff.  It is desirable to have safety and reliability personnel co-located there as well.
- CSO must be a good communicator and understand where support is needed and keep the Project in the loop.
- CSO walks a fine line between supporting the Project and remaining an independent entity.

- (CONTINUED)

- February 3, 2009

## Chief Safety and Mission Assurance Officers (CSO, Code 323, 324) 

**CSO duties in support of the Project are as follows:**
- Voting member of CCB and risk management board
- Conduct audits/assessments at hardware developers (and provide follow-up).  Responsible for determining mandatory inspection points
- Support in resolution of hardware/software problems
- Member of Source Evaluation Boards
- Member of Senior Staff
- Interface for all Printed Wiring Board (PWB) coupons
- Point of contact for all manpower in Code 300
- Ensure LOD and LOA (task order) are written and followed to support the project.  All task orders are in the Task Order Management System (TOMS).
- Attendance and participation at all major reviews
- Provide monthly presentations to Code 300 Management
- Provide presentations to Project/Program Management as required
- Development of Mission Assurance Requirements
- Present Safety and Mission Success Review to Headquarters

- February 3, 2009

## Standard MAR

- In the recent past, the CSO used as a guide the _Mission Assurance Guidelines (MAG) Procedure (300-PG-7120.2.2)_ and consultation with functional disciplines in Codes 301, 302, 320 and other GSFC organizations to develop the MAR for the Instrument, Spacecraft, and Ground System
- More recently, a “Standard MAR” began development.  This is to be used as a tailoring tool to generate project MARs.
- The Standard MAR, including the requirements narrative and DIDs, are available on the Agency's PBMA web site in a community work area called "Goddard Mission Assurance Guide.“
- The current versions of the individual components are available on the web site and are not expected to undergo further revisions.
- A controlled version of the Standard MAR has been prepared and is under review in anticipation of being placed under CM in the near future.  A new Code 320 PG is being developed to establish relevant procedures and processes for its use.

- February 3, 2009

## Support at Suppliers 

- The work activities performed by the developer and/or his suppliers are subject to evaluation and audit by government-designated representatives.
- CSO supports project by selecting on-site supplier representative’s by one of several methods:
- (1) a Defense Contract Management Agency (DCMA) person via a Letter Of Delegation (LOD),
- (2) an independent assurance contractor (IAC) via a contract
- NASA Contract Assurance Services (NCAS)
- Code 300 Mission Assurance Support Contract (MASC)
- DCMA and NCAS contract employees:
- Advantage: Funded by NASA HQ, not by the GSFC Program/Project budget
- Disadvantage – Usually working several projects at one time
- MASC contract employees:
- Advantage  -  Usually works exclusively on your project
- Disadvantage  -  Costs are directly to the GSFC Program/Project budget

- February 3, 2009

## Software Assurance

- Our primary objective is to assess program  / project products and processes to
- assure that programmatic capabilities are achieved.
- Software Assurance shall apply to flight and ground system software developed by or for GSFC.
- Government off-the-shelf (GOTS) software
- Modified off-the-shelf (MOTS) software
- Commercial off-the-shelf (COTS) software
**Overview**
- Software assurance comprises a set of disciplines that strive to improve the overall quality of the product/software while employing risk mitigation techniques.
- Software Quality	    		Software Safety		Software Reliability
- Verification and Validation (V&V)
- Independent Verification and Validation (IV&V).
**SW Quality Assurance Functions:**
- assures that the standards, processes, and procedures are appropriate for the project and correctly implemented,
- assures adherence to those software requirements, plans, procedures and standards,
- shall plan and conduct process and product assurance activities throughout the project development life cycle.
- Assures compliance to Center Software Process Improvement (SPI) initiatives and performs Product Process Quality Assurance (PPQA) assessments.

- February 3, 2009

## Goddard Review ProcessSystem Review Office, Code 301

- Name:  Mark Goans
- Title:  Office Chief
- Tel:  301-286-9763
- Email: Mark.D.Goans@nasa.gov

- February 3, 2009

## Independent Review Process

- The Systems Review Office (SRO) is the implementation arm of the GSFC independent review process.
- Types of Independent Reviews
- Mission Life-Cycle Reviews conducted by Standing Review Board (SRB) / HQ Driven
- Reference:  NPR 7120.5D NASA Space Flight Program and Project Management Requirements
- GSFC Level Independent Reviews conducted by a SRO chartered Review Team
- Reference:  GPR 8700.4F Integrated Independent Reviews
- Engineering Peer Reviews conducted by an independent peer review team
- Reference:  GPR 8700.6A Engineering Peer Reviews

- February 3, 2009

## Project Life-Cycle and Reviews

**SMSR**

- February 3, 2009

## Center Level Independent Reviews (1 of 2)

- Center Level Independent Reviews comprise life cycle reviews for the Spacecraft(s), Instrument(s), Ground System(s) and Operations.
- For larger projects dozens of reviews may be conducted
- The **SRO convenes review teams** to conduct Center Level Independent Reviews
- For each project, the **SRO assigns a Systems Review Manager  (SRM)** to serve as the review team chair.
- The SRM **develops a Systems Review Plan** in conjunction with the Project that appropriately tailors the GSFC process to the mission needs.
- For each element the SRM **establishes an appropriate independent review team** with members chosen for their management and technical expertise
- The **SRM presides at each review** and ensures compliance with center-level processes.

- February 3, 2009

## Center Level Independent Reviews (2 of 2)

- The review team evaluates the project based on compliance with the review objectives and adherence to Key Project Management Practices
**Formal Requests for Action** or additional information are generated as needed
- The **review team caucuses and out briefs the project** at the conclusion of the review
- The **SRM provides a report** to the Project documenting the review results and makes **appropriate recommendations to the GSFC Center Management Council**
- The **SRM provides feedback the to mission SRB (HQ team)** regarding key results from Center Level Reviews

- February 3, 2009

## Engineering Peer Reviews

- Each GSFC flight project is required to develop an Engineering Peer Review Plan
- Engineering Peer Reviews (EPRs) are conducted for spacecraft subsystem, instrument component, software and crosscutting functional elements.
- The project manager (PM) appoints an independent EPR chairperson for the various elements.
- For each element, the EPR chairperson recruits independent review team members based on their technical knowledge and practical experience.
- For each review the EPR chairperson provides a report with findings to the PM and the assigned SRM
- Engineering Peer Review Results are summarized at the next schedule Center Level Independent Review and/or Mission Life-cycle Review

- February 3, 2009

## Institutional Support Office,Code 302

- Name:  Mike Kelly
- Title:  Office Chief
- Tel:  301-286-0662
- Email: Michael.P.Kelly@nasa.gov

- February 3, 2009

## Code 302

**SMA Configuration Mgmt**
- *Rhonda Weaver/CSC

**Michael P. Kelly**
**Office Chief**
- Regina MartinAdministrative Assistant

**Mission Ops Support/ Anomaly Management**
- James La
- *Jim Suraci/SRS
- *Nicole Smith/SRS

**Environmental Test Verif & Problem Reporting Analysis**
- Scotty Milne

**NASA/HQ OCE Support**
- G.S. Krishnan (detail to HQ)

**GSFC Management System / Internal Audit**
- Lead (VACANT)
- Darryl Younger (Detail from 500)
- *Mark Bollard/Honeywell
- *Linda Emerson/Honeywell
- Sue Aleman

- Institutional Support Office 1

**Lead Systems Engineer**
- *Tom Clifford/SRS

- Draft 01/16/09

**GOLD Rules Management**
- Jerry Kosko

- *Contractor

**Supply Chain Management**
- Louis Thomas
- Charles Kim
- Jonathan Root
- *V. DiMarco/SRS

- (02/02/09 tc)

- 1  Added an internal auditor

- February 3, 2009

## Organization Functions

- Conducts Supplier assessments
- Maintains Records of assessments in GSFC audit database
- Sponsors Quality training (e.g. AS9100 quality system, ISO Lead Auditor)
- Sponsors suppliers conferences
- Is Technical Liaison for NASA Contract Assurance Services (NCAS)
- Is Focal Point for Defense Contract Management Agency (DCMA)
- Working with NASA Assurance Management Team (NAMT) – all NASA Centers Participation
- Working with Joint Audit Planning Committee (JAPC) – Primes and other Government Agencies (NASA, MDA, NRO, and DCMA)

- February 3, 2009

## The Assessment Approach/Process

**NASA Goddard Supply Chain Manager has a large role in the planning of the assessment in order to work issues/concerns upfront**
- He is calling supplier’s to set up the assessments (not NCAS)
- He is conducting the in-brief when possible to set the proper tone for both the assessment team and the supplier
- He is attending each out-brief (sometimes remotely)
**Draft copy of the Supplier’s Assessment Plan is forwarded to the Supplier for their comments and feedback to ensure agreements are reached prior to the assessment**
**No scoring is used during the assessment process**
- Only non-compliances, observations, & commendations and
- A final out-brief package is left with the supplier at the end of
-     the assessment

- (CONTINUED)

- February 3, 2009

## The Assessment Approach/Process (con’t)

- A final report is written and forwarded to the supplier Point of Contact for comment
- This report will be a few pages long and will contain the assessment cards and the final out-brief package
- NASA/GSFC provides a “Supplier Assessment Team Evaluation Survey Form” to solicit both positive and negative comments about the assessment process and the participation of each assessor
- NASA/GSFC does care about the Corrective Actions and wants to work with each supplier to support Closure of each one.
- Plan to conduct follow-up assessments if necessary and/or if requested by the supplier

- (CONTINUED)

- February 3, 2009

## Assessment Objectives

- Assess the supplier’s processes for compliance to:
- the requirements of ISO9001:2000 or AS9100, (if supplier is third party certified, we will assess the supplier to it.)
- to the applicable NASA Contractual Requirements, and
- to the requirements of the internal Quality Management System.
- Follow up on previous NASA assessments
- The goal of each assessment is to identify strengths and areas for improvement.

- February 3, 2009

## Assessment Reporting

- Assessment Team Members will document closed and outstanding non-compliances & observations during the course of the assessment as well as note any observed commendations
- _Critical Noncompliance_:	Failure to follow requirements that could lead to loss of life, serious injury to personnel, or damage to high-value equipment.
- _Noncompliance_:	Failure to comply with Federal, State, local, Agency, or Center requirements that would not have the impact of a Critical Noncompliance
- _Observation_:	A condition that is not contrary to documented requirements, but, in the judgment of the assessor warrants improvement or clarification.
- _Commendation_:	A process that is considered an industry benchmark by the assessor.
- Daily debrief will entail informal discussions of the day’s activities.
- Draft copies of Corrective Action Reviews will be provided at the Out-briefing.
- A formal report will be provided within 20 working days after the assessment.

- February 3, 2009

## Sample Assessment Plan “items to be reviewed”

** ****The following list provides an outline of some of the topics the assessment team will review:  **
**Flowdown of contractual requirements**
**Receiving inspection**
**Configuration Management / Change Control**
**Packaging**
**Handling**
**Parts sampling, selection, and traceability **
**Training and Certification of operators/inspectors/disposition authorities/testers**
**Process documentation adequacy (work orders, shop aids, drawings, etc.)**
**Document control**
**Workmanship and inspection**
**Travelers, routers and configuration recording**
**Nonconforming product control**
**Scrap control**
**Rework and repair processes**
**Acceptance Data Packages**
**Problem Reporting System**
**Internal Audit**

**Calibration**
**GFE**
**Industrial Safety**
**System Safety**
**GIDEP**

- February 3, 2009

**Notes:**
Technical (T, L3/C3) – An IBEX Software Reliability Plan had not been developed nor was the software reliability prediction calculated for risk(03); A waiver was submitted for rework that should have been an MRB action(07); Missing and/or lost test & measuring devices(08); A measuring device overdue for calibration was being used on an engineering development project(09);An ESD certified work station was not being maintained in a certified condition, but still contained a certification label(10).
Quality (Q, L3/C3) – There was no evidence that the GSFC SAM and SSM had reviewed and approved the IBEX MPAIP, PAIP and SSPP; the MPAIP did not require GSFC approval for PWB coupon evaluations(01); The IBEX Quality Management System was not fully compliant to ISO Q9001 i.e., Continual Improvement, Customer Satisfaction and Analysis of Data were not being measured or treated as required by ISO(02); The CM procedure lacked detail and referenced obsolete specifications(05); Product Assurance training for Division 15 was not being performed in accordance with the training procedure(06); There was not a continual process (only semi-annual evaluation) for addressing suppliers who fell below the 80% rating(11).
Schedule (S, L1/C1) – No known issues 

Industrial Safety (IS, L1/C2) – No known issues.

## The following chart is a sample assessment “One-Pager” that is presented to Code 300 management after each assessment.

- Management One-Pager

- February 3, 2009

## NCAS Supplier Assessment SummarySupplier XYZCity, State September 18-20, 2007

** ****Products**
- Developer of processors, command and data handling systems,
-  	and related electronics for space flight.  Expertise in planetary
- 	and space science instrument design and fabrication and data
-  	systems development.
**Centers/projects which may be impacted (if known)**
- GSFC Project A, JSC Project B, JPL Project C
**Summary of Issues Identified**
- No evidence that SAM and/or SSM had reviewed & approved PAIPs and Safety Plan.
- GSFC approval for PWB coupon evaluations not required by Mission PAIP.
- Missing and/or lost test & measuring devices.
- Software Reliability plan not developed, and S/W risk prediction not calculated.
- QMS not fully compliant with ISO Q9001,i.e., Continual Improvement, Customer Satisfaction, Analysis of Data, Management Responsibility.
- No continual process (semi-annual only) for addressing vendors falling below 80% rating.
- Training not performed in accordance with developed training procedure.

- February 3, 2009

## Supplier Assessments - Non-Code 302

| SUPPLIER  NAME | CITY, STATE | DATE(S) | By |
| --- | --- | --- | --- |

| Supplier A | Redmond WA | 11/16/06 | JPL |
| --- | --- | --- | --- |
| Supplier B | Goleta, CA | 11/09/06 | JPL |
| Supplier C | Palo Alto, CA | 10/04/06 | GSFC-OSSMA |
| Supplier D | Deerfield Beach, FL | 06/28/06 | JPL |
| Supplier E | Goleta, CA | 10/05 - 10/06/2005 | JPL |
| Supplier F | Baltimore, MD | 09/15/05 | GSFC-OSSMA |
| Supplier G | Glen Burnie, MD | 08/25/05 | GSFC-OSSMA |
| Supplier H | Hagerstown, MD | 08/24/05 | GSFC-OSSMA |
| Supplier I | Lorton, VA | 08/17/05 | GSFC-OSSMA |
| Supplier J | Upper Marlboro, MD | 08/12/05 | GSFC-OSSMA |
| Supplier K | Baltimore, MD | 08/09/05 | GSFC-OSSMA |

- February 3, 2009

## How to Obtain all NASA/GSFC Assessments Reports

- February 3, 2009

## GSFC Audit Database Link

- http://auditsystem.gsfc.nasa.gov

**Civil Servants Click here.  **
**Non-civil-servants do not have access to Supplier records.**

- February 3, 2009

## Introduction to AS9100 Class at GSFC

- Civil servants and contractors are invited to attend a class presenting the requirements of Aerospace Standard AS9100 and ISO 9001/2000.
- The class is sponsored by Mike Kelly, Supply Chain Manager for the Safety and Mission Assurance Directorate.
- Presented by DCMA Headquarters Representatives, Gil Kimbrough and James Rodden
- The course is an in-depth overview of requirements, organization, structure, and use of the Quality Management Standard (agenda is available per request). Contact (301) 286-4333, _[ckim@mscmail.gsfc.nasa.gov](mailto:ckim@mscmail.gsfc.nasa.gov)__ _
- The objective is to give the student a working knowledge and auditing skills of the International Organization for Standardization (ISO) Quality Management Standard (QMS) for the year 2008 and AS9100 Aerospace requirements.
- The room can accommodate 25 to 30 persons and spaces will be reserved on a first-come, first-served basis.
- Where:  Goddard Spaceflight Center or vicinity.
- When:   **Eighth class is scheduled for February 17-19 2009, Tuesday through Thursday.  Class starts at 8am.**

- (01/22/09)

- February 3, 2009

## Lead Auditor Class at GSFC

- Civil servants and contractors are invited to attend a class ....
- The class is sponsored by Mike Kelly, Supply Chain Manager for the Safety and Mission Assurance Directorate.
- Presented by Plexus Corporation
- The course is an in-depth overview of requirements, organization, structure, and use of the Quality Management Standard (agenda is available per request). Contact (301) 286-4333, _[ckim@mscmail.gsfc.nasa.gov](mailto:ckim@mscmail.gsfc.nasa.gov)_
- The course provides either new or current aerospace auditors with an understanding of the AS9100 standard, the process approach of ISO 9001:200, and how the process approach impacts auditing practices.  The course includes a simulated audit to develop and refine newly acquired knowledge and skills including the application and evaluation of a process based quality management system.  A successful completion of Plexus Corporation’s program meets the training requirements for certification of individual QMS Auditors and Aerospace Auditors.
- The class can accommodate 10 persons.
- Where:  Goddard Spaceflight Center or vicinity.
- First class occurred:   May 19 - 23, 2008
- Another class is planned for all Directorates personnel.  _***Date is TBD.***_

- February 3, 2009

## Code 300 Orientation Program

| TIME | TOPIC | WHO |
| --- | --- | --- |
| 8:30-8:45 | Welcome; overview of 300; Introduce the staff in 300 Office | Marcus Watkins |
| 8:45-8:55 | RAO | Cindy Fryer |
| 8:55-9:05 | Resource Office | Gale Fleming |
| 9:05-9:20 | Workmanship | Jeannette Plante  |
| 9:20-9:30 | SW Assurance and IV&V | Esmond Marvray |
| 9:30-9:45 | Safety | Bo Lewis |
| 9:45-10:05 | ESD Awareness | Teresa James |
| 10:05-10:15 | Role of a CSO | Mike Delmont |
| 10:15-10:35 | Reliability Office | Tony Diventi |
| 10:35-10:50 | System Review Office | Mark Goans |
| 10:50-11:05 | Mission Assurance/ Institutional Assurance | Rob Sticka |
| 11:05-11:35 | Institutional Support Office | Mike Kelly |
| 11:35-11:50 | Wrap Up | Marcus Watkins |

- Code 300 Orientation FINAL Agenda B6 - S019: October 22, 2008

- February 3, 2009

## Code 300 Education Series

- 	A series of presentations being sponsored by SMA-D in support of communication and outreach initiatives identified in NASA's Strategic Plan.  The intent of the series is to foster a greater understanding of key NASA activities, functions, and processes.** **
**1st session:  **Bryan O’Connor presented "Quality & NASA"  on July 28, 2008.
**2nd session:**  Safety and Mission Assurance Directorate (SMA-D, Code 300) welcomed Mr. Gregory Robinson on September 30, 2008. Mr. Robinson is a NASA Ambassador supporting the Office of Chief Engineer, NASA HQ.
- He presented “Systems Engineering, with Quality”.
- NPR 7123.1A: Impacts of the Implementation Architecture, SE Engine, & Application.
- Forty attended in Building 8 auditorium.
- This was the second in
**3rd session:  **The third session addressed  NPR 7120.5 D, “Program and Project Management” and the impact on mission success.
- Presented by Michael Blythe, NASA HQ, January 9th

- (01/22/09)

**(CONTINUED)**

- February 3, 2009

## Code 300 Education Series (con’t)

- Topic #4: “SMARTS”
- An internet-based information system designed to:
- Collect all SMA policy and procedural requirements at the Agency and Center levels and from appropriate sources external to NASA
- Support research into SMA requirements by providing a means to filter, collect, search, and sort requirements into “virtual” documents to meet specific needs
- Support the data associated with the verification of compliance to SMA requirements
- Support maintenance and development of new SMA policy and implementation documents/requirements
- _Trace links of SMA requirements_ to improve effectiveness and help limit duplication of requirements
- Presentation by John Lyver, NASA HQ, Date: _**18 February 09, 1-2:30 p.m., in Bldg 26, Room 205**_** **
- Also tied into NASA’s Supplier Assessment System (SAARIS)
- Proposed Topic #5
- Procurement & Quality
- Specific topic under review with James Becker
- Presenter/Date TBD
**Future Topics**
- Plan to pursue topics to be presented by QED-related Vendors and Aerospace Corp.

- (01/22/09)

- February 3, 2009

## GOLD Rules

_*G*_oddard _*O*_pen _*L*_earning _*D*_esign	http://gsfcrules.gsfc.nasa.gov
- Rules for the Design, Development, Verification, and Operation of Flight Systems, (GSFC - STD - 1000)
- The Rules are a high-level subset of all the design rules required for safety and mission success for all space flight products regardless of implementation approach. All products shall be designed, developed, verified and operated in accordance with the GSFC Rules.
- The rationale is based on sound engineering practice, systems management principles, or lessons learned; and
- A systems engineering product or other objective verification method is identified at one or more milestones in the project life cycle.
- Each "Gold Rule" includes a Principle, Rationale, and Activities and Verification by phase.
- Code 500 ‘owns’ the Rules.
- Code 300 hosts the server, involved in all waiver approvals, and currently updating the GPG.

**(CONTINUED)**

- February 3, 2009

## GOLD Rules screenshot 

- February 3, 2009

## Supply Chain 2008 2nd Annual NASA Supply Chain Quality Assurance ConferenceOctober 28-30, 2008 @ NASA’s Goddard Space Flight Center

- Go to _[http://supplychain.gsfc.nasa.gov](http://supplychain.gsfc.nasa.gov/)_ for  agenda, online registration and other conference information
- _Outstanding interest_ with 165 attended.
- Unique gathering of leaders and practitioners from aerospace industry, suppliers and across NASA, including:
- 	GSFC, JPL, JSC, KSC, NASA Headquarters,Hamilton Sundstrand, Space land & Sea, Mantech/SRS, Johns Hopkins University Applied Physics Laboratory, Lane Industries, Inc., Goodrich ISR Systems, SGT, Inc., ITT Space Systems Division, REI Systems, Inc.,  Honeywell Technology Solutions, Inc., ATK - Tactical Propulsion and Controls Elkton Division, ITT Space Systems Div., DCMA, Ball Aerospace & Technologies Corp., University of Bradford/UK, Lockheed Martin Space Systems Company, Space Dynamics Laboratory/USURF, Boeing Space and Intelligence Systems, Moog Inc. Space and Defense Group, Ball Aerospace & Technologies Corporation, Northrop Grumman Technical Services, General Dynamics C4 Systems, Raytheon Vision Systems, Raytheon Missile Systems, Harris Corporation, SSG-Tinsley, MIT Lincoln Laboratory, ITT Industries Space Systems, LLC, PolarOnyx, Inc., aPeak Inc, Qualtech Systems, Inc., Orbital Sciences Corporation, Diamond Materials, Inc., Pratt Whitney Rocketdyne, Advanced Power Solutions, Inc., The Aerospace Corporation, Advanced Cooling Technologies,   Comet Technology Corporation, Raytheon Information Systems, MDL Manufacturing Industries, Inc., L-3 Communications SSG-Tinsley Inc., TRAX International,  Booz Allen Hamilton, Inc.,  Applied Biomathematics, Southwest Research Institute, Co-eXprise, Aerojet, NOAA, MDL Manufacturing Industries, Missile Defense Agency, Syneren Technologies Corporation
- Hosted by the Institutional Support Office, Safety and Mission Assurance Directorate, GSFC

- (10/02/08 jr)

**(CONTINUED)**

- February 3, 2009

## GSFC Management System

- Moving from ISO 9000 to AS9100
- “Compliance” only, Not “certification”
- Goddard Directives Management System (GDMS)
- Lab Management GPR and Training Program

- February 3, 2009

## Number of Internal Audits

*** Excludes DNV/NQA and external audits**

- (02/02/09)

- February 3, 2009

## Internal Audit Training April 08, May 08 and September 08

- <footer>

- Training opportunity for current Goddard internal auditors and auditor candidates
- Experienced lead auditors from the Supply Chain Assessment Team provided training and mentoring (Charlie Robinson and Larry Oien)
- Auditors were from Honeywell
- Part of the NASA Contract Assurance Services (NCAS) contract
- Free service for GSFC (Headquarters Funded)
- Visited labs in eleven buildings
- Verbal results were provided prior to leaving the lab and at the daily outbriefs.
- Written results were provided to each lab point of contact.
- Follow-up assessments will be coordinated as-needed

- February 3, 2009

## Calibration
Past due equipment
Issues with calibration, no calibration required, and calibrate before use stickers
Safety
Trip hazards and egress issues
Blocked entries
Safety glasses not used
Equipment certifications expired
Improper storage of flammables, corrosives, and food items
Items stored on top of cabinets and not secured
WOA Processing
Items not performed in sequence
Inspection column not completed
Performed by and inspected by initialed and dated by the same employee
Housekeeping
Clutter in work areas
Needs in improvement in general

- Past due equipment
- Issues with calibration, no calibration required, and calibrate before use stickers
- Safety
- Trip hazards and egress issues
- Blocked entries
- Safety glasses not used
- Equipment certifications expired
- Improper storage of flammables, corrosives, and food items
- Items stored on top of cabinets and not secured
- WOA Processing
- Items not performed in sequence
- Inspection column not completed
- Performed by and inspected by initialed and dated by the same employee
- Housekeeping
- Clutter in work areas
- Needs in improvement in general

- Shelf Life
- Expired shelf life materials (ex. Rosin Core Solder and Heat Shrink)
- Items stored in unmarked or improperly marked containers
- Chemical Cabinet
- Out-of-date or incorrect inventory lists
- Spillage in cabinets
- Items stored in unmarked or improperly marked containers
- ESD
- Grounding checks not performed periodically or recorded
- Non-ESD materials (i.e., non-ESD Kimwipes) and tools in ESD areas
- Earth grounds not tagged
- ESD Benches not checked periodically
- “Tacky” mats not used or in poor condition
- Certifications not available or current
- Temperature and humidity monitoring

**2008 Internal Assessment Results**
**of GSFC Labs**

- February 3, 2009

## 2009 Internal Audit Schedule

_**Additional Code 600 Internal Audits Identified but not Scheduled**_
- NuSTAR Mirror Lab – TBD
- High Performance Super Computing Facility, Bldg 28, Code 606.2 – TBD
- Cassini Operations Center – Code 692 – TBD
- Plasma Laboratory supporting MMS – Code 670 – TBD

**Code 300 Audits -  TBD**

| Audit ID | Lead Auditor | Responsible Org For NCRs | Schedule Date | Contact Name |
| --- | --- | --- | --- | --- |
| IA09-9 | Mark A Bollard | Code 562 Parts, Packaging, and Assembly Tech Office | 1/27/2009 | KUSUM K SAHU |
| IA09-2 | JERRY G KOSKO | NPOESS Preparatory Project | 2/15/2009 | BOB L BRIDGERS |
| IA09-8 | TOM W CLIFFORD | Landsat Data Continuity Mission (LDCM) | 2/24/2009 | Michelle Marrie |
| IA09-13 | JOHN F ANDERS | ExPRESS Logistics Carrier (ELC) | 2/25/2009 | Robert L. BRIDGERS |
| IA09-11 | Linda A Emerson | Suborbital and Special Orbital Projects Directorate Code 800 | 3/9/2009 | REGENA W HAUGH |
| IA09-4 | DONNA M JACKSON | Code 561 Flight Data Systems and Radiation Effects | 3/9/2009 | KEN E LI |
| IA09-1 | TOM W CLIFFORD | Hubble Space Telescope, Operations | 3/15/2009 | Jim White |
| IA09-12 | Linda A Emerson | Code 250 Radiation Program | 3/24/2009 | DANIEL S. SIMPSON |
| IA09-14 | JOHN F ANDERS | Global Precipitation Measurement (GPM) | 3/24/2009 | BOB L. BRIDGERS |
| IA09-10 | Linda A Emerson | Supply and Equipment Management Branch | 4/7/2009 | SARA A JENSEN |
| IA09-15 | DONNA M JACKSON | Earth Science Mission Operations (ESMO) | 4/14/2009 | BOB L BRIDGERS |
| IA09-16 | Mark A Bollard | Tracking and Data Relay Satellite (TDRS) Project | 5/19/2009 | BOB L BRIDGERS |
| IA09-17 | Darryl L Younger | MAGNETOSPHERE MULTISCALE MISSION | 6/16/2009 | BOB L. BRIDGERS |
| IA09-5 | JOHN F ANDERS | Code 564 Microelectronics & Signal Processing Branch | 7/13/2009 | BOB L KASA |
| IA09-6 | JERRY G KOSKO | Propulsion Branch | 7/13/2009 | SCOTT E GLUBKE |
| IA09-7 | DONNA M JACKSON | Code 540 Recert Program | 9/7/2009 | STANLEY Y CHAN |

- February 3, 2009

## Mission Operations Assurance Goals

- Compliance with NPD 8700.1 NASA Policy for Safety & Mission Success
- Provide solutions to continuously improve overall Center mission performance  through out the its life cycle
- Provide mission assurance oversight and insight support to on-going projects
- Provide in-depth analysis and management on project relevant Anomaly & Problem data
- Provide the stakeholder feedback on cross-project critical anomaly issues and lessons learned
- Assess mission performance through policy, data analysis, compliance verification, validation, early intervention, and risk management
- Assuring the success of Center missions on an ongoing basis

- February 3, 2009

## Mission Operations Assurance Plans

- Establish central database for Center missions anomaly reports
- Coordinate Center Goddard Procedures and Guidelines (GPG) for conducting anomaly investigations
- Apply consistent practices regarding anomaly investigations
- Improve access to Project and contractor documents via “central” library
- Need to address process for dealing with international partners
- Implement process for management review and approval of corrective actions for all Center missions
- Capture and share lessons learned from investigation
- Determine systematic factors and common threats from anomalies
- Develop collaboration toolset to support anomaly investigation activities
- Introduce guidelines and information exchange for out-of-house missions

- February 3, 2009

## Code 300 EEE Parts/Workmanship Group 

- Mike Sampson (NASA Electronic Parts and Packaging Program (NEPP) Manager)
- Jeanette Plante (NASA Workmanship Technical Committee Head)

- February 3, 2009

## GIDEP Program

- GIDEP (_*G*_overnment _*I*_ndustry _*D*_ata _*E*_xchange _*P*_rogram)
- The GIDEP representative at GSFC and ALERT coordinator is Mike Sampson (the NASA Electronic Parts and Packaging Program (NEPP) Manager) _[Michael.J.Sampson.1@gsfc.nasa.gov](mailto:Michael.J.Sampson.1@gsfc.nasa.gov)_301-614-6233
- GIDEP offers WWW database access to failure experience data as a result of ALERTs, SAFE-ALERTs, Problem Advisories and Agency Action Notices
- Notifies users of nonconforming parts, components, chemicals, processes, materials, safety and hazardous situations. This data also includes failure analysis and problem information submitted from laboratory analysis.
- NASA participation is directed by 'Procedures For Exchanging Parts, Materials, and Safety Problem Data Utilizing the GIDEP and NASA Advisories', NPR 8735.1A.
- provides the procedures for the preparation, distribution, and closeout of GIDEP ALERTS,
- The GSFC ALERT data and Project status is maintained on VOSSMA.

**(CONTINUED)**

- February 3, 2009

## GIDEP Screen Shot from the Database 

**(CONTINUED)**

- February 3, 2009

## Workmanship Group

- The NASA Workmanship Technical Committee is headed by Jeanette Plante:_[jfplante@pop500.gsfc.nasa.gov](mailto:jfplante@pop500.gsfc.nasa.gov)_, 301 614-5944
- Information is available at _[http://workmanship.nasa.gov](http://workmanship.nasa.gov/)_
- The NASA Workmanship Technical Committee works closely with the NASA Manufacturing Technology Transfer Center (NMTTC) to ensure training is available and stays current with NASA's standards
- The NMTTC Eastern Region is currently managed under the Mission Assurance Services Contract (MASC) and is located at the HTSI headquarters 7000 Columbia Gateway Drive, Columbia, Maryland.
- HTSI Administrator Coordinator (course schedule, prices and registration):
-     Elaine Hill / Elaine.Hill@honeywell.com
-     Tel: 410-964-7616    Fax: 410-964-7609

**(CONTINUED)**

- February 3, 2009

## NASA workmanship standards 

- GSFC-GPR 8730.6 Electrostatic Discharge (ESD) Control
- GSFC-WM-001 GSFC Workmanship Manual for Electrostatic Discharge Control
- NASA-STD-8739.1A Workmanship Standard for Staking and Conformal Coating of Printed Wiring Boards and Electronic Assemblies
- NASA-STD-8739.2 Surface Mount Technology
- NASA-STD-8739.3 Soldered Electrical Connections
- NASA-STD-8739.4 Crimping, Interconnecting Cables, Harnesses, and Wiring Change 3
- NASA-STD-8739.5 Fiber Optics Terminations, Cable Assemblies, and Installation
- NASA-STD-8739.7 Electrostatic Discharge Control (Excluding Electrically Initiated Explosive Devices)

- February 3, 2009

## The Lab ESD Certification Process

**GSFC labs will be reviewed for ESD protection standards**.
- Pre-Certification Review (as requested by the lab)
- Full Certification Assessment
- Follow-up Assessment
**Scheduling**
- Will be scheduled using the formal assessment schedule developed by Code 300.
*Shall be requested* by lab management, branch management, and/or project quality personnel *prior to* performing ESD work *unless* the workstation has been previously certified by Code 300.
**Follow-Up Assessment**
- Scheduling will be by Code 300.
- Lab Environment will be re-assessed for continued compliance.
- Verification that the monthly workstation measurements are being performed and documented.
- To provide further assistance to the lab personnel relative to ESD.
- Documentation and Reporting
- Lab Follow Up Assessment Report
- Action Items

- February 3, 2009

## Code 300 ESD Program Control Team

| Name | Office | When to Call |
| --- | --- | --- |
| Chanel Duncan, Interim ESD Team Lead
Chanel.C.Duncan@nasa.gov

Terry James, MAES Technical Lead
tjames@pop300.gsfc.nasa.gov | 301-614-5850
c) (443) 864-0202

301-286-9671
c) 301-830-1252
 | Coordinator for scheduling initial lab/workstation certifications, requests for pre-certification reviews, and/or general issues or questions relative to the Code 300 activities (Primary)

Same as above (Secondary) |
| Al Moore, Team Risk Mgmt/Auditor
amoore@riskmc.com
 | 301-286-3263 | Same as above (Alternate) |
| Jose Sancho, ESD SME
jose.d.sancho.1@gsfc.nasa.gov

Feliz Frades

Craig Firman | 301-614-6038

410-964-7602

410-964-7272 | All Technical Questions (Primary)

NMTTC Instructor, Technical Questions (Alternate)

NMTTC Instructor, Technical Questions (Alternate)
 |
| Elaine Hill | 410-964-7616 | NMTTC Certification Training, Scheduling and Registration
www.honeywell.com/workmanship |

- February 3, 2009

## Code 321: Systems Safety Branch

- Name:  Bo Lewis
- Title:  Branch Head
- Office:  Systems Safety Branch
- Tel:  301-286-7123
- Email:  bo.lewis@nasa.gov

- February 3, 2009

## Why Do We Need Safety?

- “If eternal vigilance is the price of liberty, then chronic unease is the price of safety."
**- James Reason, “Managing the Risk of Organizational Accidents”**

- NOAA N’

- Challenger

- Delta II

- Ariane V

- Columbia

- February 3, 2009

**Notes:**
There can’t be a safety pitch without some cool disaster photos, so here they are

Usually the first question a project asks when they see our budget estimate:  why do we need safety?
It has gotten better over the years, based on the good work my people have done.

Read the quote at the bottom of page

## GSFC Safety Organizations(as documented in new GPR 8710.5 “GSFC Safety Program Management”)

- February 3, 2009

## Safety Roles at GSFC

- Assistant Director for Safety and Security (Code 100)
- Overall integration of GSFC safety program
- Institutional Safety  (Code 250)
- Occupational Safety & Health
- Environmental Management
- Safety in I&T Complex (B7, 10, 15, & 29) (Code 500)
- Recertification Program
- Lifting devices and equipment (LDE), and ground-based pressure vessels and pressurized systems (PV/S) at Greenbelt and Wallops.
- Certification and recertification of LDE Operators and Critical Lift Coordinators is also included.
- Lab Safety
- Facility Safety
- Safety at Wallops (Code 803)
- Occupational Safety & Health
- Project Safety for Wallops payloads and sounding rockets
**System Safety at Greenbelt  (Code 321)**
**Organized, disciplined approach to early identification and resolution of system hazards impacting personnel, hardware, software, operations, GSE, and facilities.**
**Support all GSFC Greenbelt managed programs & projects**
**ELV, Shuttle, ISS, etc**

- February 3, 2009

## Systems Safety Branch (Code 321)

- February 3, 2009

**Notes:**
Primarily engineering or safety degrees.

Jamie & Bob are the senior PSMs with each having 25+ years of safety experience

Roman = CSO on GOES-R, so CSOs can come from anywhere

Projects in red are STS/ISS missions where the system safety effort is very prominent

## Code 321 Charter

- Code 321, the Systems Safety Branch (SSB) is responsible for supporting the implementation of systems safety over the program life cycle (Phase A up through separation from launch vehicle) for GSFC managed space flight missions.
- It is the goal of the branch to assist the GSFC missions in **early identification and resolution of safety related issues****. **
- Safety can then be effectively addressed to better support the Projects’ challenge of managing mission risk with respect to both cost and schedule constraints.
- The office provides technical support and consultation to project teams to assist in **defining and interpreting safety requirements**** and in ****developing solutions to safety issues** to enhance the likelihood of safely achieving mission success.
- The SSB works to policy guidelines set by NASA Headquarters and the Center, and to safety implementation requirements set by the Agency, OSHA, the STS and ISS Program Offices and the **various launch range authorities (AF, ESA, JAXA****, etc.).**
- As support team members to the Projects, the SSB personnel provide expertise and other resources to fulfill Program, Center and Enterprise mission objectives.
- The SS&RO also supports the center by serving as the **NASA Safety Reporting System** (NSRS) Technical Advisory Group Member, with responsibility for investigating or supervising investigations of GSFC incidents reported through the NSRS.

- February 3, 2009

**Notes:**
Code 321 System Safety Life Cycle ends at separation from launch vehicle, because after that it is all mission success
early identification and resolution of safety related issues. 

defining and interpreting safety requirements 

developing solutions to safety issues to enhance the likelihood of safely achieving mission success.  
The SSB is unique in 320 in that our external customers are the various launch range authorities (AF, ESA, JAXA, etc.). 

  NASA Safety Reporting System:  independent & anonymouns, infrequently used

## Project Safety Functions

**Negotiating** range safety requirements (and potential **tailoring**) with the applicable launch range (AF & KSC)
**Interpreting** the range safety requirements to allow projects to meet them in cost efficient manner
**Performing hazard analysis** and implementing **closed loop hazard tracking** system to **ensure all hazards are adequately controlled and verified**
- Support Project Reviews (PDR, CDR, PER, PSR) & Launch Site Reviews (GOWG, MIWG, etc.)
- Documenting the design and analyses in **Safety Data Packages** to be **sent to KSC and AF Range Safety for approval**
- Reviewing hazardous operating procedures & **monitoring hazardous operations** at GSFC I&T facility & at launch site
- Providing technical support during the development and test of the mission hardware and software
- Providing **system safety certification letter** to project to allow them to ship payload to launch site
- Providing **on-site safety coverage** as necessary **at the launch site**

- February 3, 2009

**Notes:**
Chronological summary of how SSB supports project throughout their lifecycle

Much of safety work is coordinated through the Safety Working Group, which is composed of safety engineers from S/C and instrument developers, KSC, and AF Range Safety, amongst others

## GSFC System Safety Effort Throughout Project Lifecycle

- Proposal Support
- Requirements Definition
- Design Assessment
- Identification of Hazards
- Recommended Hazard Controls
- Assessment of Risk
- Verification of Hazard Controls
- Development of Safety Data Packages
- Interface with KSC & Range Safety
- Safety Support during I&T Activities
- Track Closure of Verification Items
- Safety Certification
- Prelaunch Safety Support

- February 3, 2009

**Notes:**
Here is a list of items that GSFC safety gets involved into support the typical ELV project
We will go through them in detail on the following slides

## Typical Safety Deliverables    (1 of 3) 

| SAFETY DELIVERABLE | OBJECTIVE | TIME OF DELIVERY |
| --- | --- | --- |
| Operations Hazard Analysis (OHA) | OHA addresses the implementation of safety requirements for personnel, all procedures, and equipment used during, testing, transportation, storage, and integration operations. | 45 days prior to PER |
| Ground Operations Procedures | GOP documents all ground operations procedures to be used at GSFC facilities, other integration facilities, or the launch site for submittal to GSFC OSSMA for review and approval.  Includes launch site ground operations procedures to be submitted to applicable Range Safety prior to use. | –  Launch Range Procedures - Provide 45 days after PSR and submit to applicable Range Safety 45 days prior to first use.  
–  GSFC Procedures - 7 days prior to first operational use. |
| Missile System Pre-Launch Safety Package (MSPSP) | Provides a detailed description of the payload design sufficient to support hazard analysis results, hazard analysis method, and other applicable safety related information.  The developer shall take measures to control and/or minimize each significant identified hazard. | –  Preliminary MSPSP, Mission PDR + 30 days
–  Intermediate MSPSP, Mission CDR – 30 days |
| Verification Tracking Log (VTL) | The VTL provides documentation that demonstrates the process of verifying the control of all hazards by test, analysis, inspection, similarity to previously qualified hardware, or any combination of these activities. | with final MSPSP, with regular updates until all hazards control verifications have been closed |

- February 3, 2009

## Typical Safety Deliverables    (2 of 3) 

| SAFETY DELIVERABLE | OBJECTIVE | TIME OF DELIVERY |
| --- | --- | --- |
| Preliminary Hazard Analysis (PHA) | PHA identifies safety provisions and alternatives needed to eliminate instrument design or function hazards or reduce their associated risk. | –  instruments or subsystems with the SAR at PDR + 30 days 
–  spacecraft with the MSPSP at PDR + 30 days (S/C or Mission). |
| Operating and Support Hazard Analysis (O&SHA) | The O&SHA evaluates procedurally controlled activities for hazards or risks introduced into the system during pre-launch processing  and to evaluate adequacy of procedures used to control identified hazards or risks. | with final MSPSP |
| Safety Assessment Report (SAR) | SAR shall identify all safety features of the hardware, software, and system design, as well as operational related hazards present in the system. | –  Deliver the Preliminary SAR, PDR + 30 days (instrument / subsystem) 
–  Deliver the Intermediate SAR, CDR - 30 days (instrument / subsystem).
–  Deliver the Final SAR, PSR - 30 days (instrument / subsystem) |
| Safety requirements compliance checklist | The checklist indicates for each requirement if the proposed design is compliant, non-compliant but meets intent, non-compliant (waiver required) or non-applicable. | –  instrument/subsystems with the SAR at PDR + 30 days 
–  spacecraft with the Missile System Pre-Launch Safety Package (MSPSP) at PDR + 30 days (S/C or Mission) |

- February 3, 2009

## Typical Safety Deliverables    (3 of 3) 

| SAFETY DELIVERABLE | OBJECTIVE | TIME OF DELIVERY |
| --- | --- | --- |
| Safety Variances | When a specific safety requirement cannot be met, the developer shall submit an associated safety variance, per NPR 8715.3; to GSFC OSSMA that identifies the hazard and shows the rationale for approval. | Deliver to GSFC OSSMA as early as known. |
| Orbital Debris Assessment (ODA) | ODA identifies any stored energy sources in instruments (pressure vessel, dewar, etc.) as well as any energy sources that can be passivated at end of life. | -  PDR
-  CDR |

- February 3, 2009

## Reliability & Risk Analysis Branch,Code 322

- Name: Tony Diventi
- Title:  Branch Head
- Tel:  301-286-6507
- Email:

- February 3, 2009

## Staffing

- * - Subject to re-assignment pending new hires.

-  - Indirect

-  - Direct

-  - Contractor

**Belkacem Manseur**
- GOES-R GND MAM

- February 3, 2009

## Code 322, Reliability and Risk AnalysisReliability Program

- The Reliability section of Code 322 performs a wide range of reliability engineering analyses for both in-house and out-of-house missions:
- Probabilistic Risk Assessment
- Fault Tree Analyses
- Failure Mode and Effects Analyses
- Reliability Block Diagrams and Numerical Assessments,
- Worst Case Analyses (facilitate/review),
- Parts Stress Analysis (facilitate/review),
- Mission Success Criteria (facilitate/review)
- Limited-Life Items
- Trend Analyses
- Numerous other statistical analyses that support design engineering and decision making functions

- February 3, 2009

## Typical Reliability Deliverables

| RELIABILITY DELIVERABLE | OBJECTIVE | TIME OF DELIVERY |
| --- | --- | --- |
| Reliability Program Plan | Describes the planned approach for the reliability activities and scheduling of those activities relative to project milestones. | Preliminary to be included with proposal for GSFC review and evaluation.
Draft 30 days after contract award for GSFC review.
Final 30 days before developer PDR for GSFC review and approval.
Updates as required including changes for GSFC review and approval. |
| Probabilistic Risk Assessment (PRA) | A comprehensive, systematic and integrated approach to identifying undesirable events, the scenarios leading to those events, the frequency or likelihood of those events and the event consequences. | Plan with proposal for GSFC review.
Preliminary 30 days before PDR for GSFC review.
Final 30 days before CDR for GSFC approval.
Updates as required for GSFC approval. |
| Failure Mode and Effects Analysis (FMEA) and Critical Items List | Used to identify all modes of failure within a system design, its first purpose is the early identification of all catastrophic and critical failure possibilities so they can be eliminated or minimized through design correction at the earliest possible time. | Preliminary 30 days before PDR for GSFC review.
Final 30 days before CDR for GSFC review
Revisions as required for GSFC review |

- February 3, 2009

## Typical Reliability Deliverables

| RELIABILITY DELIVERABLE | OBJECTIVE | TIME OF DELIVERY |
| --- | --- | --- |
| Fault Tree Analysis | Used to assess mission failure from the top level. Undesired (top-level) states are identified; all possible combinations of basic (lower-level) events are considered to derive credible failure scenarios. The technique provides a methodical approach to identify events or environments that can adversely affect mission success providing an informed basis for assessing system risks.
The developer shall consider hardware, software and human factors in the analysis. | Preliminary 30 days before PDR for GSFC review.
Revisions 30 days before CDR for GSFC review
Final 30 days before Mission Operations Review |
| Worst Case Analyses (WCA) | Demonstrate design margins in electronic circuits, optics, electromechanical and mechanical items by analyses, test or both to ensure they meet design requirements.
The developer shall consider all parameters set at worst case limits and worst case environmental stresses. | Available 30 days prior to CDR
Updates with design changes |

- February 3, 2009

## Typical Reliability Deliverables

| RELIABILITY DELIVERABLE | OBJECTIVE | TIME OF DELIVERY |
| --- | --- | --- |
| Reliability Assessments and Predictions | Comparative numerical reliability assessments and reliability predictions in order to evaluate alternative design concepts, redundancy, and part selections.   | Available at PDR and CDR for information
Available upon request |
| Software Reliability (addressed in Software Assurance section of MAG) | Activities to be undertaken to achieve the software reliability requirements, as well as the activities to be undertaken to demonstrate that the software reliability requirements have been verified.
The developer shall collect, analyze, and track measures that are consistent with IEEE Standard 982.1-1988, IEEE Standard Dictionary of Measures to Produce Reliable Software.  Measurements for evaluating reliability (e.g., defect density, mean-time-to-failure, and code complexity) shall be documented. | The developer shall document their Software Reliability program in the Software Management Plan.
Initial draft due upon project inception.
Updated periodically throughout the lifecycle, as necessary.
Final due no later than requirements phase. |
| Trend Analyses | Monitoring of selected parameters for trends.
The developer shall maintain and submit a list of subsystem and components to be assessed, and parameters to be monitored. | The developer shall provide a list of parameters to be monitored at the CDR. 
The developer shall provide trend analysis reports at the PER, PSR, and FRR. |

- February 3, 2009

## Typical Reliability Deliverables

| RELIABILITY DELIVERABLE | OBJECTIVE | TIME OF DELIVERY |
| --- | --- | --- |
| Limited-Life Items | Defines and tracks the selection, use and wear of limited-life items, and the impact on mission operations.
The developer shall obtain a program waiver approval by GSFC when the use of an item whose expected life is less than its mission design life. | Preliminary 30 days before PDR for review.
Final 30 days before CDR for approval.
Updates as changes are made; between CDR and delivery, for approval. |

- February 3, 2009

## CRM Process

**Continuous Risk Management** is a structured management practice with processes, methods, and tools for managing project risks
- CRM provides a disciplined environment for proactive decision making:
_***Identify***_**:  Continuously search for risks  **
_***Analyze***_**:  Evaluate impact, probability, timeframe; prioritize**
_***Plan***_**:   Implement strategies; accept, watch, or mitigate risks**
_***Track***_**:  Monitor watched and mitigated risks **
_***Control***_**:  Correct for deviations from****mitigation plan**
_***Communicate and Document***_**:  Provide****feedback (both internal and external)**
- See _http://CRM.nasa.gov_  and Contact
- Feng Hsu X63416 for support

- February 3, 2009

**Notes:**
NASA Started to Formalize CRM in 1997 

Instructor ask the class for the process 
(Enter)

## GSFC Risk Matrix Standard Scale

| 
Likelihood | Safety
(Estimated likelihood of safety event occurrence) | Technical
(Estimated likelihood of not meeting performance requirements) | Cost/Schedule
(Estimated likelihood of not meeting cost or schedule commitment) |
| --- | --- | --- | --- |
| 5   Very High | (PSE  > 10-1) | (PT  > 50%) | (PCS > 75%) |
| 4   High | (10-2 < PSE ≤ 10-1) | (25% < PT ≤ 50%) | (50% <  PCS  ≤ 75%) |
| 3   Moderate | (10-3 < PSE ≤ 10-2) | (15% < PT ≤ 25%) | (25% < PCS ≤ 50%) |
| 2   Low  | (10-6 < PSE ≤ 10-3) | (2% < PT ≤ 15%) | (10% < PCS ≤ 25%) |
| 1   Very Low  |  (PSE ≤ 10-6) | (0.1% <PT ≤ 2%) | (PCS ≤ 10%) |

**1    2     3     4    5**
**Consequence**

| Consequence Categories |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|   Risk | 1 Very Low | 2  Low | 3  Moderate | 4  High | 5  Very High |
| 
   Safety | Negligible or No impact.  | Could cause the need for only minor first aid treatment .  | May cause minor injury or occupational illness or minor property damage.  | May cause severe injury or occupational illness or major property damage.  | May cause death or permanently disabling injury or destruction of property.   |
| 
Technical   | No impact to full mission success criteria | Minor impact to full mission success criteria | Moderate impact to full mission success criteria.  Minimum mission success criteria is achievable with margin | Major impact to full mission success criteria. Minimum mission success criteria is achievable | Minimum mission success criteria is not achievable |
| 
 Schedule | Negligible or no schedule impact | Minor impact to schedule milestones; accommodates within reserves; no impact to critical path   | Impact to schedule milestones; accommodates within reserves; moderate impact to critical path   | Major impact to schedule milestones; major impact to critical path  
 | Cannot meet schedule and program  milestones |
| 
   Cost  | <2% increase over allocated and negligible impact on reserve | Between 2% and 5% increase over allocated and can handle with reserve | Between 5% and 7% increase over allocated and can not handle with reserve | Between 7% and 10% increase over allocated, and/or exceeds proper reserves | >10% increase over allocated, and/or can’t handle with reserves |

**Likelihood**
**1     2     3     4     5**

**GSFC Risk Matrix Standard Scale**

**Code 300**
**Rev. 021307**

- February 3, 2009

## Code 305: Resource Analysis Office 

**Cindy Fryer, Chief  **
**Resource Analysis Office**
**301-286-7204**
**Cynthia.L.Fryer@nasa.gov**

- February 3, 2009

## Organization Charter

- RAO serves as a resource to Center Management for independent assessments of cost and schedule risk, so informed decisions can be made to promote mission success, thereby enhancing the Center’s ability to deliver on its commitments.

- February 3, 2009

## Resource Analysis OfficeOrganization Chart 

- February 3, 2009

## RAO PRODUCTS

_**RAO Products**_
- Independent Assessments
- Trade Studies
- Operations Research Studies

- February 3, 2009

## RAO Timeline

- Idea

- RAO Independent
- Assessment

- Engineering
- Estimates

**Concept**

- Center
- Proposal

- Directed
- Mission

-  Programmatics
-  Multi-concepts
-  Technology
-  Risk
-  Performance

**Baseline**
**Mission**

- RAO
- Independent
- Assessment

- Project
- Grass
- Roots
- Cost Est.

- HQ

- Budget/OMB
- Congressional
- Documents

- Pre-Phase A
- Pre-Formulation

- Phase A
- Formulation

- Phase B
- Formulation

- Phase C/D
- Implementation

- New
- Business

- A/B
- Gate

- Mission
- Confirmation

**RAO Timeline**

- RAO
- Independent
- Assessment

- Project
- Grass
- Roots
- Cost Est.

- Impact on Mission Success

- Center,
- Program/
- Project

- February 3, 2009

## Story of Success

- _GSFC GPM 2005 In-House Spacecraft Win_
**“****RAO’s analysis was crucial to this in-house win.”**
- --Ed Weiler, Associate Administrator, SMD - comment made when he was the GSFC Center Director

- February 3, 2009

## Presenter’s Lessons Learned

- Mike Kelly

- February 3, 2009

## Presenter’s Lessons Learned

- Develop Mission Assurance Requirements and verify these requirements at the end of the procurement.  Never approve supplier’s Performance Assurance Implementation Plans (PAIPs).  The project can “review” but not “approve” the PAIPs.
- CSOs should develop a professional relationship with all Mission Assurance Director’s of Aerospace Companies they deal with
- CSO should develop a professional relationship with all levels within the project (this includes GSFC contracts reps., on-floor personnel, and mgmt.)
- CSO should use non-project (DCMA and NCAS/SAC) funded manpower to support the project in the field at all supplier’s including (their supplier’s-subs)

- (CONTINUED)

- February 3, 2009

## Presenter’s Lessons Learned   (continued) 

- CSO is on the same team as Project Manager’s.  Project should understand exactly what the CSO is doing in support of their hardware/software.  There must be open communication between the CSO and the Project members at all times.
- It is important that the CSO and Project communicate frequently to maintain a common understanding of intentions/expectations for resolving individual issues for monitoring of the contractor, and for communicating with the contractor.  Frequent communication precludes "surprises" and "disconnects" from arising at inopportune times (such as formal reviews or contractor meetings).
- If CSO is working an out-of-house mission, the CSO should develop Letter of Delegation or Task Order for inspections and should visit the supplier regularly.  If the CSO does not do this, then in my opinion, he/she is ineffective.

- (CONTINUED)

- February 3, 2009

## Presenter’s Lessons Learned   (continued) 

- Involve QA, Safety, and Reliability early in the project.
- Ensure that supplier or in-house GIDEP resolutions continue as a launch approaches, including searches of their subcontractor’s data.
- The frequency of GIDEP searches and status updates may need to be increased from the “normal” rate as launch date approaches.
- GIDEPs need to be dispositioned in near real-time in the days just prior to launch.
- Ensure that supplier supports post-launch anomaly resolutions.
- Ensure that supplier uses “test as you fly” methods.
- Ensure that supplier documents/tracks “unknown cause” anomalies since they will be scrutinized by Independent Review team.
- Ensure that supplier documents history of any engineering models in the event that they may become flight models.

- (CONTINUED)

- February 3, 2009

## Presenter’s Lessons Learned   (continued) 

- Institute a weekly telecon with all spacecraft/instrument supplier’s to obtain status and track action items.
- Instruments developed by universities typically have been less rigorous in complying with quality requirements.
- University developers require extra scrutiny from Goddard.
- Visit the university to become familiar with the personnel, procedures, and standards.
- Institute periodic hardware inspections and facility audits.
- Compare the university standards to GSFC standards and identify differences.
- Determine workmanship certification status of personnel and their experience levels.
- During PWB development, some projects jump from the Engineering Model to flight development (did not have a protoflight model).  Don’t do this!!!
- Requirements Flow down and Supplier Control are key areas which must be addressed and performed successfully in order to reduce future spacecraft/instrument problems.

- February 3, 2009

## Slide 93

- February 3, 2009