# Presentation: 400603

## GSFC QMSSemi-Annual Management Review4/01/2004 -- 9/30/2004

**Fran Teel – (230)  Directives Manager**
**Harold Mitchell – (170)  QMS Manager**
**Scott Milne – (170) Problem Reporting System Manager**

## Semi-Annual QMS Management Review

- Directives Status
- Fran Teel – (230) Directives Manager
- Audit Results and QMS Improvements/Changes
-  Harold Mitchell – (170) QMS Manager
- Status of Corrective Actions and Upcoming Audits
- Harold Mitchell – (170) QMS Manager
- Product Conformity
-  Scott Milne – (170) Problem Reporting System Manager
- Customer Feedback – Directorates (400, 600, 800, 900)
- Actions from Previous Reviews

## Directives Status

## Goddard Internal Rules Review

**November 3, 2003 - Administrator issued task to review internal rules**
- Remove ambiguity
- Rid the system of policy and procedures no longer required
- HQ cancelled 87, retained 255 with no change, revised 113
**July 1, 2004 - Center received task**
- Re-title GPGs to GPRs, compliance is mandatory
- Inventory and review all internal rules
- Report results to HQ by November 4, 2004
**Rules Review Team Established**
- Richard Day, Code 170, designated as Team Lead
- Team comprised of QMSC representatives and Code 230 Directives Office
- Adopted statement from NPR 7120.5B, NASA Program and Project Management Processes and Requirements, to delineate the convention used to identify requirements within the document
_*A requirement is identified by “shall”, a good practice by “should”, permission*_
*      *_*by “may” or “can”, expectation by “will”, and descriptive material by “is.”*_
**Results**
- 801 internal rules identified  (94 Center-level directives and 476 lower level directives)
- 46 cancelled, 196 retained as written, 45 revised/minor changes, 514 required substantive changes to remove ambiguity and are being worked
- HQ granted an extension through March 1, 2005 to complete the task

## Goddard Internal Rules Review

- _Directives_
- 1.	Goddard Policy Directives (GPDs)
- 2.	Goddard Procedures and Guidelines (GPGs)
- _Organizational Issuances _
- 3.	Documentation created to meet ISO Certification Requirements, including, but not limited to:  Manuals, Organizational Issuances/Work Instructions, and Guidebooks
- 4.	Centerwide/Center Specific Financial Management Rules
- 5.	Centerwide/Center Contract/Grant Management Requirements

- _Program/Project Documents_
- 6.	Technical Standards authored by the Center
- 7.	Plans
- 8.	Center-authored requirements for Major Programs/Projects
- _All Other Documents_
- 9.   Agreements entered into by the Center
- 10. Other Instructions:  documents that if not adhered to, place the mission
-       (programmatic or institutional) at risk, and documents that establish
-        practices that require significant resources to implement and maintain

## Goddard Internal Rules Review

**Goddard Internal Rules Review**

_**Directorate Directives Managers**_
-  100/101   Natalie McMurdy
-  110           Linda Letellier,  Sharon Johnson (alternate)
-  120           Lynne Slater
-  140           Beverly Wildoner
-  150           Gail Williams
-  170           Harold Mitchell
-  200           Mary Jones
-  300           Dave Cleveland
-  400           Barbara Sweeney,  Bob Bridgers (alternate)
-  500           Stan Wojnar
-  600           Michael Van Steenberg
-  800           Regena Haugh
-  900           Jack Richards

-  Required date to forward signature package: January 14, 2005

- Required date to be reviewed, revised, approved, and posted in GDMS: February 7, 2005

## Internal Audit ResultsandQMS Improvement Initiatives and Changes

## Internal Audits Performed

- The Internal Audit process ensures effective QMS Implementation

## Internal & Registration NCR 

- Procedural Issues Have Remained the Primary Causes of NCR’s

- # of IA and Registration NCR’s / Cause Categories, 2000-2004

**Internal & Registration NCR **
**Cause Report Data**

## Internal Audit NCR 

**Cycle-Time Report Data**

- Cycle time to close is decreasing year over year - Very few samples in ’04.
- As “older” NCR’s close, the Cycle-time data is expected to skew higher.
- Histogram depicts large variation in cycle-time to close NCR’s.
- Median cycle-time to close = 119 days (2002-2004 data)
- Average cycle-time to close = 171.7 days (2002-2004 data)

- Days

- Days

## Slide 11

## Slide 12

## Slide 13

## Slide 14

## Slide 15

## Internal Audit Highlights

- _Completed Functional Office Audits_
- Code 110 - Office Objectives, Metrics & Analysis
- no issues
- Code 200 – Directorate Objectives, Metrics & Analysis
- no issues
- Code 400 – Directorate Objectives, Metrics & Analysis
- Some metrics not collected per PG
- Directives change control records not maintained per GPG
- Design of CCMS not under configuration management
- On-line WOA form not up-to-date
- _Upcoming Functional Office Audit_
- Code 500 – Directorate Objectives, Metrics & Analysis
- Code 900 – Directorate Objectives, Metrics & Analysis

## Status of Preventive
 and Corrective Actions

** ****and Corrective Actions**

## Open Registrar Findingsand Observations

- There are no open registrar findings or observations

- March 29-31, 2005, at Greenbelt only
- Rick Giguere is the auditor
- Specific ISO Requirements to be audited in addition to the usual:
- Management Activities: Commitment, Customer Focus, Quality Policy, QMS Planning; Responsibility, Authority and Communication
- Measurement and Monitoring of Processes
- Measurement and Monitoring of Product
- Analysis of Data

**NQA Audit Schedule**

## QMS Improvement Initiativesand Changes Affecting QMS

- _Initiatives_
- Material handling
- Results of AS9100 assessment reveal the need to more specifically address material handling processes
- Internal audit support
- Investigating the need and the means to supplement civil service internal audit function with experienced contractor support
- _Changes_
- Center transformation (e.g., science Directorates merger)

** ****Product Conformity**

## The Evolution of the GSFC Problem Reporting System

- PR/PFR

- Dec. 1998

- All non-conformances

- Feb. 2003

- NCRCAS

- GSFC Problem Reporting System

- Systemic problems, audit findings &  customer complaints
-  Corrective Action required

- Product problems
-  Product disposition required
-  Used by in-house projects (except Swift, Astro-E2)

- GSFC Problem Reporting System

- PR/PFR

- SOARS

- NCRCAS

- Aug. 2003

- NCRCAS

-  On-orbit anomalies – mission ops entries

## Project Non-Conformance & Problem Reports in GPRS

**Project Non-Conformance & Problem Reports in GPRS**

## NCR/PFR Data Reports

- # of  Open NCR’s/PFR’s

- Level of Assembly of Product NCR’s;   2002-2004

## Product Non Conformance Report Data

- Quantity of product problems are influenced by missions in development.

## Product Non Conformance Report Data

- Disposition of Product NCR’s

**Product Non Conformance Report Data**

- Majority of “used-as-is” dispositions are negligible, dimensional discrepancies.

## On-Orbit Anomaly Data

- Number of On-Orbit Anomalies...

- By Year

- By Program

## On-Orbit Anomaly Data

- all effects are analyzed to ensure appropriate categorization and to determine causal relationships.

**On-Orbit Anomaly Data**

## On-Orbit Anomaly Data

## On-Orbit Anomaly Data

**On-Orbit Anomaly Data**

## On-Orbit Anomaly Data

**On-Orbit Anomaly Data**

## Customer Feedback

## Space Communications Customer Feedback

- The primary mechanism used by the Space Communications Program to solicit feedback from our Space and Ground Network customers is to conduct tri-annual Space Communications Customer Forums (SCCFs).
- During the SCCFs, we discuss the available SN and GN services and plans, as well as our customers’ issues and concerns.  We seek feedback and insight regarding their present and future network needs.
- A key customer satisfaction metric is the extent to which we meet the projected proficiency levels as documented in the Project Service Level Agreements.  For October, the performance metrics were as follows:
- SN Operations Proficiency for October 2004 was above Standard of Excellence (99.97%) at 99.98%. (Level of Expectation is 99.90%.)
- GN Operations Proficiency for October was above Standard of Excellence (99.50%) at 99.56%. (Level of expectation is 99.1%.)

## ESDIS Customer Feedback

- Eight DAAC User Working Groups meet once per year and provide feedback through informal and formal recommendations to the DAAC managers, the Project, and to HQ.
- The Earth Systems Science and Applications Advisory Committee (ESSAAC) and its Subcommittee on Information Systems and Services (ESISS) meet twice yearly to provide feedback on data systems and services.
- This Summer, responses from an EOSDIS user survey (conducted by CFI Group, an independent research and consulting firm under contract to OMB) were evaluated.** **
- CFI's initial evaluations shows that the EOSDIS ACSI (Customer Satisfaction) score is 75.  This is 4 points higher than the 2003 aggregated Federal Government ACSI.
- The 1016 responses were further evaluated by ESDIS to find specific trends and information useful to science operations and EOSDIS customer support. DAACs are using feedback from the survey for planning and operation.

## POES/GOES Customer Feedback

- For both POES and GOES, customer feedback from NOAA is a continual, ongoing process.  Approximately 8 - 9 on-site NOAA representatives are imbedded with the Program and take part in day-to-day operations.
- The on-site NOAA representatives review the MSR packages prior to their presentation to the PMC.  This provides an opportunity for the customer to raise issues/concerns and/or provide feedback on the way we are managing the program.  The NOAA representatives routinely extract material from our MSR packages for their own quarterly reports to senior NOAA management.
- In September, we formally requested that NOAA meet with us periodically to review all of their project activities at GSFC (including POES, GOES, and other collaborations).  As yet, no response has been received from NOAA.

## Space Science Customer Feedback as of 22 November 2004

|  | Selected/Proposed |  |  | Success Rate (%) |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Proposal Type | FY02 | FY03 | FY04 | FY02 | FY03 | FY04 |
| Research & Instrument Technology | 38/90 | 35/89 | 13/33 | 42% | 39% | 39% |
| Observing | 1/13 | 15/24 | - | 8% | 62% | - |
| Mission/Instrument | 0/6 | 7/14 | - | 0% | 50% | - |
| Mission Studies | - | 6/9 | - | - | 67% | - |
| Senior Review | 3/3 | 4/4 | 6/6 | 100% | 100% | 100% |
| Technology   (Non-Code S) | - | 0/15 | - | - | 0% | - |
| TOTAL | 42/112 | 67/155 | 19/39 | 38% | 43% | 49% |

**Proposal Success Metrics**

| Science Staff | Academic Year 2001-2002 |  | Academic Year 2002-2003 |  | Academic Year 2003-2004 |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Published | Average | Published | Average | Published | Average |
| 399 | 464 | 1.2 | 574 | 1.4 | 529 | 1.3 |

**Peer Reviewed Publications**

- Note: Many of the FY04 decisions are still pending so only the proposals where selections have been made are included

## Customer FeedbackCode 800

| Reporting Period:  FY04 Wrap-up
October 1, 2003 to September 30, 2004 |  |  |  |  |
| --- | --- | --- | --- | --- |
| Objectives
 | FY 03
Actuals | FY 04
Goals | FY 04
Actuals | Number of Events |
| 1.	Enable scientific research through the development and 	deployment of low-cost highly capable suborbital and 	orbital research payload carriers and science platform 	mission services.  |  |  |  | 77 Events
20 Sounding Rocket Missions 
12 Balloon Flights 
42 Range Events
3 Aircraft Missions
 |
| 	1.1	Customer Feedback  | 86% | 80% | 62% 	 |  |
| 	1.2	Customer Satisfaction | 95% | 100% | 85% |  |
| 2.	Enable aerospace technology and facilitate commercial use of space through advance technology development, test, operational support, and facilitation of the commercial launch activity. |  |  |  | 

3 Sounding Rocket Missions |
| 	2.1	Customer Feedback | 55% | 80% | 66% |  |
| 	2.2	Customer Satisfaction | 97% | 100% | 100% |  |
| 3.	Enable educational outreach and innovative partnerships by providing science and technology education opportunities and pursuing innovative partnerships with academia, other government agencies and industry |  |  |  | 8,512 Students/Public Reached

71 hands-on student/public activities 

91 non-flight outreach activities
 |
| 	3.1 	Customer Feedback | 83% | 80% | 50% |  |
| 	3.2 	Customer Satisfaction | 97% | 100% | 100% |  |

**Not all feedback has been received due to end of the FY launch activities**

## Customer FeedbackCode 900

- Proposals - Submitted: 311, Selected: 32, Not selected: 27, Pending: 252
- Publications – 900.3 (20), 910 (150), 920(80), 940(66), 970 (75)
- 	(Estimated numbers – these are collected annually for Division/Lab. Reports)
- Other contributions (such as conference/symposium proceedings and technical memorandum) were written but not tabulated for this report.
- 970 - Classroom Customer Satisfaction and Feedback –
- 	51 different classrooms (MD, DC, VA, PA, TX, CA): [Student Letters (200), Teacher Thank You Letters (5), Request For Return Visits (51), Teacher Follow-up (51)]
- 970 Visited: 22 schools, 60 classrooms, 60 teachers, Avg. 28 students/classroom= 1680 students reached

## Actions From Previous Reviews

## Actions From Previous ReviewsQMC2002-00011 (Closed)

_**Action**_**:  *****From 1/28/02 review***** – **Code 200 expedite corrective action plan to close out complaint regarding disruption of PG County Emergency Communications.
_**Assigned to**_**:  **Code 200
_**Status**_**:  Closed**
-  PG County signed the updated agreement.