# Presentation: 824112

## GSFC QMSSemi-Annual Management Review10/01/2003 -- 3/31/2004

**Richard Day****	**
**Management Representative**
**June 28, 2004**

## Agenda

- Upcoming Surveillance Audit
- QMS Changes and Improvement Initiatives
- Process Performance/Audit Results
- Product Conformity
- Status of Preventive and Corrective Actions
- Customer Feedback
- Actions from Previous Reviews

**July 28 – 30, 2004 at Greenbelt**
**August 2-3, 2004 at WFF**
**Audit Debrief via VITS from WFF**
** ****POES and SDO scheduled for audit of product development requirements**
** ****WFF project selection is TBD**
** ****Codes 600 and 800 to be audited for objectives/metrics and customer satisfaction**

- NQA Audit Schedule

## QMS Changes andImprovement Initiatives

- _Standards, Processes and Rules_
- Central to establishment of ITA
- Includes review up of QMS Directives with focus on requirements
- Evaluate overall architecture
- Goddard Rules initiative
- Collaborative effort among SMO, AETD & OSSMA
- Initial funding provided from Director’s reserve
- Funded by ITA pool in FY05 and beyond
- 2 Rules Retreats held with engineering divisions
- 82 rules submitted for review
- Draft rules architecture/format reviewed
- Cross check against JPL “Blue Book” in process
- Balancing top down urgency with patience for “buy in” from engineers
- JPL advises that up front work on “buy in” and architecture is essential

**QMS Changes and****Improvement Initiatives**

## Standards, Processes & Rules

**Standards, Processes & Rules**

- Directorate level detailed procedural requirements and best practices handbooks for how GSFC does work

- NASA
- Standards

## GOLD RulesDriving Objectives

- Document the high value requirements to consistently and efficiently achieve mission success for all projects
- Applicable to all projects, in or out of house.  “One stop shop”
- Each requirement has an associated verification step linked to a major lifecycle review
- Provide a closed loop process so that new knowledge and lessons learned are fed back into guiding documents
- Maintaining currency by updating the requirements and verification enables success in the future
- Provide a mechanism to evaluate progress along a project’s lifecycle
- Metrics to evaluate how projects are using the guiding documentation
- Metrics to evaluate how well projects are applying the requirements
_*Above three points are consistent with the intent of CMMI Level 5 “Optimizing”*_

## Functional Organization of GOLD Rules

**Functional Organization of GOLD Rules**

## GOLD RulesPotential Structure

**GOLD Rules****Potential Structure**

**Mission Risk**
**Classification**

**Life cycle Phase**

**Functional **
**breakdown**

**GOLD Rule with rationale**
**-Overall Objective/Requirement)**
**Examples & variations **
**Verification by Life Cycle Phase**

**Links to lessons learned documents, relevant standards, other applicable source material e.g. handbooks, guidelines, process requirements, etc.**

*Goddard Open Learning Design Book (GOLD Book)*
*-actually a hyper linked knowledge portal*

**Sorting by columns yields time phased review criteria linked back to driving requirements**

## Process Performance/Internal Audit ResultsandImprovement Initiatives

## Internal Audits Performed

## NCR’s per Internal Audit

## Internal Audit Highlights

- _Functional Office Audits_
- Code 300 - Directorate Objectives, Metrics & Analysis
- no issues

## Process ImprovementInternal Audits

- _Internal Audit Frequency_
- Internal audit program judged disproportionate to number of organizations and projects, and process problems experienced recently
- Limited auditor pool, civil servant participation and funding
- QMSC directed Center organizations to revitalize auditor pool
- Good response, thank you!
- Revitalized pool of auditors
- New auditors trained and certified
- Beginning to conduct audits
- Comprehensive audit schedule and requirements verification matrix under development

## Process ImprovementInternal Audits

- _Compliance audits of projects to NASA/GSFC directives_
- Combined objectives and focus to improve value and efficiency
- NPR 7120.5, NASA Program and Project Processes and Requirements
- Related NPRs/GPRs
- General GSFC QMS compliance
- Aligned audit timetable so results flow up to PMC gateway reviews
- “Helpful” audit prior to Mission ICRR (A to B transition)
- “Strict” audit prior to Mission CRR (B to C transition)
- I&T process focused audit prior to start of I&T
- Proven worthwhile, some important gaps identified
- NEPA compliance
- Configuration management
- Engineering units verification

## Audit Highlights

- _Project/Program Audits_
- Explorers (AIM & THEMIS)
- Both cited for absence of waiver for use of SI units
- NPR since revised to delegate waiver approval to Program Mgr.
- ESDIS
- CCB not properly documented
- NMP Program
- Program Office does not have an approved CM procedure
- ST-5 Contolled Document List incomplete
- ST-5 Work Order Authorizations not properly prepared and signed
- ST-5 Test procedure not controlled
- GLAST supplier audits performed w/o proper planning and documentation

- closed

- closed

-  open

-  open

- closed

- closed

- closed

** ****Product Conformity**
**and**
**Improvement Initiatives**

** **

**Product NCR's**

** **

**Product NCR's**

## NCR/PFR Activity

|  | 10/01/2003 – 03/31/2004 | Total to Date |
| --- | --- | --- |
| Swift | 59 | 527 |
| ASTRO-E2 | 21 | 84 |
| ST-5 | 21 | 71 |
| STEREO | 4 | 15 |
| GLAST/ACD | 7 | 8 |
| SDO | - | - |

## Process ImprovementProduct Conformity

- _New PR/PFR System_
- Responded to project complaints about NCR system
- Unfriendly
- Insufficient capability
- New system driven by project team (user) requirements
- New system now in place and functioning well
- Projects close to launch stayed with old NCR system
- Newer projects all using PR/PFR system
- New system provides more insight into defects, level of assembly, cause and disposition

## Level of Assembly of Product NCR’s Feb. 2002 to Date

## Process ImprovementProduct Conformity

- _New SOAR System_
- Initial resistance to new Center-wide system
- Incomplete user outreach and change management
- System limitations
- Enhancements in progress to improve use as the primary management tool for handling mission operations anomalies
- Improved analysis capabilities
- All new projects required to use the Center-wide system

## Process ImprovementProduct Conformity

- _Improved On-Orbit Anomaly Data Collection & Analysis_
- Reinstituting previous OAGS analysis and report process
- Cudos to Codes 300 and 400 for major data collection initiative
- Retroactive to start of FY03
- FY03 OAGS Report generated and independently peer reviewed
- Release planned for January but delayed due to questions on anomaly categorization
- ESMO and SSMO are in the final stages of ensuring that the anomaly records in the SOAR system are correct for FY 03 and FY 04
- Report release targeted for end of July

- _Lab/Work Station Flight Certification_
- Inter-Directorate working group formed to define process to certify labs and work areas for flight product realization
- Group Leader is Mike Kelly (Code 303)
- Progress reporting to QMSC
- QMSC will oversee necessary procedure updates

**QMS Changes and****Improvement Initiatives**

## Status of Preventive
 and Corrective Actions

** ****and Corrective Actions**

## Open Registrar Findingsand Observations

- There are no open registrar findings or observations

## Customer Feedback

## Customer Feedback

**There are currently no open NCR’s entered as an external customer complaint.**

## Customer FeedbackCode 800

| Objectives | Actuals
for FY 03 | Goals 
for FY 04 | Current Data
October 1, 2003 to March 31, 2004 |
| --- | --- | --- | --- |
| 1.	Enable scientific research through the development and deployment of low-cost highly capable suborbital and 	orbital research payload carriers and science platform mission services. 

(Percentages based on 6 Sounding Rocket missions, 2 Balloon Flights, 20 Range Events)    |  |  |  |
| 	1.1	Customer Feedback | 86% | 80% | 50%  		 |
| 	1.2	Customer Satisfaction | 95% | 100% | 91.5% |
| 2.	Enable aerospace technology and facilitate commercial use of space through advance technology development, test, operational support, and facilitation of the commercial launch activity.

(No launches/events during this reporting period) |  |  |  |
| 	2.1	Customer Feedback | 55% | 80% | -- |
| 	2.2	Customer Satisfaction | 97% | 100% | -- |
| 3.	Enable educational outreach and innovative partnerships by providing science and technology education opportunities and pursuing innovative partnerships with academia, other government agencies and industry

(Percentages based on approximately 56 hands-on and non-flight outreach activities) |  |  |  |
| 	3.1 	Customer Feedback | 83% | 80% | 45% 	 |
| 	3.2 	Customer Satisfaction | 97% | 100% | 100% |

**The Directorate is on-track to achieving goals by the end of the fiscal year.**

## Customer FeedbackCode 400

**A primary mechanism used by the Space Communications Program (SCP) to solicit feedback from our Space Network and Ground Network customers is to conduct periodic Space Communications Customer Forums (SCCFs), formerly entitled Mission Services Customer Forums (MSCFs).**
**During the SCCFs, SCP representatives discuss with our customers available SN and GN services and plans, as well as their issues and concerns.  We especially seek customer feedback and insight regarding their present and future networking plans and needs.  **
**The target audience includes project and program representatives, customer service representatives, and other service providers.**
**The Space Communications Program has sponsored eight SCCFs since November 2001 (tri-annual). **

**Space Communications**

## Customer FeedbackCode 400

**Participants found status updates and featured presentations informative and useful:**
- “I thought the mix of topics in this past forum was very informative. I was particularly pleased to see the NENS transition addressed, as it has caused a bit of uncertainty at WFF.”
- “The MSCF is useful to me particularly in the area of the plans for the TDRS constellation.  The ‘Forum’ helps me keep up to date on the SN, which aids me in planning HST operations.”
- “I really like the opportunity to hear of future network plans.  Another focus could be on-going missions that have presented unique challenges and how they were overcome or were being worked.  That type of information would help flight projects personnel.”
- “Thanks for including me in this forum. I attended via Webcast, using the audio to follow the Adobe presentation material from your web site. I was not able to attend the entire conference due to other commitments, but what I did hear was very interesting, especially in light of NASA's new initiative toward outer space.”
**Presenters found immediate feedback and potential for immediate interaction helpful to their efforts:**
- “I really appreciate being able to present the ‘Network Security’ topic.  I had a lot of feedback from your customers and expect further contacts.”

**Space Communications (cont.)**

## Customer FeedbackCode 400

**Space Communications (cont.)**

**Valuable feedback includes suggestions for SCCF improvement:**
- “Are there logistic reasons as to why the working groups need to meet at the same time as the general overview, status, and customer feedback session?  I could imagine a more frequent working group meeting schedule (depending on interest and topic) than the status/overview portion of the forum.”
- “For projects under development and planned projects, provide service type, data configuration and data rates in a chart to be updated at successive [SCCFs].  Purpose:  Input to TDRSS-C Conceptual Studies.”
**In a recent survey, SCCF participants revealed the following:**
- Meeting frequency (tri-annual) and length (3 hours): 89% agreed
- Meeting is focused on customers and their interests: 92% agreed
- Presentations are oriented toward project interests: 92% agreed
**Next opportunity for interaction/feedback is at the SCCF scheduled for 8/12/04**
- Our re-designed web page now accepts electronic input for customer satisfaction metrics
- Surveys will be distributed at next SCCF to assess Space Communications Program and SCCF performance

## Customer FeedbackCode 400

**ESDIS**

**There are eight DAAC User Working Groups (UWG). They meet at least once per year and provide feedback in the way of recommendations to the DAAC manager, the Project and to HQ.  ESDIS also measures customer satisfaction through surveys of the DAAC UWG members (see next page).**
**The Earth Systems Science and Applications Advisory Committee (ESSAAC) Subcommittee on Information Systems and Services (ESISS) meets twice yearly to provide feedback to HQ (and hence to the Project) on all things involving data systems and services.**
**The Science Working Group on Data (comprised of EOS instrument team representatives and DAAC Managers) sponsors topical workshops that generate recommendations to the Project and to HQ.  **
**The DAAC Alliance, formed by the eight DAACs plus the Global Hydrology Resource Center at MSFC, serves as a collaborative body to formulate and provide recommendations to the ESDIS project on the future vision and operations of EOSDIS.**
**Metrics are also collected on the distribution of EOSDIS data as a measure of customer interaction with EOSDIS.  These metrics characterize users and usage of EOSDIS and were presented at a recent ESISS/ESSAAC meeting.**

## Customer FeedbackCode 400

**ESDIS (cont.)**

**The DAAC UWG members are satisfied with the EOSDIS services and data products**
- In a recent survey the UWG members (33 responses - 30%) reported they were somewhat to very satisfied
- 17 out of 18 areas were graded “Good” or higher, while the other area was rated as “Acceptable”.
- Areas showing improvement included ease of use of data products and services, user support and documentation, and overall DAAC services.
**At the November 2003 SWGD workshop, representatives from the EOSDIS user community were laudatory of EOSDIS improvements.  Comments included:**
- Dramatic improvement in EOSDIS over the last two years
- Ability to find and retrieve data has been significantly improved
- Improvements in the overall system over the last three years has been remarkable
- People at DAACs are very helpful
**A survey was sent last month to 10,000 random EOSDIS users by CFI Group, an independent research and consulting firm under contract to OMB.**
- 1016 responses were included in the final analysis.  CFI's initial evaluations shows that the EOSDIS ACSI (Customer Satisfaction) score is 75.  This is 4 points higher than the 2003 aggregated Federal Government ACSI.

## Customer FeedbackCode 400

**POES/GOES**

**For both POES and GOES, customer feedback from our NOAA customer is a continual, ongoing process.  Approximately 8 - 9 on-site NOAA representatives are imbedded with the Program and take part in day-to-day operations.**
**The on-site NOAA representatives review the MSR packages prior to their presentation to the PMC.  This provides an opportunity for the customer to raise issues/concerns and/or provide feedback on the way we are managing the program.  The NOAA representatives routinely extract material from our MSR packages for their own quarterly reports to senior NOAA management**.
**We have formally requested that NOAA conduct a “QED” with NASA for all of their project activities at GSFC.  At a preliminary meeting in March, NOAA agreed to develop a list  of topics for the QED.**

## Customer FeedbackCode 600

**Space Science Goals from 600-PG-8730.3.1B**

| Science research results are published with sufficient frequency in peer-reviewed journals;  | Peer Reviewed Publications |
| --- | --- |
| Progress is made on multi-year research projects; | Senior Reviews |
| Contributions are made to new proposals; | Research Proposals |
| Presenting results at national and/or international scientific meetings. | Major Meetings Identified in Laboratory Annual Reports |

**Specific Space Science Goals for FY 2004**

| Coordinate proposal submissions for PI mission AOs, and submit at least two proposals that obtain Category 1 science ratings. |
| --- |
| Establish a role applying Directorate strengths to NASA’s new vision for Exploration.   |
| Complete the SWIFT instrument and produce science results.  |

## Customer FeedbackCode 600

**Peer Reviewed Publications**

| Organization  | Number of Science Staff | Academic Year  2001-2002 |  | Academic Year 2002-2003 |  |
| --- | --- | --- | --- | --- | --- |
|  |  | Number Published | Average per Scientist | Number Published | Average per Scientist |
| Space Science Directorate Total | 399 | 464 | 1.2 | 574 | 1.4 |

**Science Staff includes:**
**Civil Service scientists**
**Co-located university scientists**
**NRC RRAs**
**25% of contractors (estimate)**

## Customer FeedbackCode 600

| Data Archive | Customer Review |
| --- | --- |
| Community Coordinated Modeling Center (CCMC)
 | Senior Review scheduled for 2005
Steering Committee March 2004 |
| Legacy Archive for Microwave Background Data Analysis (LAMBDA)  | Senior Review April 2004 |
| High Energy Astrophysics Science Archive Research Center (HEASARC) | Senior Review 2000
Senior Review April 2004
Users Group meeting February 2004 |
| National Space Science Data Center (NSSDC) | Senior Review 2000, 2001
Senior Review Scheduled 2005 |
| Solar Data Analysis Center (SDAC) | Senior Review 2001
Senior Review Scheduled 2005 |

**Data Archive Customer Reviews**

- Note: Senior Reviews expected every 2 to 4 years

## Customer FeedbackCode 600

|  | Selected/Proposed |  | Success Rate (%) |  |
| --- | --- | --- | --- | --- |
| Proposal Type | FY02 | FY03 | FY02 | FY03 |
| Research & Instrument Technology | 38/90 | 35/89 | 42% | 39% |
| Observing | 1/13 | 15/24 | 8% | 62% |
| Mission/Instrument | 0/6 | 7/14 | 0% | 50% |
| Mission Studies | 0/0 | 6/9 | - | 67% |
| Senior Review | 3/3 | 4/4 | 100% | 100% |
| Technology   (Non-Code S) | 0/0 | 0/15 | - | 0% |

**Proposal Success Metrics**

## Customer FeedbackCode 900

**Customer Feedback is received by the ESD by a number of means:**
**	****On-line customer database/websites. **
**	****Education and Outreach activities.**
**	****Letters and emails from PI’s, Universities, Partners, etc.**
**Each Laboratory within the Directorate collects feedback to evaluate:**
**	****Did users find what they were looking for and were they satisfied with the**
**       ****end product?**
**b.****	****  How would you rate your overall interaction with us? Good or Bad?**
**      ****Information is collected by each Laboratory and others doing Education and Outreach. This information is collected at the Directorate and forwarded to the QMSC.  The many websites and varied responses make it very difficult to tailor the ESD response to the way someone else is reporting. Our goals are customer satisfaction and correcting any negative response that we may receive.**

## Customer FeedbackCode 900

**	****	**_**# - Positive**_**	**_**# - Negative**_**	**_**Total # requesting data**_
**	****	****	****	****	****	****         **_**or information**_
**902/ DAAC >>** **	****         35****	****	****        8******	****	****                 700**
**	****	****	****            ****** Tried to access data during maintenance**
**902/TSDIS >>****	****         17****	****	****        0****	****	****                   43**
**902/GCMD >>****	****         27****	****	****        0****	****	****	****   64**
**The GCMD web site recorded **_***monthly***_** averages from October-March of 16,861 unique visitors and 1,308,048 page hits**
**920/Web responses >>   29****	****	****        6******	****	****	****   83**
**	****	****	****              ****** 3- server down; 3 wanted higher resolution**
**The remainder are questions, permission/publication requests, award notifications, exhibits planning, information requests.**
**910/Web Page >> 3569 unique visitors, 28,097 pages of information requested. The TOMS web site provided an average of 2,300 pages of information per day The TRMM web site had 14,653 visitors for 2004 through March.**

## Customer FeedbackCode 900

**970/K-12 Classroom Visits and Presentations**
**During this period, educationally-interesting science within the Laboratory for Hydrospheric Processes has made its way directly into 42 different classrooms (MD, DC, VA, PA).**
**Customer Satisfaction and Feedback in **
**the classroom come back in the form of:**
**Student Letters (515)**
**Teacher Thank You Letters (14)**
**Request For Return Visits (35)**
**Teacher Follow-up (45)**
**NUMBERS:**
**14 schools**
**42 classrooms:**
**45 teachers:**
**Avg. 28 students/classroom=1176 students**

**970/National and International Teacher Conferences and Conventions**
**During this period, educationally-interesting science within the Laboratory for Hydrospheric Processes has made its way indirectly into hundreds of classrooms through hydrosphere**
**teacher workshops given at conferences/conventions in PA, MD, TX, IL)  Former workshops have been given in PA, MD, DC, TX, IL, CA, and Canada**
**Customer Satisfaction and Feedback**
**from conferences/conventions are in **
**the form of feedback forms distributed**
**by the conference directly after each**
**Workshop**
**NUMBERS:**
**4 conferences**
**13 workshops**
**Approximately 420 teachers (representing over 400 different schools)**
**Reaching over 70,000 students (estimate)**

## Actions From Previous Reviews

## Actions From Previous ReviewsQMC2002-00011 (open)

_**Action**_**:  *****From 1/28/02 review***** – **Code 200 expedite corrective action plan to close out complaint regarding disruption of PG County Emergency Communications.
_**Assigned to**_**:  **Code 200
_**Status**_**:  **Open.
-  Action completion pending PG County sign-off.

## Additional Data
(back up)

**(back up)**

## Audit Cause Codes(Number of NCR’s)

## Audit Cause Codes (Percentage of NCR’s)

**Audit Cause Codes**** ****(Percentage of NCR’s)**

## Defect Codes for Product NCR’s10/01/2003 to 3/31/2004

## Cause Codes for Product NCR’s 10/01/2003 to 3/31/2004

## Dispositions of Product NCR’s 10/01/2003 to 3/31/2004

## Level of Assembly of Product NCR’s 10/1/2003 to 3/31/2004