# Presentation: 591759

## EOS Lessons Learned Pilot

**Presentation to the GSFC Management Council**

- 21-Jul-03

**Paul Brandinger:  301-286-4571; Paul.Brandinger@gsfc.nasa.gov**

## EOS Pilot for Lessons Learned

- To improve sharing lessons learned among EOS Projects
- Leveraging the results of related Center efforts
- SMO Knowledge Management (KM) activities (E. Rogers)
- Library efforts on content organization and services (J. Ormes)
- IT work on Web portals and associated tools (S. Naus)
- HR experience with mentoring and incentives (D. Wolford)
- AETD insight with mini-courses and root-cause analysis (M. Butler)
- Application of PAO communications/outreach to in-reach (L. Chandler)

## Why Lessons Learned?

- Prevent problems from occurring/recurring and repeat successes
- MAP, WIRE, UARS and common spacecraft
- Benefit from collective knowledge and experience
- For in-house work (GPM, SDO, etc.)
- Support future opportunities
- Blue Horizons initiative development

## Why Lessons Learned (cont’d)

- Loss of key personnel
- Rapidly evolving technology
- Competitive challenges
- Shortage of experienced personnel
- Organization culture

## Solutions

**Solutions**

## Process for Sharing Lessons Learned

**Apply**

**Feedback**

**Capture**

**Disseminate**

**Analyze**

**Identify**

**Target**

**Organize**

## Methods Applied to FacilitateSharing of Lessons Learned

- People-related
- Focus groups, surveys and interviews
- Workshops, brown-bag lunch mini-courses and Webcasts
- In-reach communications and informal mentoring
- Project-based storytelling at weekly meetings
- Incentives, rewards and recognition
- Content-related
- Simplified techniques for sharing lessons
- Community of Practice portal; aggregate and categorize information
- Harvest existing sources; problem, failure, anomaly, incident and risk
- Targeted alerts and information updates based on needs/interests

## Accomplishments to Date

- Obtained management support at all levels for EOS Pilot efforts
- Active involvement of Center and Project representatives
- Feedback via focus groups, surveys and staff meetings
- Directorate endorsement of awards process for the Pilot
- Workshops and mini-courses are providing useful exchanges
- Lessons Learned TV (Webcasts) receiving positive reception

## What Have We Learned so Far?

- One size does not fit all; multiple process and techniques are required
- Databases do not solve problems for Projects and people
- Staff interest alone cannot support active involvement
- Value-adds to existing work process will drive participation
- Project, staff and organization needs, interests and perspectives result in different priorities for the same activities

## What is Needed from GSFCfor Lessons Learned to Succeed?

- Project Manager endorsement is essential
- Center, Directorates and Program Offices must establish capture and sharing of knowledge across Projects as a priority
- Projects require infrastructure to be provided by the Center
- Tools and resources from other Directorates are part of this process
- Center practices for lessons learned need to be aligned with Agency guidelines (capture knowledge per 7120.5)
- The review process should reinforce sharing of lessons learned
- Cultural change cannot be motivated from the grass-roots and it takes time
- Center leadership is necessary to encourage staff involvement

## Upcoming Activities

- Rollout Community of Practice portal
- Work with Projects on capture process
- Individuals
- Content resources
- Encourage participation in sharing activities
- Integrate recurring process within workflow
- Draft GPG being developed
- Validate metrics for performance assessment

## How Will We Know ifthis Experiment has Succeeded?

- Recognition that the lessons learning process offers value
- Active participation in the process at all levels
- Improved communications between those who have lessons to share and those having the interest or need to learn from these lessons
- Better access to sources of information and people
- Support for extension/expansion of lessons learning activities
- Validation of performance measurements for the process

## EOS Lessons Learned;  Summary

- A unique experiment is underway
- This Center team is leading the way in a OneNASA process
- FPPD intends that the EOS Pilot
- Demonstrates value to Projects and people through a real case study
- Provides a template for application to all Code 400 Projects
- Acts as a test-bed to support KM initiatives throughout the Center
- Develops a prototype for potential Agency-wide application
- Addresses human capital aspects per the President’s Management Agenda
- Supports the FY03 performance plan:  _**Strategic Objective**_:  Capture engineering and technological best practices and process knowledge to continuously improve NASA’s program/project management
- GSFC is NASA’s lead for the Lessons Learned Information System

## EOS Pilot for Lessons Learned

**Background Material**

- 21-July-03

**Paul Brandinger:  301-286-4571; Paul.Brandinger@gsfc.nasa.gov**

## What are Lessons Learned?

- Stories about successes and failures
- Insights and experience of interest and value to others
- Examples of novel and innovative developments
- Suggestions on how processes can be improved
- Harvesting knowledge from those who have acquired it and sharing with others to allow reuse

## Methods to Improve Sharingof Lessons Learned

- Discussions at regular staff/status meetings, weekly informal exchanges and special lessons-learned reviews based on story-telling
- Lessons-learned TV (Webcasts), formal seminars, mini-courses, brown-bag lunches and workshops (videotaping to preserve these events)
- Incorporation within orientation and training materials
- Targeted email and announcements including special alerts
- Common-access Web sites based on Communities of Practice providing aggregation of Project/Center libraries, lessons learned, common threads, technical questions data base, and related reference resources
- Center publications and in-reach activities-tours/lectures
- Guest speakers during kick-off and other meetings
- People-to-people techniques including; mentoring, apprenticeships, retiree outreach, personnel locators, online chat and question-answer services

## How Lessons Learned can Help?

- Offers benefit from current and prior experience
- Allows the transfer of insights gained to other activities
- Startup of new Projects can leverage lessons learned
- Saves time in locating people and information
- Encourages application of consistent processes
- Understanding how to prevent mistakes from recurring
- Provides recognition and encouragement for successes
- Retains experience for future Project Phases
- Supports the injection of knowledge management efforts and best practices

## Typical Questions Asked by Projects

- What are the standard/best practices for a particular function?
- How did another Project do this?
- What is an example of a success in this area?
- What difficulties were encountered and how were they resolved?
- Has this problem been seen before?
- Who has experience in dealing with this area?
- What information resources exist?
- How can this be communicated to others?

## Project, Staff and OrganizationPriorities are Different

## Sources of Lessons

- Grey-beards and retirees
- Management and staff
- Ongoing and past Projects
- Program and functional offices
- Libraries and personal files
- Contractors and consultants
- Stakeholders/Customers
- Review Boards

## Specific Items to Capture and Share

- Lessons learned; singles, doubles, triples, home runs and strikeouts
- Problems, failures, incidents, anomalies and risks
- Issues, needs, results, and trends
- Best practices and pitfalls to avoid
- Design/decision processes and rationale
- Tools, resources and capabilities
- Wisdom and experience

## Lessons Extraction

- Numerous existing resources contain lessons
- Problem, failure, incident and anomaly reports
- Weekly activity reports and monthly status reviews
- Independent review team reports
- Top Ten lists
- Requests for actions
- Project risks
- Award fee letters and Quarterly Executive Dialogs
- Final reports, after-action reviews and case studies
- Direct extraction from these sources will ease data entry efforts
- Staff has expressed interest in gaining access to this data
- Facilitation from various organizations will be required

## Capture Timing and Process

## Barriers to Capturing and Sharing Knowledge

- Cultural history and management practices are ingrained
- Internal priorities and interests, separate systems (stovepipes)
- Sensitivity of information limits access
- Projects are self-contained entities; require motivation to share (silo- thinking)
- Staff
- pride, professionalism
- emphasis on individuals/projects not knowledge sharing
- difficulty in finding time
- determining the priority of this effort

## Why is Capturing and Sharing ofLessons Learned Difficult?

- Organization structures promote **“silo” thinking** where each area focuses on maximizing their own accomplishment and rewards, as epitomized by the “not-invented-here” syndrome.
- Science/engineering cultures generally value **personal technical expertise** and knowledge creation **over knowledge sharing**.
**Lack of contact**, relationships, and common perspectives among people who don’t work side-by-side.
- An over-reliance on transmitting **“explicit” rather than “tacit” information**.  Most of the important information people need to implement a practice cannot be codified or written down-it has to be shown to them or it requires dialogue and interactive problem solving.  Just creating databases will not cause change to happen.  It is important to capture tacit knowledge including; know-how, judgment, intuition and little tricks.
**Not allowing or rewarding people for taking the time** to learn, share and help each other outside of their own work-area.

## Best Practices Through Lessons Learned

**IMPLEMENTATION**
**“****an idea has no value – **
**until it has been implemented !”**

**ASSESS &**
**EVALUATE**

**APPROVE & **
**DISTRIBUTE**

**COLLECT**

**MANAGEMENT**
**REVIEW OF**
**RESULTS**

**FEEDBACK**

**PROVEN VALUED PRACTICES**

**BEST PRACTICE DEVELOPMENT PROCESS**
**WITH PRESCRIBED ROLES & RESPONSIBILITIES**

**Lessons**
**Learned**

**Dreams**
**Nightmares**

**Reviews**

**CPI**

**Site**
**Visits**

**Task**

**MANY SOURCES OF KNOWLEDGE**

**TQM**

**Workshops**

**Lessons**
**Learned**

**Reviews**

**Site**
**Visits**

**Task**

**Dreams**
**Nightmares**

**CPI**

## Incentives, Rewards and Recognition

**Time to use and create knowledge** has to be recognized and rewarded; if participants feel that they have to “steal” time from the “real” work to do this, they won't.
- Using the knowledge system has to be **self-rewarding to the consumer**; users have to get something out of it, knowledge they need or a sense of status and recognition.
- Recognition lies in **being perceived as an expert** by employees and management. Ensure that an internal expert's name is attached to documents, guidelines, and presentations they created.
**Encourage sharing, transferring, using knowledge and best practices**; by celebrating best-practice success stories and propagating tales of significant savings and important contributions.
**Recognize all parties or units involved**, both those who share knowledge and those who receive knowledge.  Both ends need to feel rewarded to achieve the desired results.
- A **“standardized” system** for rewards may help institutionalize the practice into the common culture.

## What was Done and What Exists Now?

- Lessons learned systems were previously in operation at each Center
- Prior development at GSFC of a lessons learned system
- Reusable Experience with Case-Based Reasoning for Automating Lessons Learned (RECALL)
- Development started in 1995, used through 1999
- Approximately 300 lessons from 8 projects were maintained in this system
- Lessons Learned Information System (LLIS)
- Agency-wide repository of selected lessons based on Committee review
- Developed by LaRC and transferred to GSFC, this is a public-accessible system
- Approximately 1,300 lessons are maintained in this system
- JPL Technical Questions Database
- Available only within JPL, planned for near-term Agency-wide access
- NASA Technical Standards Site
- Integrates NASA, government, commercial and international aerospace lessons learned
- Develops a tie-in with standards documentation
- Approximately 1,400 references to lessons are maintained in this system

## Experience with Lessons Learned

- GAO survey of NASA’s approach
- Limited sharing of lessons learned Agency-wide; focus is within Projects
- LLIS is not a primary mechanism for collecting and sharing lessons
- Dissatisfaction with lessons learned processes and systems
- Inability to retrieve the right lessons at the right time
- Barriers exist to lessons learning
- Lack of time and reluctance to share negative lessons
- Results from other organizations
- Management support for cultural change is essential
- Providing incentives and recognition will encourage participation
- Demonstration of value to individuals and organization is key to success
- Open and free feedback from users and to initiators should be supported
- Failures and success need to be equally reported
- Simple techniques for capturing and distributing lessons must be provided

## What is the Purpose ofLessons Learned for EOS?

- Provide a process for sharing lessons learned that:
- Adds value to Projects by reducing time to find resources that improve performance and mitigate risk
- Supports access to lessons learned:
- In terms of; relevancy, timeliness, ease-of-use, application within current work habits, and minimal added time burden
- In the form of information or knowledge that others have such as:
- Historical Project documents and related resources
- Staff, experts and retirees who represent a vast storehouse of experience
- Offers techniques that will be used
- Databases and Web-based resources are not for everyone
- Mentoring, storytelling and mini-courses interest people
- Push relevant information rather than requiring people to seek it out

## Why Does EOS Need Lessons Learned?

- Large number of high-value projects that can benefit from current and prior experience
- Many useful insights gained during project development are not captured, retained or transferred to other activities
- Startup of new projects consistently relearn lessons
- Opportunity to save time in locating people and information
- Encourage the capture of process knowledge
- Understand how to prevent mistakes from recurring
- Provide recognition and encouragement for successes
- Retain the experience of individuals that will retire
- Support the injection of KM efforts and best practices

## How to Develop an Effective Process

- Start with consideration of individual and Project needs
- Develop a people-centric approach
- Listen to what is wanted
- Facilitate what people are already trying to do
- Try not change people’s habits
- Make it easy to use, leverage the experience of others and add high value
- One-size will not fit all, develop multiple techniques and adapt over time
- Engage the community
- Involve different groups within the Center that have something to offer
- Find ways to create positive interest from the bottom up
- Develop a holistic approach rather than a narrowly-defined solution
- Build-in measurement methods during the design

## Effectiveness Concerns

- Ability to obtain and sustain a level of management support
- Endorsement
- Encouragement
- Motivation
- Approaches to provide individual reinforcement for participation
- Project scope for effort
- Peer support
- Incentives, rewards and recognition
- Demonstration of value-adds to people and projects
- Less time to find information and people
- Ability to exchange best practices, insights and experience
- Greater awareness of and access to other activities

## EOS Lessons Learned WorkingsubGroup Charter

**Purpose**:  The formation of an EOS Lessons Learned Working Group is in response to an EOS Program Office request and Flight Programs and Projects Directorate support for a lessons learned pilot effort as one of the GSFC Knowledge Management Working Group activities.
**Objective**:  The EOS Pilot for Lessons Learned is to develop an improved process for sharing lessons learned within the EOS Program Office
**Activities**:  The EOS Lessons Learned Working Group shall perform the following activities:
- Review and concur with the EOS lessons Learned Pilot (“Pilot”) plan,
- Meet monthly to review progress on the Pilot,
- Determine where changes to Pilot activities are appropriate,
- Provide a report after approximately 6 months assessing progress on the Pilot,
- Provide active direct support to Pilot activities in the areas of;
- Human resources, Library, IT and PAO-related aspects,
- Facilitating tie-in with other GSFC KM pilots,
- Developing and applying performance metrics,
- Recommending appropriate use of incentives, rewards and recognition,
- Software prototyping and integration of applicable tools, and
- Reviewing, validating and verifying lessons learned
- Assess how to ensure effective continuation of an improved lessons learning process, and
- Provide a report after one year assessing progress on the Pilot and recommending next steps for lessons learning activities within GSFC.

## Focus Groups

- Two focus groups were held
- Representatives from each Project
- Questions posed:
- How is the process for sharing lessons learned now performed by EOS Projects?
- Comments on the NASA Lessons Learned Information System.
- What are the issues associated with capture and use of lessons learned by the Projects?
- What suggestions can be offered to improve the process for sharing of lessons learned and how would these tie-in with the Project life cycle?

## Focus Group Summary;Success Factors

- Make it interesting
- Keep it simple
- Provide the ability to find what is wanted (direct/indirect) with summaries
- Offer links to related content
- Deliver quality information (reviewed)
- Ensure timeliness/currency of content
- Encourage capture and addition of content
- One size does not fit all (allow alternate pathways/look and feel)

## Focus Group Suggestions

- Methods to encourage sharing of lessons learned
- Weekly functional and cross-Project roundtable meetings
- Attendance at Project monthly reviews to include;
- AETD branch heads
- Senior staff (Program Office and AETD)
- Add an MSR chart on lessons learned
- Periodic briefings to Directorate management on lessons learned
- Center/Directorate/Project management needs to motivate culture change
- At every meeting and review ask “What is the Lesson Learned”
- Do not impose (versus formalize and mandate)
- Provide additional sources of content
- Project monthly reviews, after-action reports/debrief
- Project risks and IV&V material
- Offer people-to-people contact information
- List of subject matter experts and staff with specific Project experience
- Access to retirees

## Focus Group Suggestions; cont.

- Lessons learned process aspects
- Make it a part of normal activities
- It should not be labor intensive
- Staff needs to write up lessons
- Integrate within Project reviews
- Capture considerations
- Extract and identify key topics from reports
- Link related lessons and organize by Project phase
- Generate practical handbooks for ...
- Develop guidelines and introductory briefings (best/good practices)
- Incorporate uniform practices among contractors
- Dissemination techniques
- Articles in the Critical Path
- Push out routine updates, indexes and alerts
- Provide information that is relevant/related to lessons

## Initial Survey - Results

- *10% response

## Summary of Survey Leading Indicators

- The majority of EOS Project staff consider that a lessons learned system _definitely_ could help them:
- Gain benefit from prior experience (79%),
- Share what they have learned with others (66%),
- Find information more quickly and identify people in GSFC with relevant skills (~50%).
- 79% of staff would _definitely_ prefer to obtain lessons learned through searching an online database.
- Staff would _definitely_ find specific lessons learned occurrence reports to be most useful (67%).
- Most staff would _definitely_ find Project documentation and identifying GSFC points of contact with specific skills to be useful (~2/3).
- More than 2/3 of the staff would _definitely_ be encouraged to use a lessons learned process to improve their job performance.

## Project Staff Feedback

- Presentations were provided at each Project’s staff meeting
- Staff offered mixed responses
- Projects in formulation were generally receptive and supportive of lessons learned activities
- Projects in operations have immediate needs for lessons-learned information
- Projects in implementation have standard practices in place and based on schedule as well as cost constraints are less supportive of changes to existing processes
- Staff considers that the concept of sharing lessons learned makes sense; however everyone wants to know what value they can obtain from this process
- The time and effort to perform sharing of lessons learned is not within the current scope of Project activities
- Project Managers need to actively support these efforts by allocating resources
- Project staff should understand what priority management attaches to this process

## 1st GPM Lessons Learned Workshop; In House Development

*Come out and see center experts share their thoughts and feelings concerning in-house builds*

- Wednesday, November 13, 2002
- 9:00 a.m. – 4:00 p.m.
- Building 32
- Rooms E103 & E109

- Please contact Leslie Cusick at x6-9094 or leslie.cusick@gsfc.nasa.gov if you plan on attending the first of this series
- Also contact Leslie If you would like to suggest topics or speakers for future sessions

## Earth Science Enterprise

- Science forSociety ————*December 13, 2002*

***“******Accelerating the realization of economic and societal benefits ***
***from Earth science, information, and technology ...”***

**BROWN BAG LUNCH**

## Global Carbon Cycle Plan (GCCP) BROWN BAG LUNCH

**Project Formulation Manager:  Dr. Jan Gervin**
**Project Manager:  Dr. Chuck McClain**
**Project Scientist:  Dr. Scott Denning**
**Mission Systems Engineer:Jaime Esper**
**Study Scientists:  Drs. Jim Abshire, Jim Collatz, Watson Gregg, Forrest Hall, James Hansen, Randy Kawa, Bob Knox, Antonio Mannino, Betsy Middleton,  and Compton Tucker**
** **

**January 10, 2003**

## GSFC Technology Overview Presentation 

**GSFC Technology Overview ****Presentation **

- Robert M Savage
- Flight Programs Technologist
- NASA Goddard Space Flight Center
- February 14, 2003

**EOS BROWN BAG LUNCH**

## Aerosol Mission Development

**Aerosol Mission Development**

- Mark Schoeberl
- NASA/GSFC

**Acknowledgement: Yoram Kaufman**

**March 14, 2003**

**BROWN BAG LUNCH**

## Mini-Course Schedule

- Previous presentations are available online

## Lessons Learned TV

- 250 items distributed to date
- Mix of mission, engineering, technology and science
- Interviews/presentations with senior staff and managers
- “New” programming scheduled for Webcast through July
- Archive of Webcasts, along with presentations where available, is accessible online

## Lessons Learned TV Webcast Viewers

**Lessons Learned TV Webcast Viewers**

- GPM Lessons Learned Workshop

- Ron Birk Mini-Course

- Mike Bay; MAP Reliability

- Jerry Madden
- GRO Lessons Learned

- Marty Davis
- GOES Lessons Learned

- Jerry Madden
- Profiles of a Project Manager

- Diane Evans; SAR Remote Sensing
- Mitch Ruda Opto-mechanical Alignment

- Rich Schnurr
- Space INTERNET

- GPM; The Promise of Rain

- Gary Martin
- NASA’ Space Architect

- Orlando Figueroa
- Engineering Excellence

## Upcoming Webcast Schedule

## Slide 51

## Slide 52

## Formulation Community PortalPreliminary Layout

## Lessons Search Return – Folders

## Lessons Search Return – Listing

## EOS Pilot – Project Documents

## In-Reach Communications Plan

- Material for the Community of Practice sites
- Image of the day
- “Did You Know?”
- Story of the Week
- News links; tailored for each Community
- Email to the EOS Projects
- Post flyers on upcoming events
- Staff meeting announcements
- Pilot activities on the GSFC/PAO site
- EOS newsletter
- Biographies of the EOS Program/Project staff
- Organize seminars by general-interest speakers
- Offer internal tours for staff

## EOS Lessons Learned Process Flow

- Archive

## Assessment/Metricsfor the Lessons Learned Pilot

**Measurement objectives**
**Performance of process** elements
**Impact of lessons learned** by identifying improvements to projects and processes
**Qualitative aspects to staff**
**Use of various media** (resources/tools people use to obtain and share information)
- Portal access, online communities, database; entries; hits; number of distinct users
- Mentoring and storytelling, anecdotal discussions, experience from retired employees
**Application of lessons learned**; number and magnitude of benefit
**Value to projects** (performance, cost, schedule), improvement in risk and safety
**Benefit to individuals**; training, recognition, job satisfaction, networking
**Knowledge sharing**; reduced time to find information and people
- Use of systems providing extended access and communities of practice
- Access to resources and implementation of prior experiences
**Periodically survey** results to evaluate interest, effectiveness and suggestions
- Conclusion; **need to design-in measurement techniques** for the entire process

## Learn to Fly; Leveraging What We Have Learned

- Professor John Henderson from Boston University once told a powerful story about the US Army to a gathering of senior BP managers. Before he started; he asked whether BP had any formal approach capturing strategic knowledge. The chief engineer raised and described a database of 'project lessons learned'. John acknowledged this, and went on to tell this story:
- I interviewed a colonel. Now, this colonel was a colonel in one of more elite groups in the US Army. He got a call on Saturday morning at 8 o'clock reminding him that a hurricane had just hit. He was told that because the current administration had very strong ties to that particular part of the country that they did not believe that this should be left to the reserve group because they wanted "no screw-ups".
- So the orders to the colonel were very clear: go down there, provide any support necessary to the people after this hurricane and don't screw up. Clear orders. The army calls it intent - strategic intent.
- The strategic intent was clear.
- 'This particular colonel had never actually commanded any type of civilian-related activity. He 'd always been right on the front lines in hot action. It turns out as part of the executive education in the army; he had been exposed to the "Centre for Army Lessons Learned" as part of their executive education process.
- So he got on his laptop computer he dialed into Army net, hooked into the Centre for Army Lessons Learned and asked the following question - he actually showed me the type:
- “What does the Army know about hurricane clean-up?"

## Learn to Fly; cont’d

- Within four hours he had:
- A profile of the deployment of troops in the last three hurricanes that occurred in North America that the army was sent to provide support and clean-up including types of staff, types of skills, numbers of skills.
- A pro-forma budget- both what budget was required and what the actual budget was and where the cost overruns were.
- The ten questions that you will be asked by CNN in the first 30 minutes on your arrival.
- A list of every state agency and federal agency that had to be contacted and coordinated with and the name of the person that he had to contact with, and the army liaison person who was currently working with that group some place in North America.
- And he had established an advisory team of the three commanders, who agreed to be his advisory group in this command structure, four hours later.
- Then John Henderson asked: 'Is this of relevance to British Petroleum?' A pregnant pause followed, after which the chief engineer again raised his hand.
- 'You asked whether we had a formal approach to capturing knowledge? Well, we have nothing, nothing at all that is anything like that!'
- (provided by Mikael Lindvall from the book Learning to Fly by BP)