# Presentation: 069676

## Surveillance Planning Team

- Presentation to the
- Engineering Management Council
- February 21, 20001
- Greg Robinson
- Revision 4

** **

## Purpose of team

- To improve probability of mission success for NASA projects by improving Agency guidance for risk-based, functionally integrated surveillance.
***Surveillance suitable to the project***
***Surveillance proportional to the risk***
***Surveillance that meets project management needs***

- Team Products:
-   Policy Direction
-   Procedural Guidance
-   Streamlined Simplified Tools to Do the Job

## Surveillance as a method to improve likelihood of mission success

*Surveillance as a method to improve likelihood of mission success*

***Key objectives:******  ***
*** ******Integrate surveillance approach:***
***Engineering, S&MA, Finance, Procurement***
***  ******Improve effective utilization of surveillance resources***
***  ******Enhance communication of surveillance results***
***  ******Surveillance to proactively mitigate risk (part of RBAM)***
***  ******Make surveillance more response to project management***

## Background (PBC Report)

- July 1999, Administrator tasked the AA for Procurement to assess how the Agency could improve implementation of Performance Based Contracting (PBC)
- June 2000, the PBC Team released their report.

- A Significant Finding:
-   Many NASA Centers have not developed or implemented contract surveillance plans.
-   This may be due to the lack of Agency guidance for the development of meaningful surveillance planning.

## Background (NIAT)

- October 20, NASA Integrated Action Team (NIAT) Report, Enhancing Mission Success - A Framework for the Future

- NIAT-13 Action Summary:
-   Ensure that programs and projects receive adequate surveillance based on risk knowledge and that there is ample opportunity for the communication of issues and concerns.

## Surveillance Planning Team:

***Chartered by AA for Procurement and AA for OSMA, November 2000***
***Also sponsored by the Office of the Chief Engineer***
***Representation:***

***NASA Centers:***
***  ******JSC***
***  ******KSC***
***  ******MSFC***
***  ******SSC***
***  ******GSFC***

***NASA Headquarters:***
***  ******Code Q***
***  ******Code H***
***  ******Code M***
***  ******Code AE***
***  ******Code S***
***  ******Code Y***

*Team Sponsorship*

## Current Members:

**JSC ****- Jim Nise  (SOMO Engineering Management) **
**JSC - ****Lee Norbraten (Management Systems Office)**
**KSC - ****Larry Tucci (S&MA)**
**KSC - ****David Culp (Procurement)**
**KSC- ****Terry Smith (Quality)**
**MSFC -  ****Barry Musick (Engineering) (for Jim Kennedy) **
**MSFC - ****Charlie Chesser (S&MA)**
**SSC - ****Mike Smiles (S&MA)**
**GSFC -**** Mary Clark (EVM)**
**S -  ****Nancy Porter (Program Office - Space Science)**
**Y -  ****Tom Magner (Program Office - Earth Science)**
**M -  ****John Castellano (Program Office - HEDS)**
**Q -  ****Steve Newman (PBMA)**
**Q -****  Tom Whitmeyer (Quality)**
**H -****  Jeff Cullen (Procurement)**

*Team Membership*

## Tools To Be Used

***Root Cause Analysis***

***Customer Coordination***

***Functional Analysis***

***Benchmarking of Best Practices***

-   NIAT and PBC findings analyzed to determine underlying contributing factors.

-   EMC and PMC review.

-   Team determines Engineering, Procurement, S&MA objectives for surveillance, and identify methods for improved implementation and integration.

-   Team validates approach against best-of the best in surveillance activity.

##   Lack of meaningful guidance for conducting contract surveillance

*  **Lack of meaningful guidance for conducting contract surveillance*
- Existing NASA Policy, Procedural Guidance, and FAR Supplementation inadequate
- Need for continuous risk evaluation of contractor performance(not a one-time plan)
- “Stove pipe” approach to performing surveillance
**Contributing Factors**
**  ****Lack of common definition**
**  ****Range of contract mechanisms**
**  ****Involvement of multiple functional organizations**
**  ****Differences in approaches across programs and centers**
**  ****Lack of agency wide training**

*Root Causes Currently Identified*

## Define surveillance needs

**CHALLENGES:**
**Large/Diverse Project Base  **
**Large/Diverse Functional Support Organizations**

***Define surveillance needs***
***Provide “risk based” options***
***Utilize Existing Guidance Where Possible:***
*** ******NPG 7120.5***
***NPG 8735.2***
***NASA FAR and Supplement***
***Other (EVM etc)***

## Schedule and Products

*Schedule and Products*

**Initial **
**Meeting**

**Agency **
**Objectives**

**Outline for NPD**

**Draft NPD**

**One**
** ****Week **
**Benchmark**
** ****Study**

**Benchmark **
**Report**

**Draft NPG For**
**Surveillance**

**Publish **
**Summary of**
** ****Guidance **
**Examples**

**Jan-00**

**Feb-00**

**April-00**

**Telecon**
**1/30/00**

**July-00**

**Final Report **
**for **
**Chief Engineer**

**Recommended **
**Follow On Actions **
**(Training,  **
**Etc.)**

**Meeting**

**Meeting**
**1/9/00**

**EMC Meeting**
**2/22/00**

**Meeting**

**Telecon**

## Current Status -  Next Steps

*Current Status -  Next Steps*

***   ******Draft NPD ***
***  ******Survey ***
***  ******NPG Initial Approach (see attached)***
***  ******Benchmarking Plans ***
***Industry***
***Agencies***
***  ******Status meeting with PMC***
*** ***

## Surveillance NPD - Key Elements 

- Plans are required for contracts, grants, cooperative agreements and MOUs unless otherwise specified by the Center Director
- The surveillance plan will:
- Establish and maintain direct linkage with risk of the contract
- Have a level of surveillance penetration that is proportional to risk areas and to the scope and risk of work to be performed.
- Be a collaborative integrated effort with procurement, engineering, S&MA, and finance.
- Be initiated during development of the SOW
- Provide definition of government surveillance by type, responsibility, and frequency.
- Specify if or how sub contract work is addressed.
- Provide for the communication of surveillance plans and results in a defined manner to appropriate levels of management.
- NASA engineering, procurement, safety and mission assurance, and other technical organizations shall provide surveillance support to program/project management through collaborative involvement in the development and maintenance of surveillance plans and performance of associated levels of surveillance penetration.

*Surveillance NPD - Key Elements *

## EMC BACKUP CHARTS

*EMC BACKUP CHARTS*

## 
“Prime” (PBC) Contractor

NASA  as Integrator

In-House

- “Prime” (PBC) Contractor
- NASA  as Integrator
- In-House

***Increasing  level of ***
***integration ***
***responsibility and***
***accountability***
***assigned to***
***contractors***

**80% PBC Goal**

*Changing Environment*

## Project Management/Risk Management

***Risk Management ***
***is a Project Management Tool to Balance Requirements/Resources***

**S&MA**

**Procurement**

**Engineering**

**Functional Organizations**
** ****Provide Expertise/Information to **
**Support this process**

**Finance**

## Elements of a PBC Contract

## Surveillance Methods and Tools

**Control**

**Analyze**

**Plan**

**Track**

**Identify**

- • Project Metrics

- • Project Metrics

## Risk Drives Areas of Surveillance

## Surveillance Proportional To Risk

## Based on “Risk Management”

***Based on “Risk Management”***

**Dynamic**

**Low Dollar Amount**
**Non Flight**
**No Hazardous Operations**

**High Dollar Amount**
**Flight**
**Hazardous Operations**

## Surveillance Dynamics (RBAM) 

- _Final Surveillance _
- _Plan_
- (High/Medium
- Risk) based on
-  selected
-  contractor)

- ACQUISITION  STRATEGY PROCESS

- Engineering
- Procurement
- S&MA, Other

- REQUIREMENTS

- POST AWARD

- SOURCE
- SELECTION

- SOLICITATION
- DEVELOPMENT

- ACQUISITION
- STRATEGY
- DEVELOPMENT

- Contract
- Performance

- _RISK RANKING_
- HIGH RISK
- MEDIUM RISK
- LOW RISK

- _Project Manager_
- _Risk Management_

- _Surveillance _
- _Results_

- Preliminary
- Surveillance Plan

## Level of Surveillance Penetration

**Mapped on Risk Chart**

## Surveillance Continuum

## Surveillance Evolves as Risk is Better Defined

## Technical Penetration Levels 

-  **No Penetration**
-   Accept performing organization’s tasks at face value  (based on assessment that no penetration is required)
-   Contractor develops and implements verification plan

-  **Intermediate Penetration **
** **Includes low penetration with addition of daily or weekly involvement to identify and resolve issues
-   Review verification plan, its implementation, and selected verification closure data

-  **In-depth Penetration**
-   Includes intermediate penetration with addition of:
-   Methodical review of details
-   Independent models to check and compare vendor data, as required
-   Review verification plan, implementation, and concur in all verification closure data

-  **Total Penetration**
-  Perform a complete and independent evaluation of each task
-   Independent review of all verification documentation (including closure data) and witness verification testing

**Technical Penetration Levels **
**(by discipline areas)**

**Low Penetration**
-   Participate in reviews and Technical Interchange Meetings and assess only the data presented
-   Perform periodic audits on pre-defined process(es)
-   Chair board or serve as board member, or RID writer, at a formal review
-   Participate in resolution and closure of issues
-   Review verification plan and its implementation

**Level 0**

**Level 1**

**Level 2**

**Level 3**

**Level 4**

## Business Systems 

-  **Intermediate Penetration **
** **Includes low penetration with addition of daily or weekly involvement to identify and resolve evolving issues
-   Institute specific, regularly planned status and open action/issues reviews

-  **In-depth Penetration**
-   Includes intermediate penetration with addition of methodical review of details
-   Government Senior Business Managers may participate in reviews

**Business Systems **
**Penetration Levels**

**Low Penetration**
-   Participate in or perform periodic audits on pre-defined process(es) and/or systems
-   Participate in resolution and closure of issues
-   Hold periodic status and open actions/issues reviews

**Level 0**

**Level 1**

**Level 2**

**Level 3**

**Level 4**

-  **Total Penetration**
-   Perform a complete and independent evaluation of each task
-   Independent review of all Contractor final contract documentation
-   Regularly scheduled meetings of Senior Business Managers (Contractor and Government) to review current/anticipated issues and overall status

-  **No Penetration**
-   Accept performance at face value  (assess that no penetration is required)

## Technical Surveillance Plan 

## Business Surveillance Plan

- Estimating System

- Accounting System

- Property Mgmt. System

- Major Sub-K Approval

- Periodic Floor Audits

- Proposal development and timely negotiation

- Procurement System

- Socio-Economic Performance

- Labor Relations

- Special Contract Schedules (LOE or T&M)

- Performance Measurement Reviews or Audits

- Corporate Financial Health

- Performance of audits delegated to DCMA

- Performance of audits delegated to DCMA

- Performance of audits delegated to DCMA

- Performance of audits delegated to DCMA

- Performance of audits delegated to DCMA

- NASA review & consent on all Sub-K action over $1M

- Union contract negotiations scheduled for this summer.
- NASA Labor Relations Specialist involved.

- Fortune 500 Company. No formal surveillance required.

- Excellent performance to date. Reviewed every 6 Months

- Monthly review with Senior Managers on status of all
- Proposal Development activity and negotiation status

- Performance of audits delegated to DCMA

- Not applicable

**Surveillance Approach**

**System/Area**