IG-00-043

AUDIT

REPORT CONSOLIDATED SPACE OPERATIONS CONTRACT –

COST-BENEFIT ANALYSIS AND AWARD FEE

STRUCTURE

September 20, 2000

National Aeronautics and

Space Administration

OFFICE OF INSPECTOR GENERAL
Additional Copies

To obtain additional copies of this report, contact the Assistant Inspector General for Auditing

at (202) 358-1232, or visit www.hq.nasa.gov/office/oig/hq/issuedaudits.html.

Suggestions for Future Audits

To suggest ideas for or to request future audits, contact the Assistant Inspector General for

Auditing. Ideas and requests can also be mailed to:

Assistant Inspector General for Auditing

Code W

NASA Headquarters

Washington, DC 20546-0001

NASA Hotline

To report fraud, waste, abuse, or mismanagement contact the NASA Hotline at (800) 424-

9183, (800) 535-8134 (TDD), or at www.hq.nasa.gov/office/oig/hq/hotline.html#form; or write

to the NASA Inspector General, P.O. Box 23089, L’Enfant Plaza Station, Washington, DC

20026. The identity of each writer and caller can be kept confidential, upon request, to the

extent permitted by law.

Reader Survey

Please complete the reader survey at the end of this report or at

http://www.hq.nasa.gov/office/oig/hq/audits.html.

Acronyms

CSOC Consolidated Space Operations Contract

FAR Federal Acquisition Regulation

IOA Integrated Operations Architecture

OMB Office of Management and Budget

SEB Source Evaluation Board

SOMO Space Operations Management Office
W September 20, 2000

TO: A/Administrator

FROM: W/Inspector General

SUBJECT: INFORMATION: Consolidated Space Operations Contract --

Cost-Benefit Analysis and Award Fee Structure

Report Number IG-00-043

The NASA Office of Inspector General (OIG) conducted an audit of the Consolidated Space

Operations Contract (CSOC). The Space Operations Management Office (SOMO) at

Johnson Space Center (Johnson) is responsible for the CSOC. The SOMO estimated that

consolidating existing space operations contracts under one contract would yield savings of $1.2

billion dollars over the next 10 years. However, the SOMO did not perform a cost-benefit

analysis as part of the decisionmaking process prior to awarding the CSOC. As a result,

NASA is not assured that CSOC is the best approach for fulfilling the space operations1

requirements and that it will achieve the anticipated cost savings.

In addition, we found that NASA did not properly structure the award fee for the CSOC to

evaluate performance of the Integrated Operations Architecture (IOA).2 The CSOC Award

Fee Plan lacks defined criteria for measuring performance, appropriate evaluation periods, and

proper emphasis on cost performance. Without these provisions, NASA cannot measure

contractor performance to assess the appropriate amount of award fee and provide an effective

incentive for the contractor.

Background

The CSOC reflects a consolidation of most existing NASA-wide space operations contracts.

The intent of the consolidation was to reduce the cost of operations by managing all of NASA's

 

1

 Space operations are those activities that provide products and services to enable the utilization and exploration of

space.

2

 The IOA consists of an operations concept, a plan for developing the hardware and facilities, and the blueprints for

the plan to provide space operations services under the CSOC.
2

data collection, telemetry, 3 and communications operations supporting Earth-orbiting satellites,

planetary exploration, and human space flight activities under one contract. Johnson awarded

the CSOC contract to Lockheed Martin Space Operations Company on September 25, 1998.

The cost-plus award fee contract has a 5-year basic period and a 5-year option period. The

current basic contract value is about $1.9 billion. Additional services may be transitioned to the

contract through the exercise of options that could increase the contract by about $1.7 billion.

The CSOC Award Fee Plan provides for two award fee pools to be administered separately

under the contract. A 6-month pool will be awarded at the end of each 6-month evaluation

period. A "lookback"4 pool will be awarded at the end of years 2 and 5 under the basic

contract. If Option 1 is exercised, extending the contract period of performance for 5 additional

years, a "lookback" pool will also be awarded at the end of years 7 and 10. Total available

award fee for the 10-year period is $221.6 million for 6-month evaluations and $55.4 million for

"lookback" evaluations.

Recommendations

We recommended that NASA perform a cost-benefit analysis before completing the

consolidation of contracts into the CSOC or exercising options and evaluate at least annually

whether the projected benefits are being realized. Without a cost-benefit analysis, NASA is not

assured that the integrated operations approach will reduce the cost of operations by the

estimated $1.2 billion over 10 years. In addition, we recommended that the Associate

Administrator for Space Flight, in coordination with the contracting officer and SOMO officials,

establish performance evaluation criteria for the "lookback" award fee pool; reallocate award

fee after criteria are established and meaningful evaluations can be performed; establish

"lookback" award fee periods that do not exceed 12 months; and revise the CSOC Award Fee

Plan to increase emphasis on cost control. These improvements will allow NASA to measure

contractor performance and assess the appropriate amount of award fee and provide an

effective incentive for contractor performance. Finally, we recommended that NASA require

progress reports on the architecture baseline beyond the initial submission. Without current

information, NASA cannot measure contractor performance to assess the appropriate amount

of award fee.

Management's Response and OIG Evaluation

Management concurred with three of the seven recommendations. Management concurred in

principal with the recommendations to determine whether future contract options are cost

beneficial and to evaluate annually whether projected benefits have been realized. We ask that

management provide additional comments to clarify that a

 

3

 Telemetry is the technology of automatic measurement and transmission of data by wire, radio, or other means from

remote sources, as from space vehicles, to a receiving station for recording and analysis.

4

 “Lookback” is the term NASA uses to describe the separate award fee pool for assessing the overall, long-term

aggregate effects of contractor activities on the contract as a whole.
3

cost-benefit analysis, as described in OMB Circular No. A-94,5 will be included in corrective

actions. Without this analysis, NASA cannot determine whether the estimated cost savings will

be achieved. Management's planned corrective actions for the recommendation to determine an

appropriate frequency for the contractor's submission of progress reports on the architecture

baseline is responsive to our recommendation.

NASA could award up to $14.1 million of "lookback" award fee to the contractor for the

period ending December 31, 2000, without an objective basis related to contractor

performance. The purpose of award fee is to provide an incentive for the contractor and to

recognize its performance against established criteria. Awarding any amount without properly

defined criteria, appropriate award fee periods and emphasis on cost control would not

accomplish the intended purpose. Management nonconcurred with all recommendations to

improve the "lookback" award fee provision in the Award Fee Plan because it believes that the

current provisions are in the Government's best interest. However, after issuance of the draft

report, management initiated corrective actions that partially satisfy the recommendations to

establish performance evaluation criteria for the "lookback" award fee and to revise the Award

Fee Plan to reflect a cost control factor of at least 25 percent.6 These actions apply only to the

current "lookback" period. Therefore, we ask that management provide additional comments to

clarify that the corrective actions will apply to the "lookback" provision beyond the current

award fee period. To make the award fee an effective incentive, we also ask that management

reconsider its position on the recommendations:

• to allocate (after criteria are established) a proportionate share of the total "lookback"

award fee, $3.1 million to the 6-month period June 30, 2000, through December 31, 2000,

and transfer the remaining $10.9 million to future "lookback" award fee periods and

• to revise evaluation periods for the "lookback" award fee to be no longer than 12 months.

Details on the status of the recommendations are in the findings section of the report.

[Original signed by]

Roberta L. Gross

Enclosure

 

5Office of Management and Budget Circular A-94, "Guidelines and Discount Rates for Benefit-Cost Analysis of Federal

Programs, dated October 29,1992, provides guidance for conducting cost-benefit and cost-effectiveness analyses.

6

 The cost control factor is discussed on page 15 of the report in the section entitled, "Guidance for Evaluating Cost

Control."
4

Final Report on Audit of Consolidated Space Operations Contract --

 Cost Benefit Analysis and Award Fee Structure
FINAL REPORT

AUDIT OF CONSOLIDATED SPACE OPERATIONS

CONTRACT – COST-BENEFIT ANALYSIS AND

AWARD FEE STRUCTURE
W September 20, 2000

TO: M/Associate Administrator for Space Flight

FROM: Assistant Inspector General for Auditing

SUBJECT: Final Report on the Audit of Consolidated Space Operations Contract

Assignment Number A0000400

Report Number IG-00-043

The subject final report is provided for your use and comment. Please refer to the Executive

Summary for the overall audit results. Our evaluation of your response is incorporated into the

body of the report.

We request additional information for recommendations 1, and 2 as described in the report.

We request that management reconsider its position on recommendations 3, 4, 5, and 7 based

on the additional information provided and indicate its position in additional comments by

November 20, 2000. The corrective action planned for recommendation 6 was responsive.

Please notify us when action has been completed on the recommendation, including the extent of

testing performed to ensure corrective actions are effective. All recommendations will remain

open for reporting purposes until agreed-to corrective actions are completed.

If you have questions concerning the report, please contact Mr. Daniel Samoviski, Program

Director, Earth and Space Science Audits, at (301) 286-6890, or Ms. Esther Judd, Program

Manager/Auditor-in-Charge, at (301) 286-3359. We appreciate the courtesies extended to

the audit staff. The final report distribution is in Appendix G.

[Original signed by]

Russell A. Rau

Enclosure
2

cc:

B/Chief Financial Officer

B/Comptroller

BF/Director, Financial Management Division

G/General Counsel

JM/Acting Director, Management Assessment Division

JSC/AA/Director, Lyndon B. Johnson Space Center
Contents

Executive Summary, i

Introduction, 1

Findings and Recommendations, 2

Finding A. Cost-Benefit Analysis, 2

Finding B. “Lookback” Award Fee Evaluation Criteria, 6 

Finding C. “Lookback” Award Fee Periods, 10

Finding D. Updates to Architecture Baseline, 13

Finding E. Cost Control in Award Fee Plan, 15

Appendix A – Objectives, Scope and Methodology, 17

Appendix B – Elements of Benefit-Cost or Cost Effectiveness Analysis,

19

Appendix C – CSOC Transition Schedule, 20

Appendix D – Procedure for Requesting Deviations, 21

Appendix E – Management's Response, 22

Appendix F – OIG Response to Selected Management Comments, 32

Appendix G – Report Distribution, 34
NASA Office of Inspector General

IG-00-043 September 20, 2000

A0000400

Consolidated Space Operations Contract –

Cost-Benefit Analysis and Award Fee Structure

Executive Summary

Background. In response to the challenge of the National Performance Review to provide

higher quality service at lower cost, NASA and other Government agencies began to restructure

their individual roles and responsibilities. As part of the NASA restructuring, the NASA

Administrator designated the Johnson Space Center (Johnson) as Lead Center for Space

Operations. The SOMO is responsible for the CSOC. The SOMO responsibilities include

providing a strategy for accomplishing mission objectives, establishing goals, maintaining

standards and architectural control,7 providing financial management, and administering the

CSOC. The CSOC reflects a consolidation of most existing NASA-wide space operations

contracts. NASA's expectation is that the CSOC contractor will leverage aerospace industry

experience with commercial expertise, processes, and services to develop innovative and costeffective solutions for providing effective mission and data services.

Objective. The overall objective was to determine whether the CSOC goals were

accomplished. This report identifies conditions regarding a cost-benefit analysis and award fee

structure for the CSOC that warrant timely action by NASA management. Details on the

objectives, scope, and methodology are in Appendix A. A subsequent report will address

other issues associated with the CSOC contract.

Results of Audit. The SOMO neither effectively used cost-benefit analyses nor properly

structured the award fee for CSOC.

• The SOMO has not performed cost-benefit analyses prior to consolidations that have

occurred to ensure that the CSOC is the best approach for fulfilling the consolidated space

operations requirements. Without a cost-benefit analysis, NASA is not assured that the

integrated operations approach will reduce the cost of operations by the estimated $1.2

billion over 10 years (see Finding A).

 

7

 Architectural control refers to the SOMO's oversight of the IOA (see footnote 1) that provides the space operations

services and manages that architecture's planned evolution over time.
ii

• NASA has not defined the criteria upon which to evaluate the contractor's performance for

the IOA. Further, NASA cannot measure contractor performance and assess the

appropriate amount of award fee. Consequently, NASA could inappropriately award up to

$14.1 million in award fee to the contractor for the period ending December 31, 2000,

without an objective basis related to contractor performance (see Finding B).

• The CSOC procurement development team did not establish appropriate evaluation periods

for the contractor's long-term performance (the “lookback” provision of the CSOC Award

Fee Plan). Consequently, the lengthy award fee periods do not provide an effective

incentive for the contractor (see Finding C).

• The contract does not require progress reports on the architecture baseline beyond the

initial submission. As a result, NASA cannot ensure that the supporting infrastructure and

capabilities are maintained to sustain product delivery activities. Without current

information, NASA cannot measure contractor performance to assess the appropriate

amount of award fee (see Finding D).

• Cost control is not sufficiently emphasized under the CSOC Award Fee Plan. This results

in an allocation of award fee to cost performance that is $1.6 million less than the amount

required to be allocated to cost performance. Therefore, NASA has less than the desired

leverage to control cost incurred by the contractor (see Finding E).

Recommendations. NASA should perform a cost-benefit analysis before completing the

consolidation of contracts into the CSOC or exercising options and should evaluate at least

annually whether the projected benefits are being realized. In addition, the Associate

Administrator for Space Flight, in coordination with the contracting officer and SOMO officials,

should establish performance evaluation criteria for the “lookback” award fee pool, reallocate

award fee after criteria are established and meaningful evaluations can be performed, establish

“lookback” award fee periods that do not exceed 12 months, determine appropriate frequency

for the contractor’s submission of progress reports, and revise the CSOC Award Fee Plan to

increase emphasis on cost control. 

Management’s Response and Evaluation of Response

Management’s Response. Management concurred in principal with the recommendations to

determine whether future contract options are cost beneficial and to evaluate annually through a

Cost Savings Profile review whether projected benefits have been realized. Also, management

concurred with the recommendation to determine the appropriate frequency for the contractor's

submission of the architecture baseline and to revise the contract to reflect this change.

Management nonconcurred with all recommendations to revise the "lookback" award fee

provision in the Award Fee Plan. The complete text of management’s response is in Appendix

E.
iii

Evaluation of Response. Management's response and planned corrective actions to require

that the contractor update their architecture baseline annually are responsive to our

recommendation. Although management agreed in principal with the recommendations

concerning a cost-benefit analysis, the comments are not fully responsive because they do not

address performing a cost-benefit analysis or evaluating benefits that result from a cost-benefit

analysis.

Management's responses to the remaining recommendations are not fully nonresponsive.

Although management nonconcurred with all recommendations relating to the "lookback" award

fee, management initiated corrective actions that partially satisfy two of the recommendations.

Management issued a letter to the contractor on July 28, 2000, that established performance

evaluation criteria for the "lookback" award fee and specified that the cost control factor for the

"lookback" award fee is 30 percent. We asked that management provide additional comments

to clarify that the corrective actions will apply to the "lookback" provision beyond the current

period.

We request that management reconsider its position on the recommendations to reallocate the

"lookback" award fee to periods after criteria have been developed and to revise evaluation

periods for the "lookback" award fee to be no longer than 12 months.

A detailed evaluation of management’s comments is provided with each recommendation in the

body of the report. In addition, Appendix F provides our response to selected management

comments.
Introduction

Johnson awarded the cost-plus award fee CSOC contract (NAS9-98100) to Lockheed Martin

Space Operations Company (Lockheed) on September 25, 1998. The contract has a 5-year

base period and a 5-year option period. The contract is valued at more than $3.4 billion over

the 10 years. Lockheed has a supporting team consisting of Allied Signal, Booz-Allen

Hamilton, Computer Sciences Corporation, GTE Government Systems Corporation, and about

36 subcontractors.

The CSOC contractor will provide and manage space operations services to meet the

requirements of the NASA space flight programs. The contractor will also be accountable for

data transmission to the end user, data processing and storage, mission support display and

control, spacecraft operations support, mission planning and analysis, and mission control center

operations. Services will also include trajectory data processing, navigation analysis, and

attitude determination when required.

The CSOC Award Fee Plan provides for two award fee pools to be administered separately

under the contract. A 6-month pool will be awarded at the end of each 6-month evaluation

period. A "lookback" pool will be awarded at the end of years 2 and 5. If Option 1 is

exercised, a “lookback” pool will also be awarded at the end of years 7 and 10.

Total Award Fee Available:

 Basic Contract

6-month Evaluation Pool $129.6 million

“Lookback” Evaluation Pool $ 32.4 million

 Option 1 (if exercised)

6-month Evaluation Pool $ 92.0 million

“Lookback” Evaluation Pool $ 23.0 million
2

Findings and Recommendations

Finding A. Cost-Benefit Analysis

The SOMO had not performed cost-benefit analyses to ensure that each consolidation of existing

space operations contracts is in the best interest of the Government. Additionally, cost-benefit

analyses have not been performed in support of future consolidation of contracts into CSOC.

The SOMO was unaware of the Federal and Agency guidance to perform cost-benefit analyses

and proceeded with the consolidation based on the assumption that it would yield cost savings.

Without a cost-benefit analysis, NASA is not assured that the integrated operations approach will

reduce cost of operations by the estimated $1.2 billion8 over the next 10 years.

Requirement to Perform Cost-Benefit Analyses

Office of Management and Budget (OMB) Guidance. OMB Circular A-94, "Guidelines

and Discount Rates for Benefit-Cost Analysis of Federal Programs,” October 29, 1992, provides

general guidance for conducting cost-benefit9 and cost-effectiveness analyses that are intended to

promote efficient resource allocation through well-informed decisionmaking by the Federal

Government. The Circular recommends performing a cost-benefit analysis as the formal

economic analysis of Government programs or projects. The guidelines apply to any analysis

used to support Government decisions to initiate, renew, or expand programs or projects that

would result in a series of measurable benefits or cost extending for 3 or more years into the

future. The guidelines are shown in Appendix B.

NASA Guidance. A March 13,1997, memorandum from the then Acting Deputy Administrator

directed all NASA offices to perform cost-benefit analyses in the process of considering issues

related to consolidation, downsizing, outsourcing, and research or program elimination.10 The

memorandum states that, in order for NASA to meet its goals, the Agency must make decisions

based on the best information available. Independent and up-front cost-benefit analyses should

be a key element in NASA’s decisionmaking process. Further, the memorandum requires all

NASA offices to perform the analysis in a reasonable and timely manner. The analysis should be

sufficient to provide NASA management with the information it needs to make decisions as well

as withstand the scrutiny of others.

 

8

 The SOMO developed an estimate that indicated the consolidation of space operations contracts would yield a total of

$1.2 billion dollars over the CSOC’s 5-year base period and the 5-year option period.

9

 OMB Circular A-94 provides guidelines for a “benefit-cost analysis.” We use the term cost-benefit analysis for

consistency in the report.

10 The Deputy Administrator addressed the memorandum to Officials-in-Charge of Headquarters Offices; Directors,

NASA Field Installations; and the Director, Jet Propulsion Laboratory.
3

SOMO’s Basis for Consolidation

In 1995, NASA conducted an Agency-wide Zero-Base Review in response to the National

Performance Review and budget challenges. The Zero-Base Review made several

recommendations, which included maximizing outsourcing and commercialization and adopting

consolidated, performance-based contracts. This prompted the SOMO to initiate an approach to

consolidate existing space operations contracts.

Fact-Finding Review. In January 1996, the SOMO initiated a fact-finding review at Johnson,

the Jet Propulsion Laboratory, Goddard Space Flight Center, and Marshall Space Flight Center.

Each Center provided cost and technical data for its current operations services and operations

capability. The fact-finding process was completed in March 1996. The fact-finding team's

summary of the results showed that 58 percent11 of total costs could transition to CSOC. The

fact-finding team did not perform a cost-benefit analysis as recommended by OMB Circular A94 to support its finding.

Procurement Development Team. In April 1996, the SOMO established a procurement

development team to develop the acquisition strategy for the CSOC contract. The Acting Deputy

Administrator directed in a March 13, 1997, memorandum that all NASA offices perform costbenefit analyses in the process of considering issues related to consolidation. Although the CSOC

procurement began prior to the direction from the Acting Deputy Administrator, the contract was

not awarded until September 1998, 18 months after the requirement was in place. However, the

procurement development team did not perform a cost-benefit analysis during this time in

compliance with the NASA direction before awarding the CSOC.

Need for a Cost-Benefit Analysis and Periodic Evaluation

A cost-benefit analysis establishes the basis and support for making well-informed decisions and

provides a baseline that can be used for future evaluation of decisions. OMB Circular A-94

states that a cost-benefit analysis should be used to support program or project decisions that will

result in measurable benefits extending for 3 or more years. CSOC is expected to yield savings

over a 10-year period. In addition, the Acting Deputy Administrator’s memorandum states that

up-front cost-benefit analyses should be a key element in NASA’s decisionmaking process. The

SOMO should have performed a cost-benefit analysis prior to each contract consolidation and

completed periodic evaluations after consolidation to assure that the estimated saving of $1.2

billion is being achieved. Periodic evaluation would assist in identifying the need for corrective

actions to achieve savings. Establishment of performance measures against which contractor

performance can be assessed would be a useful management tool.

 

11 Fifty-eight percent of the total cost for redundant efforts in the following functional activities could transition to CSOC:

administration and management; facility and contract management; strategic engineering; maintenance and operations;

sustaining engineering; project development and engineering; and research and development technology. The SOMO

determined that the remaining 42 percent of efforts in these functional activities could not be transitioned.
4

Future Transitions Could Significantly Increase Contract Value

All or some of the services performed previously under 13 NASA contracts are transitioned to the

CSOC under the basic contract. See Appendix C for the transition schedule. The current basic

contract value is about $1.9 billion and includes all of the effort transitioned from the contracts and

Option 6A, which was exercised February 18, 1999. Additional services may be transitioned to

the contract through one or more of the remaining nine options. Eight of the options, listed in the

table below, have been priced by the contractor. If exercised, these options could increase the

contract by about $1.7 billion.12

Contract Options

Option

Number Description

Effective

Dates

Total

Estimated

Cost

(Millions)

1 Extend Contract Period of Performance 1/01/04 -

12/31/08 $1,302.6

2A Kennedy Space Center (KSC) -

 Expendable Launch Vehicle

 Telemetry Station

2/01/01 -

12/31/03 $ 12.7

2B KSC - Expendable Launch Vehicle

 Telemetry Station

1/01/04 -

12/31/08 $ 23.5

3A KSC Effort - Communications System 10/01/01 -

12/31/03 $ 47.5

3B KSC Effort - Communications System 1/01/04 -

12/31/08 $ 142.6

4A KSC Effort - Checkout and

 Launch Control System

10/01/02 -

12/31/03 $ 27.3

4B KSC Effort - Checkout and

 Launch Control System

1/01/04 -

12/31/08 $ 100.3

5 Additional Labor Hours

 (Maximum of 100,000 hours)

Basic contract or

 5-year extension

To Be

Proposed

6A Space Station Enhanced Mission and

 Data Service Requirements

10/01/98 -

9/30/03

Exercised

2/18/99*

6B Space Station Enhanced Mission and

 Data Service Requirements

1/01/04 -

12/31/08 $ 10.4

Total Increase to Contract Value $ 1,667.0

 

12Options 1 through 4B provide for additional direct labor hours for the "Level of Effort" portion of the contract. The

additional 4,385,260 hours for these options are not included in the option prices. Estimated value of the additional direct

labor hours at a rate of $53.31/hour is about $233.8 million.
5

*Option 6A, Space Station Enhanced Mission and Data Service Requirements, increased the basic contract value by $26.1

million and is included in the current contract value of $1.9 billion.

Recommendations, Management’s Response, and Evaluation of

Response

The Associate Administrator for Space Flight should:

1. Direct the SOMO to perform a cost-benefit analysis prior to exercising any contract

options.

Management's Response. Concur. The Associate Administrator for Space Flight indicated

that prior to exercising any contract options, management will determine whether exercising the

option is the most advantageous method of fulfilling the Government's need.

Evaluation of Response. Management's comments are not fully responsive because they do

not directly address the recommendation. Management did not state that the determination of

whether to transition other services to the contract will include a cost- benefit analysis in

accordance with the guidance in OMB Circular A-94 as recommended. Therefore, we request

that management provide additional comments to specifically address the performance of a costbenefit analysis. The recommendation is resolved but will remain undispositioned and open until

the agreed-to corrective actions are completed.

2. Direct the SOMO to evaluate at least annually whether the projected benefits have

been realized.

Management's Response. Concur. A periodic evaluation of projected benefits has already

been put in place, and the contractor performs a quarterly Cost Savings Profile review of the

initiatives contributing to cost savings. Further, NASA reports CSOC savings to Congress

semiannually.

Evaluation of Response. Management's comments are not fully responsive because they do

not address evaluating projected benefits resulting from a cost-benefit analysis. The intent of the

recommendation is to ensure that prospectively, a cost-benefit analysis will be performed prior to

future consolidations under the contract and that the SOMO will evaluate at least annually whether

the projected benefits of the consolidation are realized. The evaluation of the contractor's

performance against the contract cost baseline is a contract administration function that does not

satisfy the intent of the recommendation. Further, we question the basis of reports to Congress of

CSOC savings if the savings are not related to an analysis performed in support of the decision to

consolidate contracts. Therefore, we request that management provide additional comments to

address the evaluation of projected benefits resulting from a cost-benefit analysis. The

recommendation will remain undispositioned and open until the agreed-to corrective action is

completed.
7

Finding B. “Lookback” Award Fee Evaluation Criteria

NASA has not defined criteria upon which to evaluate the contractor’s performance for the IOA.

The contracting officer’s technical representative did not develop the necessary criteria to evaluate

the contractor’s performance as required by Federal and Agency guidelines. Therefore, NASA

could award up to $14.1 million of “lookback” award fee13 to the contractor for the period

ending December 31, 2000, without an objective basis related to contractor performance.

Requirements to Establish Criteria

Federal Acquisition Regulation. Federal Acquisition Regulation (FAR)16.401, “Incentive

Contracts,” states that incentive contracts, including cost-plus-award-fee contracts, are designed

to obtain specific acquisition objectives by:

• establishing reasonable and attainable targets that are clearly communicated to the contractor,

and

• including appropriate incentive arrangements designed to:

- motivate contractor efforts that might not otherwise be emphasized and

- discourage contractor inefficiency and waste.

NASA FAR Supplement. NASA FAR Supplement 1816.405-273, “Award Fee Evaluations,”

states that on contracts for which the deliverable is the performance of a service over a specified

period, contractor performance is definitively measurable within each evaluation period.

Subsection 1816.405-274 states that explicit evaluation factors shall be established for each

award fee period.

NASA Award Fee Contracting Guide. The NASA Award Fee Contracting Guide, dated

December 2, 1997, states that factors such as technical, project management, and cost control

should be selected and supplemented by a limited number of subfactors describing significant

evaluation elements over which the contractor has effective management control. Paragraph 3.4.2

in the Guide states that once evaluation factors are selected, standards or criteria are developed

for measuring contractor performance and assessing the amount of award fee earned. Criteria for

contract performance will be included in the contract, and the contractor is then judged on how

well it performs in relation to those criteria.

“Lookback” Award Fee Evaluations

 

13 Award fee is a monetary award paid to the contractor based on the Government’s judgmental evaluation, which must be

sufficient to provide motivation for excellence in contract performance. Under a cost-plus-award-fee contract, an available

award fee pool is negotiated and included in the contract. However, the actual award fee earned by the contractor is

determined by the Government and is based on the contractor’s performance.
8

Under the “lookback” award fee provision, NASA will evaluate the contractor’s performance on

the IOA. The evaluations will be based on the contractor’s success in implementing the IOA as

submitted in the contractor’s proposal and will be subsequently documented in data requirements

description 2.3-b, “Architecture Baseline.”14 According to the Award Fee Plan, this evaluation

will not address specific contractor actions or accomplishments but will assess the overall, longterm aggregate effects of such activities on the contract as a whole.

Criteria for Evaluating “Lookback” Award Fee

NASA did not define criteria upon which to evaluate the contractor’s performance on the IOA

prior to the start of the current evaluation period. The contracting officer’s technical representative

stated that he was still in the process of developing the “lookback” award fee evaluation criteria

and its respective weightings.15 The criteria the contracting officer’s technical representative is

considering are for the contractor’s performance in:

• meeting the proposed cost savings profile,

• implementing the IOA development initiatives,

• achieving an adequate return on investment for the Government’s investment in the IOA

initiatives, and

• meeting the original goals of the IOA.

In an attempt to clarify the basis for evaluations, on February 25, 2000, the contractor submitted

to the NASA contracting officer’s technical representative a preliminary draft of suggested criteria

for the first evaluation period. A contractor representative stated that NASA has not responded

to the suggestions or communicated the criteria for the “lookback” award fee evaluation criteria.

Therefore, the contractor is not aware of how its performance will be evaluated for the first

“lookback” award fee evaluation period.

Motivating the Contractor and Determining Appropriate Award Fee

Without properly defined criteria, NASA is unable to effectively motivate the contractor and

discourage inefficiency and waste. Further, without criteria, NASA cannot measure contractor

performance and assess the appropriate amount of award fee. Consequently, NASA cannot

perform meaningful evaluations and appropriately award up to $14.1 million for the award fee

period ending December 31, 2000. Finally awarding any amount without properly defined criteria

and an objective evaluation of contractor performance related to those criteria is inconsistent with

the intent of acquisition regulations.

 

14 Data requirement description 2.3-b requires the contractor to submit a description of the IOA and to manage its planned

evolution over time. The contract requires an initial submission at the end of contract phase-in, December 31, 1998, with

updates as necessary.

15 In addition to identifying how performance will be evaluated and measured, the performance evaluation plan should

indicate the relative priorities assigned to the various performance areas and evaluation factors or subfactors. This may be

accomplished through using percentage weights for those areas, factors, or subfactors.
9

“Lookback” Award Fee Performance Period Ending December 2000

NASA allocated a proportionate amount of the “lookback” award fee, $14.1 million, to the 2-

year16 period ending December 31, 2000. However, NASA has not provided the contractor the

criteria or standards of performance that will be used to evaluate performance under the

“lookback” provision. The contractor has operated since October 1998 without properly

defined criteria and could inappropriately be awarded $14.1 million in award fee.

Adjustment to Current Award Fee Periods and Amount

If NASA develops criteria by June 30, 2000, 21 of the 27 months in the first evaluation period

will have already passed, but the criteria would be in place for the last 6 months of the evaluation

period -- July 1, 2000, through December 31, 2000. A proportionate share of the $14.1 million

“lookback” award fee, $3.1 million, should be allocated to that 6-month period. The remaining

share associated with the first 21 months, $10.9 million, should be allocated to subsequent

“lookback” award fee periods when criteria for evaluating performance under the “lookback”

award fee are in place.

Recommendations, Management’s Response, and Evaluation of

Response 

3. The Associate Administrator for Space Flight should direct the CSOC contracting

officer to establish performance evaluation criteria for the “lookback” portion of the

award fee pool prior to June 30, 2000.

Management's Response. Nonconcur. Management stated that the Government has

performance evaluation criteria in place against which the CSOC IOA is evaluated. The

contractor is being provided an Award Fee Areas of Emphasis letter that further clarifies the

Government's plan to evaluate the "lookback" portion of the award fee. The response included

excerpts from the letter dated July 28, 2000. Further, management explained that NASA uses

the contract Award Fee Plan, the recently issued “Lookback” Areas of Emphasis letter, and the

6-month Award Fee Area of Emphasis letter to determine the contractor's success in meeting the

IOA.

Evaluation of Response. Although the response indicates nonconcurrence, management has

taken corrective action that partially addresses the recommendation. The contractor and the

NASA contracting officer's technical representative confirmed our conclusion that criteria for

 

16 The first period includes the 3-month phase-in period ended December 31, 1998, and the first 2 years of contract

performance.
10

evaluating the "lookback" fee had not been developed and communicated to the contractor. They

explained that criteria were being developed and that the contractor had submitted suggested

evaluation criteria. Further, the contractor pointed out that paragraph VII of the Award Fee Plan

that addresses the “lookback” provision contained no evaluation factors or weightings. The plan

differentiates between the 6-month evaluations and the "lookback" provision and makes no

reference to the weightings being used for both. It was not clear to the contractor or us whether

NASA intended for the evaluation criteria and weighting presented in paragraph VI to apply to

the “lookback” provision in paragraph VII and to the 6-month evaluations.

We consider the issuance of management's letter to the contractor, specifying evaluation criteria

and weightings, to be partial corrective action for this recommendation. Management may have

been in the process of developing the criteria during our review, but it did not formalize and

communicate this information to the contractor until after our discussions. We reviewed the

Award Fee Plan again and concluded that it does not adequately convey criteria for the

“lookback” award fee. If the criteria and weightings presented in the letter are applicable to all

“lookback” award fee periods, this information should be clarified in the Award Fee Plan.

References to implementation of the IOA in the Award Fee Plan and the Statement of Work,

while descriptive of services to be provided, do not provide criteria for evaluation. Therefore, we

request that management reconsider its position and provide additional comments.

4. Direct the CSOC contracting officer to:

• Evaluate the contractor’s performance for the period June 30, 2000, through

December 31, 2000, after evaluation criteria are in place and meaningful

evaluations can be performed.

• Allocate a proportionate share of the total “lookback” award fee, $3.1 million,

(22 percent of $14.1 million) to the 6-month period June 30, 2000, through

December 31, 2000, after criteria have been developed.

• Transfer the remaining $10.9 million (78 percent of $14.1 million) to future

“lookback” award fee periods.

Management's Response. Nonconcur. Management stated that criteria for the "lookback"

portion of the award fee pool is already in place and that no changes are needed to the contract.

An Award Fee Areas of Emphasis letter is being provided to the contractor that further clarifies

the Government's plan to evaluate the "lookback" portion of the award fee.

Evaluation of Response. Management's comments are nonresponsive. The letter described in

the response covers the IOA "lookback" period October 1, 1998, through December 31, 2000,

but was not issued until July 28, 2000, which was the same date that comments were provided to

a draft of this report (see Appendix E). We maintain that the contractor has operated since

October 1998 without properly defined criteria. The Performance Evaluation Board can use
11

criteria recently developed to evaluate performance for the 6-month period ending December 31,

2000. Accordingly, only a proportionate share of the available "lookback" award fee should be

allocated to this period. We maintain our position that the remaining share should be allocated to

subsequent award fee periods when criteria for evaluating performance are in place. We request

that management reconsider its position and provide additional comments.
12

Finding C. “Lookback” Award Fee Periods

The CSOC procurement development team did not establish appropriate evaluation periods for

the “lookback” award fee provision of the CSOC Award Fee Plan. The team did not follow

guidance in the NASA FAR Supplement or NASA Award Fee Contracting Guide in developing

these evaluation periods. The two periods for the “lookback” award fee evaluation under the

basic contract are 2 and 3 years, respectively. These lengthy award fee periods do not provide

an effective incentive for the contractor. As a result, NASA may not receive the desired

performance on the IOA.

Requirements to Establish Award Fee Periods

Federal Acquisition Regulation. FAR 16.405-2, “Cost-Plus-Award-Fee Contracts,” states

that such contracts shall provide for evaluation at stated intervals during performance, so that the

contractor will periodically be informed of the quality of its performance and the areas in which

improvement is expected. Partial payment of fee shall generally correspond to the evaluation

periods. Therefore, the award fee can create an effective incentive by inducing the contractor to

improve poor performance or to continue good performance.

NASA FAR Supplement. The NASA FAR Supplement 1816.405-272 (a) states that award

fee evaluation periods including those for interim evaluations should be at least 6 months in length

and should in no case be longer than 12 months. Paragraph (b) states that a portion of the total

fee available shall be allocated to each of the evaluation periods. This allocation may result in an

equal or unequal distribution of fee among the periods. The contracting officer should consider

the nature of each contract and the incentive effects of fee distribution in determining the

appropriate allocation structure.

NASA Award Fee Contracting Guide. NASA Award Fee Contracting Guide, paragraph

3.5.1, states that one of the benefits of using award fee contracts is improved communications

between the Government and contractor personnel. The Guide cautions that this benefit may be

jeopardized if evaluation periods are too lengthy.

“Lookback” Award Fee to Assess Long-Term Performance Trends

The “lookback” award fee evaluations will be based on the contractor’s success in implementing

the IOA as submitted in the contractor’s proposal and as subsequently documented in data

requirements description 2.3-b, “Architecture Baseline.” This evaluation is intended to assess

long-term performance trends and its impact on the program. The “lookback” award fee

evaluation will not address specific contractor actions or accomplishments, but will assess the

overall long-term aggregate effects of such activities on the contract as a whole.
13

Evaluation Periods Longer Than 2 Years

The CSOC procurement development team did not establish appropriate evaluation periods in the

CSOC Award Fee Plan for the “lookback” award fee provision. The two periods for the

“lookback” award fee evaluation under the basic contract are 217 and 3 years, respectively.

According to the award fee distribution schedule in the Plan, the contractor will not be evaluated

and will not have the opportunity to earn “lookback” award fee until the end of these periods.

This distribution schedule does not provide an effective incentive to the contractor to improve

poor performance or to continue good performance.

Guidance Specifies Approval Needed for Deviations

The procurement development team did not follow guidance in the NASA FAR Supplement or

the NASA Award Fee Contracting Guide in developing evaluation periods for the “lookback”

provision of the CSOC Award Fee Plan. Further, the procurement development team did not

obtain a waiver for establishing periods longer than those specified in the NASA FAR

Supplement. FAR 1.403, “Individual Deviations,” requires that justification and agency approval

of deviations shall be documented in the contract file. The NASA FAR Supplement 1801.471,

“Procedures for Requesting Deviations,” lists the minimum elements that shall be contained in the

request and specifies that the request shall be submitted to the NASA Headquarters Office of

Procurement. The minimum requirements are shown in Appendix D.

Johnson procurement officials stated that the procurement development team submitted to the

NASA Headquarters Procurement Director an acquisition strategy that included award fee

periods longer than 1 year. Because the NASA Headquarters Procurement Director did not

object, the team concluded that this approach was approved. Paragraph 1807.103(d)(v) of the

NASA FAR Supplement states that approval of an acquisition plan does not constitute approval

of any special conditions or special clauses that may be required unless the plan so specifies and

the individual having approval authority is a signatory of the plan. The contract file documentation

for the acquisition strategy consists of briefing charts, minutes of the acquisition strategy meetings,

and memoranda to the file. None of these documents include specific reference to the evaluation

periods of the “lookback” award fee provision being longer than the 12 months prescribed by the

NASA FAR Supplement.

Effective Incentive for the Contractor

The CSOC procurement development team intended to assess long-term aggregate effects of the

contractor’s activities through the “lookback” award fee. However, lengthy periods for

evaluating performance jeopardize the benefit of improved communications between Government

and contractor personnel. To emphasize long-term performance, Johnson

 

17 The initial evaluation period is extended by 3 months to include the contract phase-in period from October 1, 1998,

through December 31, 1998.
14

procurement officials can provide an incentive for the contractor by establishing periods no longer

than 12 months and allocating a larger amount of award fee to the last period of the contract.

Recommendation, Management’s Response, and Evaluation of

Response

5. The Associate Administrator for Space Flight should direct the CSOC contracting

officer to revise evaluation periods to be no longer than 12 months for the

“lookback” award fee provision of the CSOC Award Fee Plan and to allocate total

available award fee under this provision so as to emphasize long-term performance.

Management's Response. Nonconcur. The award fee plan to establish both a biannual award

fee and a “lookback” award fee was presented in the Acquisition Strategy Meeting and approved

by NASA Headquarters. The NASA Office of Procurement provided its approval of the

Request for Proposal that contained the “lookback” award fee feature. The unequal distribution

of the award fee pool provided what management considers an incentive for the contractor to

provide long-term improvements related to the IOA and for the Government to adequately

evaluate those long-term performance trends. Furthermore, Johnson requested a deviation from

the NASA FAR Supplement 1816.405-273(a), which states " . . . all evaluations are final, and

the contractor keeps the fee earned in any period regardless of the evaluations of subsequent

periods. Unearned award fee in any given period in a service contract is lost and shall not be

carried forward or 'rolled-over' into subsequent periods."

Evaluation of Response. Management's comments are nonresponsive to the recommendation.

The deviation request was not located and provided to the auditors for review until after issuance

of the draft report. We subsequently reviewed the document and the accompanying handwritten

comments from the Associate Administrator for Procurement. Although the comments do not

raise any objections to the 2- and 3-year periods provided in the deviation request, we found no

formal approval for the deviation. Even if the evidence clearly indicated approval of the lengthy

periods, we still maintain our position that this arrangement does not provide an effective incentive

to the contractor. We agree with the unequal distribution of the award fee pool described in the

response and encourage this technique to emphasize long-term performance rather than the use of

periods longer than 12 months. We request that management reconsider its position and provide

additional comments.
15

Finding D. Updates to Architecture Baseline

The contract data requirement description 2.3-b, “Architecture Baseline,” does not require

revisions or progress reports beyond the initial submission to NASA management. The

procurement development team did not anticipate the need for regular status updates on the IOA.

As a result, NASA cannot ensure that supporting infrastructure and capabilities are maintained to

sustain product delivery activities. Further, without current information, NASA management

cannot measure progress toward IOA goals in order to determine the appropriate amount of

award fee and to effectively motivate the contractor.

Guidance Regarding Architecture Baseline

NASA Procedures and Guidelines. NASA Procedures and Guidelines 7120.5A, “NASA

Program and Project Management Processes and Requirements,” governs the formulation,

approval, implementation, and evaluation of all Agency programs and projects. In section

2.3.5.1(c), “Deliver Products and Services,” the guidance specifies that one of its purposes is to

ensure that supporting infrastructure and capabilities are maintained to sustain product delivery

activities. Further, it is the responsibility of the program/project office and the contracting officer

to monitor performance in order to ensure timely identification of all project performance

problems. Monitoring performance ensures that value is received commensurate with funds

expended and that the contractor complies with the terms of the contract.

CSOC Award Fee Plan. The CSOC Award Fee Plan states that progress against the

contractor’s architecture will be evaluated and scored. The contractor will be evaluated for the

long-term viability of its IOA plans and approaches based on their impact to program goals and

objectives.

CSOC Statement of Work. The CSOC statement of work provides that the contractor shall

document the current architecture and planned architecture changes in the data requirement

description 2.3-b, “Architecture Baseline.” The purpose of the document is to describe the

contractor’s proposed architecture that will provide the space operations services and the manner

in which the contractor will manage the architecture’s planned evolution over time. The current

data requirement description does not require progress reports beyond the initial submission.

After the initial baseline submission, at the end of contract phase-in on December 31, 1998,

updates are required only as necessary.

Need for Revised Architecture Baseline

When the procurement development team drafted the data requirement description 2.3-b, the

team did not anticipate the need for status updates of the architecture baseline. After the

contractor submitted the initial baseline, the team envisioned that the contractor would

commercialize the CSOC services and would no longer need the NASA infrastructure.

However, the contractor has not commercialized the CSOC services and still utilizes the NASA
16

infrastructure.18 Therefore, periodic updates of the contractor changes to the IOA baseline are

necessary. Without current information, NASA cannot measure progress toward IOA goals in

order to determine the appropriate amount of award fee and to effectively motivate the

contractor.

The contractor submitted a progress report19 on April 19, 2000, and plans to submit quarterly

reports in the future. However, the contract data requirements description has not been revised to

require regular updates to the IOA.

Recommendation, Management’s Response, and Evaluation of

Response

6. The Associate Administrator for Space Flight should direct the CSOC contracting

officer to determine the appropriate frequency for the contractor’s submission of the

architecture baseline, and revise the contract data requirement description 2.3-b,

"Architecture Baseline," to reflect this change.

Management's Response. Concur. The architecture baseline should be updated and

submitted annually. The contracting officer's technical representative is developing a revised data

requirement description to further clarify the contractor's requirements to update its architecture

baseline annually.

Evaluation of Response. The action planned is responsive to the recommendation. The

recommendation is considered resolved but will remain undispositioned and open until the agreedto corrective action is completed.

 

18 NASA still owns about 98 percent of the CSOC architecture, and only 2 percent of the architecture is new commercial

endeavors.

19 The first progress report covered IOA cost savings management, risk management, status of projects, and other related

topics.
17

Finding E. Cost Control in Award Fee Plan

The current CSOC Award Fee Plan does not sufficiently emphasize contractor cost control

performance. The procurement development team did not follow guidance in the NASA FAR

Supplement or the NASA Award Fee Contracting Guide in developing the cost control factor of

the CSOC Award Fee Plan. The current cost control factor is $1.6 million less than the amount

required for allocation to cost performance by the contractor. Consequently, NASA may receive

less than desired cost control performance.

Guidance for Evaluating Cost Control

NASA FAR Supplement. The NASA FAR Supplement 1816.405-274, “Award Fee

Evaluation Factors,” states that cost control shall be an evaluation factor in all cost-plus-awardfee contracts. When explicit performance evaluation factor weightings20 are used, cost control

shall be no less than 25 percent of the total weighted evaluation factors. Also, the NASA Award

Fee Contracting Guide, paragraph 3.4.3, states that cost control should always be a substantial

factor. When percentage weights are used, cost control will be at least 25 percent of the total

award fee.

CSOC Award Fee Plan. The CSOC Award Fee Plan provides for two award fee pools to be

administered separately under the contract. A 6-month pool will be awarded at the end of each

6-month evaluation period; a “lookback” pool will be awarded at the end of years 2, 5, 7, and

10. The total award fee available under the basic contract is $162.0 million. The plan assigns 80

percent of the total award fee, about $129.6 million, to the 6-month evaluation pool, and 20

percent, about $32.4 million, to the “lookback” evaluation pool. Two areas are included in the 6-

month evaluations. The managerial and technical performance area is weighted 70 percent, while

cost performance is weighted 30 percent. However, when compared to the total available award

fee, only 24 percent (.30 X .80 = .24) of the total award fee is assigned to cost performance. The

appropriate allocation of total award fee available to cost control is 25 percent, or $40.5 million.

The allocation of only 24 percent, or $38.8 million, is $1.6 million less than the appropriate

amount.21

Emphasis on Cost Control

The procurement development team did not follow guidance in the NASA FAR Supplement or

the NASA Award Fee Contracting Guide in developing the cost control factor of the CSOC

 

20 In addition to identifying how performance will be evaluated and measured, the performance evaluation plan should

indicate the relative priorities assigned to the various performance areas and evaluation factors or subfactors. This may be

accomplished by using percentage weights for the areas, factors, or subfactors.

21 The appropriate allocation of total award fee available to cost control, 25 percent, is $40.5 million (.25 X $162.0

million). The allocation of only 24 percent, $38.8 million (.24 X $162.0 million) is $1.6 million less than the appropriate

amount ($40.5 - $38.8 = $1.6 million).
18

Award Fee Plan. The Agency policies specify that when percentage weights are used, cost

control will be at least 25 percent of the total award fee.

The lower total award fee may not sufficiently motivate the contractor, and NASA may receive

less than the desired cost performance. If the contracting officer does not revise the weighting for

total award fee, then the effective cost control factor is only 24 percent. The NASA FAR

Supplement and the NASA Award Fee Contracting Guide require a cost control factor of 25

percent.

Recommendation, Management’s Response, and Evaluation of

Response

7. The Associate Administrator for Space Flight should direct the CSOC contracting

officer to revise the CSOC Award Fee Plan to reflect a cost control factor of at least

25 percent of the total award fee as required by NASA policy.

Management's Response. Nonconcur. Management stated it is utilizing the same award fee

weighting guidelines in both the “lookback” portion of the award fee and the 6-month evaluations.

Therefore, 30 percent of the overall award fee is being dedicated to cost performance rather than

the 24 percent noted in the draft audit report.

Evaluation of Response. Although management nonconcurred, it has taken corrective action

that partially addresses the recommendation. At the conclusion of our field work, management

had not yet specified or communicated to the contractor the factors or weightings that would be

included in the evaluation of performance for the "lookback” portion of award fee. Consequently,

at that time, the cost control factor was less than 25 percent, as stated in the draft report. Prior to

issuance of the draft report, management told us that it intended to assign the same award fee

weighting used in the 6-month evaluations to the "lookback" portion of the award fee. We

responded that this action, if taken, would satisfy our concerns. On July 28, 2000, management

issued a letter to the contractor specifying that a weighting equivalent to that assigned for the 6-

month evaluations would be assigned to the "lookback" portion of the award fee. As a result, 30-

percent of the total award fee will be dedicated to cost performance. We consider the issuance

of management's letter to the contractor, specifying a 30-percent weighting for cost under both the

6-month and "lookback" portion of the award fee to be partial corrective action for this

recommendation. If the weightings presented in the letter are applicable to all “lookback” award

fee periods, this information should be clarified in the Award Fee Plan. We request that

management reconsider its position and provide additional comments.
19

Appendix A. Objectives, Scope, and Methodology

Objectives

The overall objective, which will be addressed in a separate report, was to determine whether

the CSOC goals were being accomplished. This report identifies conditions regarding a costbenefit analysis and award fee structure that warrant timely action by NASA management.

Scope and Methodology

The audit included a review of the rationale and plan for consolidation and award fee structure.

We reviewed budget and spending data for fiscal year 1999 provided by the Space Operations

Management Office. In addition, we reviewed NASA guidance for developing award fee

plans. We interviewed program and contractor personnel to understand the history and the

current and planned progress of the procurement. We did not assess the reliability of

computer-processed data because we did not rely on computer-process to achieve the audit

objectives.

Management Controls Reviewed

We reviewed the following management controls:

• Office of Management and Budget Circular A-94, “Guidelines and Discount Rates

for Benefit-Cost Analysis of Federal Programs,” October 29, 1992, provides

guidance for conducting cost-benefit and cost-effectiveness analyses.

• March 13, 1997, NASA Deputy Administrator’s memorandum requiring

cost-benefit analyses addressed to Officials-in-Charge of Headquarters Offices;

Directors, NASA field installations; and Director, Jet Propulsion Laboratory.

• FAR 1.403 “Individual Deviations” states that justification and agency approval for

contract deviations shall be documented in the contract file.

• NASA FAR Supplement 1801.471, “Procedures for Requesting Deviations,” lists

the requirements for securing a deviation from established contracting procedures.

• FAR, Subpart 16.4, “Incentive Contracts,” provides guidance for developing

incentive contracts including cost-plus-award-fee contracts.

• NASA FAR Supplement 1816.405-273, “Award Evaluation,” describes

development of award fee evaluation factors.
20

Appendix A

• “The NASA Award Fee Contracting Guide,” December 2,1997, explains and

provides guidance for NASA’s award fee policy.

• NASA Procedures and Guidelines 7120.5A, “NASA Program and Project

Management Processes and Requirements,” governs formulation, approval,

implementation, and evaluation of all NASA programs and projects.

Audit Field Work

We performed the audit field work from October 1999 through April 2000. We conducted the

audit in accordance with generally accepted government auditing standards.
21

Appendix B. Elements of Benefit-Cost or Cost-Effectiveness Analysis

The Office of Management and Budget Circular No. A-94, “Guidelines and Discount Rates for

Benefit-Cost Analysis of Federal Programs, “ October 29, 1992, provides the following

guidance.

c. Elements of Benefit-Cost or Cost-Effectiveness Analysis.

1. Policy Rationale. The rationale for the Government program being examined should be clearly

stated in the analysis. Programs may be justified on efficiency grounds where they address market

failure, such as public goods and externalities. They may also be justified where they improve the

efficiency of the Government’s internal operations, such as cost-saving investments.

2. Explicit Assumptions. Analyses should be explicit about the underlying assumptions used to

arrive at estimates of future benefits and costs. In the case of public health programs, for example,

it may be necessary to make assumptions about the number of future beneficiaries, the intensity of

service, and the rate of increase in medical prices. The analysis should include a statement of the

assumptions, the rationale behind them, and a review of their strengths and weaknesses. Key data

and results, such as year-by-year estimates of benefits and costs, should be reported to promote

independent analysis and review.

3. Evaluation of Alternatives. Analyses should also consider alternative means of

achieving program objectives by examining different program scales, different methods

of provision, and different degrees of Government involvement . For example, in

evaluating a decision to acquire a capital asset, the analysis should generally consider:

(i) doing nothing; (ii) direct purchase; (iii) upgrading, renovating, sharing, or converting existing

government property; or (iv) leasing or contracting for services.

4. Verification. Retrospective studies to determine whether anticipated benefits and costs have

been realized are potentially valuable. Such studies can be used to determine necessary corrections

in existing programs, and to improve future estimates of benefits and costs in these programs or

related ones.

Agencies should have a plan for periodic, results-oriented evaluation of program effectiveness.

They should also discuss the results of relevant evaluation studies when proposing reauthorizations

or increased program funding.
22

Appendix C. CSOC Transition Schedule

 Schedule of Contracts

 Transitioned Under the Basic Contract

Contracts

By Center

Name Previous

Contractor

Scheduled

Transition

Date

Goddard Space Flight Center

5-31000 Consolidated Network and

Mission Operations Support

Allied Signal 9/25/98

5-31500 Systems, Engineering &

Analysis Support Services

Computer Sciences

Corporation

"

5-32153 Operation and Maintenance of

Radar Telecommunications

Allied Signal "

5-32617 Upper Atmospheric Research

Satellite (UARS)/Total Ozone

Mapping Spectrometer

(TOMS) Flight Operations

Lockheed Martin "

5-32660 Earth Observing System Data

and Operations (EDOS)

TRW 1/1/99

7/1/00

8 Months after

AM-1 launch

5-32700 Logistics Support Raytheon 4/1/99

5-60000 Earth Observing System Data

and Information System Core

System (EOSDIS Core)

Hughes/

Raytheon

11/1/02

Marshall Space Flight Center

8-60000 Program Information Systems

Mission Support

Computer Sciences

Corporation

5/1/99

8-44000 Utilization & Mission Support Lockheed Martin 8/19/01

Johnson Space Center

9-20000 Space Flight Operations

Contract

USA 9/25/98

9-18300 Mission Systems Maintenance

& Operations and Sustaining

Engineering

Lockheed

Martin

10/1/99

12/10/99

Jet Propulsion Laboratory

957890 Maintenance & Operations of

Deep Space Network

Allied Signal 9/25/98

958226 Mission & Computing Support OAO Corporation 1/1/00
23

 Appendix D. Procedure for Requesting Deviations

NASA FAR Supplement 1801.471, “Procedure for requesting deviations,” states:

(a) Requests for authority to deviate from the FAR or the NFS [NASA FAR Supplement] shall be

submitted by the Procurement Officer to the Headquarters Office of Procurement (Code HS).

(b) Each request for a deviation shall contain, as a minimum --

(1) Identification of the FAR or the NFS requirement from which a

deviation is sought;

(2) A full description of the deviation, the circumstances in which it will be

used, and the specific contract action(s) to which it applies;

 (3) A description of its intended effect;

(4) A statement as to whether the deviation has been requested previously

and, if so, the circumstances of the previous request;

(5) Identification of the contractor(s) and the contract(s) affected, including dollar

value(s);

(6) Detailed reasons supporting the request, including any pertinent background

information; and

(7) A copy of counsel's concurrence or comments.

(c) In addition to the information required by 1801.471(b), requests for individual

deviations from FAR cost principles under FAR section 31.101 should include a copy of the

contractor's request for cost allowance.
24

Appendix E. Management’s Response
25

Appendix E

See Appendix F,

OIG Comment 2
26

Appendix E

See Appendix F,

OIG Comment 1
27

Appendix E
28

Appendix E
29

Appendix E
30

Appendix E
31

Appendix E
32

Appendix E
33

 Appendix

E
34

Appendix F. OIG Response to Selected Management Comments

Management’s response discusses the OIG participation on the Source Evaluation Board (SEB)

and the support for the CSOC cost savings. Our evaluation of these comments follows:

Management's Comments. During the SEB process, the Procurement Development Team

developed a cost analysis utilizing the "should cost" baseline for the scope of work contained in the

CSOC Request for Proposal already developed by the Agency. The OIG participated in this

evaluation. The Procurement Development Team presented the plan for performing a “lookback”

award fee at the Acquisition Strategy Meeting. Throughout all of the reviews, the OIG provided

personnel who actively participated in the SEB activities.

1. OIG Comments. Management inaccurately characterizes the OIG's participation on the SEB.

The OIG assigned representatives to the SEB to provide audit advisory services only. The

representatives attended meetings and observed the activities of the SEB. The representatives were

shown data at a summary level but were not permitted to review the data in detail and did not

perform any analysis of the data reviewed. The OIG stated at the outset that the role of

representatives on the SEB was advisory only and that the OIG did not have a role in the

management or the decisionmaking processes of the team. Further, we clarified that our presence

did not alter our responsibilities under the Inspector General's Act to conduct independent audit and

investigations of NASA's programs and operations.

Management's Comments. NASA made extensive briefings to industry, Congress, and the

OMB regarding NASA's objective for the CSOC acquisition. In addition, fact-finding activities

were conducted. The NASA CSOC SEB developed a Government estimate based on NASA's

Fiscal Year 1999 Congressional Budget. Subsequently, the Agency established a "should cost"

baseline for the scope of work contained in the CSOC Request for Proposal. During the SEB

process, the Procurement Development Team performed a cost analysis and determined that

potential savings of $1.4 billion could be realized through the selection of Lockheed Martin Space

Operations Company (LMSOC) performing the CSOC contract. To date, NASA has determined

that it has realized savings when comparing LMSOC's cost performance against the contract

baseline.

2. OIG Comments. During the audit, the auditors requested evidence that the Agency performed

a cost-benefit analysis with quantitative data in accordance with Circular A-94. Program officials

told the auditors that while no formal cost-benefit analysis had been performed, various briefings,

fact finding, and other activities such as those provided by procurement officials and described in the

response satisfied the intent of a cost-benefit analysis. The Agency must perform a cost-benefit

analysis to justify the programmatic decision to undertake a major consolidation prior to beginning a

procurement activity.

However, a key element described in Circular A-94 is not present in the programmatic activities

described. Specifically, the Circular specifies that:
35

Appendix F

. . . analyses should be explicit about the underlying assumptions used to arrive at estimates of future

benefits and costs. The analysis should include a statement of the assumptions, therationale behind

them, and a review of their strengths and weaknesses. Key data and results, such as year-by-year

estimates of benefits and costs, should be reported to promote independent analysis and review.

In addition, management comments state that activities performed by the CSOC SEB satisfy the

requirement. However, the Government estimate and "should cost" baseline are procurement

activities that did not support examination of various alternatives and the decision to consolidate

multiple contracts. We reviewed the additional information provided by procurement officials

before preparing our evaluation of management's response and issuance of the final report. The

information also supports the procurement activity rather than the programmatic consideration that

led to the decision to consolidate contracts. Therefore, we stand by our earlier conclusion that the

required analysis was not performed.
36

Appendix G. Report Distribution

National Aeronautics and Space Administration (NASA) Headquarters

A/Administrator

AI/Associate Deputy Administrator

B/Chief Financial Officer

B/Comptroller

BF/Director, Financial Management Division

G/General Counsel

H/Associate Administrator for Procurement

HK/Director, Contract Management Division

HS/Director, Program Operations Division

J/Associate Administrator for Management Systems

JM/Acting Director, Management Assessment Division

L/Associate Administrator for Legislative Affairs

M/Associate Administrator for Space Flight

NASA Centers

Director, Goddard Space Flight Center

Director, Lyndon B. Johnson Space Center

 Director, Space Operations

Director, John F. Kennedy Space Center

Director, Marshall Space Flight Center

Jet Propulsion Laboratory

Chief Counsel, John F. Kennedy Space Center

Director, Space Operations, Lyndon B. Johnson Space Center

Non-NASA Federal Organizations and Individuals

Assistant to the President for Science and Technology Policy

Deputy Associate Director, Energy and Science Division, Office of Management and

 Budget

Branch Chief, Science and Space Programs Branch, Energy and Science Division, Office

 of Management and Budget

Associate Director, National Security and International Affairs Division, Defense

 Acquisitions Issues, General Accounting Office

Professional Assistant, Senate Subcommittee on Science, Technology, and Space
37

Appendix G

Chairman and Ranking Minority Member – Congressional Committees and

Subcommittees

Senate Committee on Appropriations

Senate Subcommittee on VA, HUD, and Independent Agencies

Senate Committee on Commerce, Science, and Transportation

Senate Subcommittee on Science, Technology, and Space

Senate Committee on Governmental Affairs

House Committee on Appropriations

House Subcommittee on VA, HUD, and Independent Agencies

House Committee on Government Reform

House Subcommittee on National Security, Veterans Affairs, and International Relations

House Committee on Science

House Subcommittee on Space and Aeronautics, Committee on Science

Congressional Member

Honorable Pete Sessions, U.S. House of Representatives
NASA Assistant Inspector General for Auditing

Reader Survey

The NASA Office of Inspector General has a continuing interest in improving the usefulness of

our reports. We wish to make our reports responsive to our customers’ interests, consistent

with our statutory responsibility. Could you help us by completing our reader survey? For your

convenience, the questionnaire can be completed electronically through our homepage at

http://www.hq.nasa.gov/office/oig/hq/audits.html or can be mailed to the Assistant Inspector

General for Auditing; NASA Headquarters, Code W, Washington, DC 20546-0001.

Report Title: Final Report on Audit of Consolidated Space Operations Contract

Report Number: Report Date: 

Circle the appropriate rating for the following statements.

Strongl

y

Agree

Agree Neutra

l

Disagre

e

Strongl

y

Disagre

e

N/A

1. The report was clear, readable, and

logically organized.

5 4 3 2 1 N/A

2. The report was concise and to the point. 5 4 3 2 1 N/A

3. We effectively communicated the audit

objectives, scope, and methodology.

5 4 3 2 1 N/A

4. The report contained sufficient

information to support the finding(s) in a

balanced and objective manner.

5 4 3 2 1 N/A

Overall, how would you rate the report?

• Excellent • Fair

• Very Good • Poor

• Good

If you have any additional comments or wish to elaborate on any of the above

responses, please write them here. Use additional paper if necessary. 

 

 

 

 
How did you use the report? 

 

 

 

 

 

 

How could we improve our report? 

 

 

 

 

 

 

How would you identify yourself? (Select one)

Congressional Staff Media

NASA Employee Public Interest

Private Citizen Other: 

Government: Federal: State: Local: 

May we contact you about your comments?

Yes: ______ No: ______

Name: ____________________________

Telephone: ________________________

Thank you for your cooperation in completing this survey.
Major Contributors to the Report

Daniel Samoviski, Program Director, Earth and Space Science Audits

Esther Judd, Program Manager/Auditor-in-Charge

Clara L. Seger, Auditor

Mary S. Anderson, Auditor

Joe Kroener, Procurement Analyst

Nancy Cipolla, Report Process Manager

Iris Purcarey, Program Assistant