-----------------------------------------------------------------------
  The Department's Management of the EDNet Contract

  -----------------------------------------------------------------------

## **FINAL AUDIT REPORT**

![](media/image1.pct)

**ED-OIG/A19G0009**

**April 2007**

+-------------------------+----------------------+---------------------+
| Our mission is to       | !                    | U.S Department of   |
| promote the efficiency, | [](media/image2.bmp) | Education           |
| effectiveness, and      |                      |                     |
| integrity of the        |                      | Office of Inspector |
| Department\'s programs  |                      | General             |
| and operations.         |                      |                     |
|                         |                      | Washington, DC      |
+-------------------------+----------------------+---------------------+

**NOTICE**

> Statements that managerial practices need improvements, as well as
> other conclusions and recommendations in this report represent the
> opinions of the Office of Inspector General. Determinations of
> corrective action to be taken will be made by the appropriate
> Department of Education officials.
>
> In accordance with the Freedom of Information Act (5 U.S.C. § 552),
> reports issued by the Office of Inspector General are available to
> members of the press and general public to the extent information
> contained therein is not subject to exemptions in the Act.

THE INSPECTOR GENERAL

![](media/image3.png){width="0.9in" height="0.8604166666666667in"}April
17, 2007

##### Memorandum {#memorandum .Enclosure-Number}

**TO:** David Dunn

Chief of Staff

Office of the Secretary

Thomas L. Sipes for

**FROM:** John P. Higgins, Jr. /s/

**SUBJECT:** Final Audit Report

The Department's Management of the EDNet Contract

Control Number ED-OIG/A19G0009

Attached is the subject final audit report that covers the results of
our review of the Department of Education's (Department) management of
the Education Network (EDNet) contract. We received your comments
concurring with the findings and associated recommendations in our draft
report.

Corrective actions proposed (resolution phase) and implemented (closure
phase) by your

office(s) will be monitored and tracked through the Department's Audit
Accountability and

Resolution Tracking System (AARTS). Department policy requires that you
develop a final corrective action plan (CAP) for our review in the
automated system within 30 days of the issuance of this report. The CAP
should set forth the specific action items, and targeted completion
dates, necessary to implement final corrective actions on the findings
and recommendations contained in this final audit report.

In accordance with the Inspector General Act of 1978, as amended, the
Office of Inspector

General is required to report to Congress twice a year on the audits
that remain unresolved after

six months from the date of issuance.

In accordance with the Freedom of Information Act (5 U.S.C. §552),
reports issued by the Office

of Inspector General are available to members of the press and general
public to the extent

information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given us during this review. If you have
any questions, please

call Michele Weaver-Dugan at (202) 245-6941.

Enclosure

cc: William Vajda, Chief Information Officer, Office of the Chief
Information Officer (OCIO)

> Lawrence Warder, Chief Financial Officer, Office of the Chief
> Financial Officer (OCFO)
>
> Michell Clark, Assistant Secretary, Office of Management (OM)
>
> Phil Link, Director, Executive Secretariat
>
> Glenn Perry, Director, Contracts and Acquisitions Management (CAM),
> OCFO
>
> Sally Budd, Acting Chief of Staff and Audit Liaison Officer (ALO),
> OCIO

Jeanie Banks, ALO, OM

Cynthia Bond-Butler, ALO, CAM, OCFO

TABLE OF CONTENTS

####### Page

[**EXECUTIVE SUMMARY 1**](#__RefHeading___Toc156030071)

[BACKGROUND 3](#__RefHeading___Toc156030072)

[AUDIT RESULTS 5](#__RefHeading___Toc156030073)

[FINDING NO. 1 -- EDNet Contract Structure and Subsequent Changes Were
Not Effective in Managing Contractor Performance.
6](#__RefHeading___Toc156030074)

[FINDING NO. 2 -- Department Controls Did Not Ensure That the Contractor
Provided the Quality and Services Required in the Contract.
13](#refheading___toc156030075)

[FINDING NO. 3 -- Contract Administration Practices Were Not Effective.
23](#__RefHeading___Toc156030076)

[OBJECTIVE, SCOPE, AND METHODOLOGY 33](#__RefHeading___Toc156030077)

**ATTACHMENTS**

> Attachment 1 -- Original Service Level Agreements 35
>
> Attachment 2 -- Modified Service Level Agreements 37
>
> Attachment 3 -- Capability Maturity Model Integration (CMMI) Levels 38
>
> Attachment 4 -- Category C Staff Labor Categories and Qualifications
> 39
>
> Attachment 5 -- List of Acronyms 40
>
> Attachment 6 -- Department Response to Draft Report

[]{#__RefHeading___Toc156030071 .anchor}EXECUTIVE SUMMARY

The objective of our audit was to determine the effectiveness of the
Department of Education's (Department) management of the Education
Network (EDNet) contract. The Department awarded the EDNet contract,
effective May 1, 2005, to acquire Information Technology (IT) network
services. The main goals of the EDNet contract were to improve all
services provided to the Department customers and to lower costs to the
Department through IT integration.

The EDNet contract is managed jointly by the Contracts and Acquisitions
Management (CAM) office, under the Department's Office of the Chief
Financial Officer (OCFO), and the Office of the Chief Information
Officer (OCIO). The principal parties involved in managing the EDNet
contract are the Contracting Officer (CO) and Contract Specialist from
OCFO/CAM, and the Program Manager and Contracting Officer's
Representative from OCIO. The Department separately acquired the
services of an Independent Verification and Validation (IV&V) contractor
to provide assistance in determining whether the EDNet contractor's
performance conforms to contract requirements.

Overall, we found that significant improvement is needed in the
Department's management of the EDNet contract. Our audit found the EDNet
contract structure and subsequent changes were not effective in managing
contractor performance. Specifically, the structure of the EDNet
contract did not provide effective performance incentives or
disincentives to allow for timely enforcement of an acceptable level of
performance. We also found a contract modification was not fully
evaluated to consider whether a reduction in cost was appropriate for
the reduced level of effort required by the contractor to meet
acceptable levels of performance. As a result, the contractor had little
incentive to perform during the base year (July 2005 through June 2006)
or in the final year of the contract. Services provided during the base
year were rated as unacceptable, and the Department's ability to improve
performance was hampered.

We also found that the Department's controls did not ensure the
contractor provided the quality and services required by the contract.
Department officials also provided inappropriate direction to the
contractor that changed the scope, requirements and/or due dates for
some deliverables. As a result, the Department paid for a quality or
level of services it did not receive, an asset management system that
did not meet contract requirements and duplicated some activities of
other Department contractors, and did not timely receive a critical tool
for monitoring contractor compliance with Service Level Agreements. In
addition, because a Department official directed the contractor to
develop and host the tool on its own servers, the Department may have
difficulty enforcing its rights to own the tool.

Finally, we noted the Department's contract administration practices
were not effective. These practices did not ensure appropriate personnel
from within the Department were assigned to manage and oversee the
contract. In addition, Department staff did not always comply with
applicable Federal regulations and Department policies and procedures.
As a result, the Department does not have complete documentation of the
contractor's performance and its monitoring of the EDNet contract. The
Department also lacks assurance that the contractor fully complied with
the terms and conditions of the contract, and provided all required
deliverables for which the Department paid. In the event of a future
dispute, the Department's position would be weakened by this lack of
documentation.

To correct the weaknesses identified, we recommended that the
Department, among other things:

-   Establish and implement a corrective action plan and schedule to
    address the contractor's performance issues and ensure that minimum
    quality levels of services are provided under the contract.

-   Based on the corrective action plan implemented, modify the current
    EDNet contract to obtain an equitable adjustment for the Department
    for any reduced level of effort required to meet performance rating
    standards.

```{=html}
<!-- -->
```
-   Ensure that OCIO staff do not provide direction to the contractor
    that changes the scope, delivery dates, or other contract terms and
    conditions without going through the CO.

```{=html}
<!-- -->
```
-   Reinforce to the contractor its responsibility in ensuring that any
    direction provided that impacts the scope, delivery dates, or other
    terms and conditions, is provided by the CO prior to making any
    changes.

```{=html}
<!-- -->
```
-   Confirm that ownership of the Management Information Dashboard
    (Dashboard) remains with the Department. If possible, take action to
    transition the Dashboard to the Department's systems so that the
    Department will not lose control of this tool in the event of a
    change in contractors.

```{=html}
<!-- -->
```
-   Immediately develop and implement a contract monitoring plan for the
    EDNet contract to include the roles and responsibilities of all
    involved OCIO staff, OCFO/CAM staff, and the IV&V contractor.
    Conduct a meeting with all involved parties to ensure that all are
    familiar with their responsibilities, as required by OCFO procedure.

```{=html}
<!-- -->
```
-   Ensure that appropriate resources, whether Department staff, IV&V,
    and/or other contract assistance, are assigned so that contract
    monitoring responsibilities can be effectively accomplished.

```{=html}
<!-- -->
```
-   Establish a process to track receipt of deliverables, review the
    deliverables for compliance with contract requirements, and provide
    written recommendation to the CO for acceptance or rejection of
    deliverables. Ensure that constructive acceptance does not occur and
    harm the Department's interests under the contract.

```{=html}
<!-- -->
```
-   Establish and implement a process to ensure that all significant
    actions and monitoring under the contract are adequately documented,
    organized, and accessible to all involved parties. Ensure that CAM
    and program office files contain complete information, whether in
    electronic format or hard copy, to constitute a complete history of
    the contract that is accessible to all involved parties.

In its response to the draft audit report, the Department concurred with
the findings and associated recommendations and provided a corrective
action plan to address each recommendation. The Department's response is
included as Attachment 6 to this report.

[]{#__RefHeading___Toc156030072 .anchor}BACKGROUND

The Department of Education (Department) awarded the Education Network
(EDNet) contract effective May 1, 2005, to acquire Information
Technology (IT) network services. The EDNet contract includes three
principal objectives:

-   Improving all services provided to the Department customers, as
    measured by meeting Service Level Agreements (SLAs)[^1] and ongoing
    independent third party customer satisfaction surveys.

-   Lowering cost and demonstrating better service through the adoption
    of Capability Maturity Model Integration (CMMI) Level 3 processes
    and competence.

-   Continuously lowering costs and improving customer satisfaction
    going forward for all services provided to the Department customers.

In order to achieve the principal objectives outlined above, the EDNet
contractor provides IT support services in three areas:

-   Category A -- Managed services, billed on a firm fixed price per
    unit basis, which includes server maintenance, messaging (E-mail and
    Blackberry), and end-user support for hardware and software.

-   Category B -- Resources including, but not limited to, subject
    matter expertise billed on a fixed price and/or time and materials
    basis.

-   Category C -- Qualified labor on a time and materials basis for
    continuous staffing of services whose processes will be managed by
    the Department.

The EDNet contract is a performance-based contract under the Office of
the Chief Information Officer (OCIO). The EDNet contractor's performance
is measured semiannually in categories such as SLA performance,
achievement of small business goals, and customer satisfaction survey
results. Department officials stated a performance-based contract, which
included SLAs, was adopted because they believed it added value when
compared to a time and materials contract and it allowed related
incentives to encourage superior contractor performance. Based on
performance beginning with the second year of the contract, the
contractor is eligible for award of future years of the contract. On
July 1, 2006, the Department exercised the first option to extend the
EDNet contract for an additional year. As a result, the funding for the
contract was increased to \$45,801,743. Payments under the EDNet
contract through July 2006 totaled \$20,609,890.

The EDNet contract is managed jointly by the Contracts and Acquisitions
Management (CAM) office under the Department's Office of the Chief
Financial Officer (OCFO), and OCIO. The principal parties involved in
managing the EDNet contract are the Contracting Officer (CO) and
Contract Specialist (CS) from OCFO/CAM, and the Program Manager (PM) and
Contracting Officer's Representative (COR) from OCIO. The Department
separately acquired the services of an Independent Verification and
Validation (IV&V) contractor to provide assistance in determining
whether the EDNet contractor's performance conforms to contract
requirements.

[]{#__RefHeading___Toc156030073 .anchor}AUDIT RESULTS

Our audit disclosed significant improvement is needed in the
Department's management of the EDNet contract. Specifically, we found:

1.  The EDNet contract structure and subsequent changes were not
    effective in managing contractor performance,

2.  Department controls did not ensure the contractor provided the
    quality and services required in the contract, and

3.  Contract administration practices were not effective.

The structure of the EDNet contract did not provide effective
performance incentives or disincentives to allow for timely enforcement
of an acceptable level of performance. As a result, the contractor had
little incentive to perform and the services provided were rated as
unacceptable during the base year. Further, a contract modification
executed after the base year reduced the level of effort required to
meet acceptable levels of performance. The modification did not provide
the Department with corresponding consideration through reduced costs.

The Department's controls over contract management did not ensure that
the contractor provided contractually required items. As a result, the
Department paid for a quality and services it did not receive. For
example, in Category C, the time and materials staff did not meet the
requirements of the labor categories charged. Additionally, an asset
management system (AMS) was not provided timely and did not meet all
contract requirements. Finally, a critical tool for monitoring
contractor performance was not provided timely and did not meet all
contract requirements.

Overall, the contract administration practices followed for the EDNet
contract were not effective. These weaknesses included the lack of a
contract monitoring plan, failure to routinely review deliverables, and
contract files that did not record significant actions taken under the
contract. As a result, the Department did not have complete
documentation of the contractor's performance and its own monitoring of
the contract. As such, the Department lacks assurance that the
contractor fully complied with the terms and conditions of the contract,
and provided all services and deliverables required and for which the
Department paid.

In its response to the draft audit report, the Department concurred with
the findings and associated recommendations and provided a corrective
action plan to address each recommendation. The Department's response is
included as Attachment 6 to this report.

[]{#__RefHeading___Toc156030074 .anchor}FINDING NO. 1 -- EDNet Contract
Structure and Subsequent Changes Were Not Effective in Managing
Contractor Performance.

**[Summary]{.underline}**

The initial EDNet contract structure and subsequent changes were not
effective in managing contractor performance. Specifically, we found:

a.  The structure of the EDNet contract did not provide effective
    performance incentives or disincentives to allow for timely
    enforcement of an acceptable level of performance, and

b.  A contract modification executed after the base year reduced the
    level of effort required to meet acceptable levels of performance
    under SLAs without receiving corresponding consideration through
    reduced costs.

This occurred because the process for developing the performance
evaluation plan did not adequately consider the need for more immediate
incentives or disincentives to ensure adequate performance and an
acceptable level of service. With respect to the modification,
Department staff did not fully evaluate the impact of the changes and
consider whether a reduction in cost was appropriate for reduced effort
by the contractor.

As a result, the contractor had little incentive to meet performance
requirements during the base year and during the final year when the
contactor is not eligible for award of any additional years. Services
provided during the base year were rated as unacceptable, and the
Department's ability to improve performance under the contract through
the use of incentives or disincentives was limited. In addition, the
modification executed significantly increased the contractor's chances
of obtaining a higher performance rating without increasing its actual
level of effort or performance.

**Issue 1a -- The structure of the EDNet contract did not provide
effective performance incentives or disincentives to allow for
enforcement of a minimum level of performance.**

The structure of the EDNet contract did not provide effective
performance incentives or disincentives to allow for timely enforcement
of a minimum level of performance. While the contract included an
incentive related to award of future contract performance years, this
approach provided only limited incentive during the base year.

The EDNet contract included a complex performance evaluation process
based on the contractor's ability to satisfy SLAs and other factors. The
following is an overview of the performance evaluation process contained
in the original contract:

-   Performance is evaluated every six months, and the two scores
    averaged to determine the final score for the year. The performance
    rating is based on several factors, including 50 points allocated to
    the contractor's performance in meeting SLAs.[^2] An additional 50
    points are allocated to inputs from key customers, end users, and IT
    management, for a total of 100 points.

-   The annual score is applied to an overall rating table established
    in the contract to quantify contractor performance.

-   An average of 85-93 points is defined in the contract as "Superior"
    performance, and a score of 85 represents the minimum level of
    performance required to allow the contractor to receive future award
    years.

-   An average of 70-84 points is defined in the contract as
    "Satisfactory." The contract further states that 84 points is the
    "expressed level of satisfactory performance that can be expected
    from a good contractor." However, attaining this score does not
    qualify the contractor for future award years.

-   Failure to meet specific SLAs results in the assignment of demerit
    points to the contractor. The number of related demerit points is
    based on the reporting frequency and disincentive level assigned to
    the SLA.

-   During the base year, SLA demerit points were converted to
    disincentive points in a ratio of 5:1, where 5 demerit points yield
    1 disincentive point. At the end of the six-month rating periods,
    the total number of disincentive points is subtracted from an
    initial pool of 50 possible SLA performance points to calculate the
    contractor's score for SLA performance. These point totals, plus the
    point total from other factors, are combined to determine the
    overall point total out of 100 possible points.

-   Performance in the 2005-2006 base year does not result in a
    determination of award of future work.

-   A one-year lag is built into the contract between the performance
    evaluation year and the award of future years. For example,
    "Superior" performance in 2006-2007 could result in award of work
    for 2008-2009.

-   Should the contractor not achieve the minimum average score of 85 in
    2006-2007, it would still have the contract for 2007-2008, to allow
    time for the Department to recompete and transition to a new
    contractor.

The contractor's self-assessment of its performance during the base year
averaged 35.5 points. The Department's assessment of the contractor's
performance over the same period averaged 31 points. The Department's
assessment gave the contractor a performance score of 0 for its
achievement of SLA performance standards during the base year. The
contract defines a score of 49 and below as "Unacceptable" and states,

> Performance of 49 and below is indicative of serious mismanagement,
> negligence and/or incompetence. Continued performance at this level
> will require Education to consider terminating the order for default.

While some interim reporting of SLAs and other factors is conducted, the
Department lacks more immediate means of ensuring performance meets
standards. An annual scoring with the determination of an
incentive/disincentive for work that would actually occur one year later
does not provide the Department with timely remedies for poor
performance or ensure adequate performance in each SLA or performance
area. The contract requires corrective action at the "Substandard"
level, 50-69 points, one level above "Unacceptable," as follows,

> Performance between 50 and 69 points requires the Contractor to
> establish a deficiency correction plan and schedule to take the
> necessary corrective actions. Implementation of the plan and schedule
> will be tracked by the COR and Contracting Officer.

No deficiency correction plan is required by the contract at the lowest
level of performance, or at any other level other than the "Substandard"
level of 50-69 points, further hampering the Department's ability to
ensure the contractor takes corrective action and improves performance.

*Federal Acquisition Regulation* (FAR), Subpart 16.401(a), states,

> . . . Incentive contracts are designed to obtain specific acquisition
> objectives by. . . (2) Including appropriate incentive arrangements
> designed to (i) motivate contractor efforts that might not otherwise
> be emphasized; and (ii) discourage contractor inefficiency and waste.

FAR Subpart 16.402-2 states,

> \(b\) To the maximum extent practicable, positive and negative
> performance incentives shall be considered in connection with service
> contracts for performance of objectively measurable tasks when quality
> of performance is critical and incentives are likely to motivate the
> contractor.
>
> \(c\) Technical performance incentives may be particularly appropriate
> in major systems contracts, both in development (when performance
> objectives are known and the fabrication of prototypes for test and
> evaluation is required) and in production (if improved performance is
> attainable and highly desirable to the Government).

OCIO staff stated that the Department had not had any prior experience
with SLA-based performance contracts, so this was a new concept. OCIO
staff further stated the performance-based statement of work and
performance evaluation system were developed based on the work performed
by a consulting firm, and suggestions and feedback received from
potential bidders. Overall, the process followed to develop the
structure of the EDNet contract did not adequately consider the need for
more immediate incentives or disincentives to ensure adequate
performance and an acceptable level of service.

The structure of the performance evaluation system does not provide the
Department with an established method to impose disincentives for poor
performance or provide incentives for superior performance. Amounts paid
to the contractor do not have a direct relationship to performance. The
contractor's performance in the base year was at the unacceptable level,
yet the contractor received the same payment for its services that it
would at any other level. The Department paid the full price for
services that did not meet acceptable levels. Since performance in the
base year did not count toward the award of future years under the
contract, the contractor had no incentive to meet higher performance
levels. For future years, the contractor has no incentive to perform at
a level greater than 85 points, since any performance above that amount
is not rewarded.

The Department's recourse for unacceptable performance is to withhold
payments for unsatisfactory performance, potentially entering a
protracted dispute with the contractor, or to end the contract by
terminating for default, or by not exercising option years or awarding
future years. This action would require the Department to recompete the
contract, resulting in additional costs to the Department, without any
reduction in contract costs. During the time period while the contract
is being recompeted, the incumbent contractor would have little
incentive to perform in accordance with contract standards.

In addition, the structure of the performance evaluation system was
potentially inefficient, as the contractor would not qualify for future
award years even if they achieve a "Satisfactory" performance level
during an annual rating period. Under this level of performance, the
Department would have to invest resources, time, and effort into a
competition process, and potentially select a new "unproven" contractor,
even if the current contractor had proven its ability to satisfactorily
fulfill contract terms.

**Issue 1b -- The contract was modified to reduce the level of effort
required under Service Level Agreements.**

After the base year, and effective July 1, 2006, the Department executed
a contract modification that materially reduced the effort required to
meet performance ratings without obtaining a corresponding reduction in
cost. As discussed above, the contractor is required to achieve defined
performance thresholds for specified SLAs. A matrix assigned a fixed
number of demerit points for each SLA performance shortfall based upon
its defined measurement frequency and disincentive value. For example,
failure to achieve the performance standard for a SLA with a weekly
measurement frequency and a high disincentive level would result in the
assessment of six demerit points. These demerit points were converted to
disincentive points by an established ratio, and the disincentive points
were in turn deducted from a pool of 50 available SLA performance
points.

The modification reduced the total number of SLAs from 43 in the
original contract to 26. In addition, the contract modification impacted
three key areas that materially reduced contract performance
requirements. Specifically, the modification (1) reduced the demerit to
disincentive point conversion ratio, (2) established disincentive point
caps for individual SLAs, and (3) reduced performance standards,
frequency measurements, and/or disincentive levels for certain SLAs.

[Demerit to Disincentive Point Conversion Ratio Was Reduced]{.underline}

#### As previously stated, the original conversion of demerits to disincentive points applied a ratio of 5:1. The modification changed this conversion to a 50:1 ratio, requiring a 10-fold increase in the number of demerits required to earn 1 disincentive point. Table 1 below compares the results for the base year performance periods using only the change from the 5:1 ratio to the 50:1 ratio. 

This change alone would have raised the contractor's SLA performance
score from 0 to 30 points for the base year (50 total points less 20 SLA
performance disincentive points per column D). The contractor earned 31
points in other rating categories such as customer satisfaction surveys
and achievement of small business goals. This resulted in a performance
score of 31 points during the base year (column E). Overall, with all
other factors remaining the same, the change in the ratio would have
resulted in an overall performance score of 61 points, rather than 31
points, a 97 percent increase. This would have raised the contractor's
score from "Unacceptable" to "Substandard" -- still not eligible for
future award years, but a better performance rating than under the
original methodology.

#### Table 1: Impact of Changes in Conversion Point Ratio in Contract Base Year 

+--------+------------+---------------+----------------+--------------+
| **Co   | **Total**  | **SLA**       | **SLA          | **Total      |
| ntract |            |               | Performance    | Performance  |
| T      | **Demerit  | **            | Points Earned  | Points       |
| erms** | Points     | Performance** | (50 Total      | Earned**     |
|        | (Base Year |               | Points Less    |              |
|        | Average)** | *             | Disincentive   | **(D + 31)** |
|        |            | *Disincentive | Points)**      |              |
|        |            | Points (5:1   |                |              |
|        |            | Original or   |                |              |
|        |            | 50:1          |                |              |
|        |            | Modified)**   |                |              |
+--------+------------+---------------+----------------+--------------+
| **A**  | **B**      | **C**         | **D**          | **E**        |
+--------+------------+---------------+----------------+--------------+
| Or     | 979        | 196 (B/5)     | 0              | 31           |
| iginal |            |               |                |              |
+--------+------------+---------------+----------------+--------------+
| Mo     | 979        | 20 (B/50)     | 30             | 61           |
| dified |            |               |                |              |
+--------+------------+---------------+----------------+--------------+

[Disincentive Point Caps Established for Individual SLAs]{.underline}

## The original contract did not limit the total number of disincentives that could be accumulated for repeated failures within an individual SLA, other than that negative SLA performance scores were not reported. The lowest score that could be reported for the 50 available SLA points was zero. In addition, certain SLAs originally contained multiple measures based on the number of servers in use to provide the service. As demonstrated in Table 1 above, column C, the contractor's performance could result in disincentive points that exceeded the 50 available SLA performance points. Under this methodology, the contractor's technical proposal estimated that a total of 34,243 disincentive points could be earned. 

## The modification limited the total number of disincentive points that could be assessed as equal to the 50 available SLA performance points. This was completed by establishing a maximum amount of disincentive points that could be earned for each individual SLA regardless of how many demerit points were earned. The established disincentive caps ranged from one to five points for each of the SLAs (for disincentive cap points assigned to the new SLAs, see Attachment 2). Overall, the caps limited the number of disincentive points that could be earned by one or more points for 12 of the 26 SLAs (46 percent). As shown in Table 2 below, the caps on the number of disincentive points that could be earned further increases the contractor's performance scores. 

## Using the revised methodology, the contractor would earn a performance score of 74 points during the base year (43 points per Table 2 column F + 31 points earned from other rating categories). This is an increase of 13 points from the 61 points earned when disincentive caps were not applied, and is an increase of 139 percent over the original methodology. Overall, this further increases the contractor's performance rating to a level of "Satisfactory." While this rating would still not qualify the contractor for award of future years, it would increase the contractor's performance by two rating levels, and significantly increase the contractor's ability to reach the "Superior" level through improvements in other rating factors.

#### **Table 2: Overall Impact of Methodology Change in Contract Base Year**

+------+--------+----------+--------+--------+------------+---------+
| **   | **T    | **SLA**  | **Red  | **Net  | **SLA      | **Total |
| Cont | otal** |          | uction | Disinc | P          | Perf    |
| ract |        | **Perfo  | in     | entive | erformance | ormance |
| Ter  | **D    | rmance** | Disinc | Po     | Points     | Points  |
| ms** | emerit |          | entive | ints** | Earned (50 | Earned  |
|      | Points | **Disi   | Points |        | Total      | (F +    |
|      | (Base  | ncentive | Due to | **(    | Points     | 31)**   |
|      | Year   | Points   | Cap**  | C-D)** | Less       |         |
|      | Aver   | (5:1     |        |        | Di         |         |
|      | age)** | Original |        |        | sincentive |         |
|      |        | or 50:1  |        |        | Points)**  |         |
|      |        | Mod      |        |        |            |         |
|      |        | ified)** |        |        |            |         |
+------+--------+----------+--------+--------+------------+---------+
| *    | **B**  | **C**    | **D**  | **E**  | **F**      | **G**   |
| *A** |        |          |        |        |            |         |
+------+--------+----------+--------+--------+------------+---------+
| Orig | 979    | 196      | 0      | 196    | 0          | 31      |
| inal |        | (B/5)    |        |        |            |         |
+------+--------+----------+--------+--------+------------+---------+
| Modi | 979    | 20       | 13     | 7      | 43         | 74      |
| fied |        | (B/50)   |        |        |            |         |
+------+--------+----------+--------+--------+------------+---------+

## We also noted inconsistencies in the manner in which the caps were applied in that certain SLAs with the same reporting frequency and disincentive levels had different cap values. For example, 6 of the 26 SLAs established under the modification were created with a weekly reporting frequency and a high disincentive value. Of these, two SLAs had caps of five points, one SLA had a cap of two points, and three SLAs had caps of one point.

## SLA Performance Standards Reduced

The modification resulted in material reductions in or eliminated
measured performance standards in 16 of the 43 original SLAs (37
percent). Specifically, the modification reduced percentage performance
requirements, or frequency measurement, or disincentive levels for 12 of
the original SLAs. In addition, four of the original SLAs that had
disincentive levels no longer appeared in the SLAs established under the
modification.

Contract Section H.15, *Service Level Agreement Modification Process*,
states,

> It is recognized that, over the course of the contract, there are
> innovation factors and process improvements that will enable the
> Service Provider (SP) to provide incrementally better services to the
> Department of Education (ED or the Department) over time. As a result,
> the Department seeks to benefit from standard innovations, application
> of best practices, and process changes that can be leveraged from the
> provider's experience into the ED environment....
>
> Subsequent to Transition, both ED and the Service Provider will
> convene periodically to establish which service levels of the 43
> performance standards will be subject to continuous improvement, or
> change. ED or the service provider may periodically recommend changes
> to specific service measurements that are subject to this term and
> condition and, as part of the Balanced Scorecard Measure and Changes
> Clause, introduce new candidates for measure....
>
> As such, the Parties agree to adopt the concept of continuous
> improvement and that the Performance Standards may be modified during
> the term of this Agreement. However, ED acknowledges that some changes
> may impact cost and introduce variability. . ..

This section of the contract acknowledges the potential need for
adjustments to the SLAs, but it is based on continued improvement. The
Department has not yet seen improvement in the SLA performance.

OCIO officials stated that changes in the performance system were
initiated at the end of the first six months of the base year, when it
was determined that the contractor was failing. In addition, OCIO
officials stated there was confusion among senior managers regarding how
the contractor's performance was evaluated. The former OCIO program
manager for the EDNet contract provided a slide show presentation he
prepared that included as guiding principles that the Department should
only measure important areas, and that failure in one SLA should not put
the entire contract at risk. No documentation was found in the contract
files or provided by OCIO that detailed the justifications for the
changes made or analyzed the potential impact of the revisions.

The former OCIO program manager stated there was no analysis to support
the selection of the original 5:1 conversion ratio of demerit points to
disincentive points in the original contract. The former program manager
further stated there was no analysis to support adoption of the 50:1
ratio in the modification. OCIO officials stated the SLA caps were
established to relieve confusion over the disagreement between the
number of disincentive points and the total number of SLA performance
points. They further acknowledged there was no defined rule for the
application of cap values to individual SLAs.

The changes in the performance measurement factors amount to a material
change in the contract. The changes reduced the level of effort needed
to meet performance standards, and they significantly increased the
contractor's performance rating. While the contract terms permitted
alteration of the SLAs to allow for continuous improvement, the
contractor did not earn any SLA performance points during the base year.
Rather than encouraging improvement, the revisions made to the SLAs, in
effect, rewarded the contractor by allowing reduced effort to meet the
same level of performance, or stated another way, by allowing the same
level of effort to meet higher levels of performance.

The Department did not receive any consideration such as reduced costs
for this change in level of effort and ratings calculations. The former
OCIO program manager stated he did not know whether the contract
modification resulted in additional or reduced costs to the Department.
CAM staff stated that no mention was made of seeking an equitable
adjustment from the contractor for reduced effort or more achievable
performance measures. CAM staff agreed that had the performance measures
been increased or made more difficult to achieve, it would have been
reasonable for the Department to provide the contractor with an
equitable adjustment for increased effort.

**[Recommendations]{.underline}**

We recommend that the Chief of Staff ensure that the Chief Information
Officer and Chief Financial Officer take action to:

[]{.mark}

1.  Establish and implement a corrective action plan and schedule to
    address the contractor's performance issues and ensure that minimum
    quality levels of services are provided under the contract.

2.  Based on the corrective action plan implemented, modify the current
    EDNet contract to reflect equitable adjustment to the Department for
    any reduced level of effort required to meet performance rating
    standards.

3.  Ensure that future performance-based contracts include appropriate
    incentives and disincentives to motivate contractor performance,
    provide a correlation between performance and payments to the
    contractor, assure minimum quality levels for all critical services,
    provide the Department with alternatives to address unsatisfactory
    contractor performance, and allow for execution of options years for
    achievement of satisfactory performance levels if such continuation
    is in the best interest of the Department.

**Department Comments**

The Department concurred with the finding and recommendations.

[]{#refheading___toc156030075 .anchor}FINDING NO. 2 -- Department
Controls Did Not Ensure That the Contractor Provided the Quality and
Services Required in the Contract.

**[Summary]{.underline}**

The Department's controls over contract management did not ensure that
the contractor provided the quality and services required in the
contract. Specifically, we noted the following:

a.  The Department did not ensure that Category A services were provided
    at CMMI Level 3 as required in the contract,

b.  The Department did not ensure Category C time and materials staff
    provided by the contractor met the requirements of the labor
    categories for which the Department was charged,

c.  The EDNet AMS did not meet all contract requirements, and
    duplication exists between the EDNet AMS and the Department's AMS
    operated by the Office of Management (OM), and

d.  The Management Information Dashboard (Dashboard) was not provided
    timely and did not meet all contract requirements.

This occurred because the Department did not effectively monitor the
contractor's compliance with the quality of services to be provided. In
addition, Department staff inappropriately provided direction to the
contractor that changed the scope, requirements, and/or due dates for
some deliverables.

As a result, the Department paid for a quality or level of services it
did not receive, an AMS that did not meet requirements and duplicated
some activities of other Department contractors, and did not timely
receive the Dashboard, a critical tool for monitoring contractor
performance with respect to the SLAs. Finally, because a Department
official directed the contractor to host and develop the Dashboard on
its own servers at its own facilities, the Department may have
difficulty enforcing its rights to own the tool it is paying the
contractor to develop.

**Issue 2a -- The Department did not ensure that Category A services
were provided at CMMI Level 3 as required in the contract.**

Capability Maturity Model Integration helps to integrate traditionally
separate functions, set process improvement goals and priorities,
provide guidance for quality processes, and provide a point of reference
for appraising current processes (for a complete list of CMMI levels,
see Attachment 3). The EDNet's IV&V contractor reported in its March 2,
2006, Quarterly Performance Evaluation Report, that the contractor was
not complying with the CMMI Level 3 requirements in the contract. The
IV&V contractor determined that the EDNet contractor was making
significant progress towards CMMI Level 2 compliance in six of six
applicable process areas that were to have been provided at CMMI Level
3. The IV&V contractor's determination was based on review of the
self-assessment performed by the EDNet contractor's own Quality

Assurance Team. In its August 21, 2006, Quarterly Performance Evaluation
Report, the IV&V contractor reevaluated this area and concluded the
EDNet contractor was making significant progress towards CMMI Level 2 in
four of six applicable areas.

We also reviewed monthly CMMI compliance reports submitted by the
contractor from November 2005 through April 2006 that indicated the
contractor had not yet achieved CMMI Level 2 compliance.

The Performance Work Statement (PWS) of the EDNet contract, Section I.A,
"Category A (Firm Fixed Price per Unit Services)," states,

> The contractor shall provide firm fixed price managed services billed
> on a per unit basis at a minimum CMMI Level 3 on day one progressing
> to Level 4 within three to five (3-5) years. The contractor is
> responsible for all processes used in the completion of this work, and
> shall have the capacity to handle multiple critical operational issues
> at one time.

Category A services include such tasks as maintaining the production
servers, providing messaging services, and supporting end users,
including assistive technology and help desk operations.

One of the evaluation factors considered for selecting the EDNet
contractor was the offeror's plan to provide CMMI Level 3 processes for
all Category A work on the first day of the contract. The contractor's
technical proposal dated October 7, 2004, incorporated into the EDNet
contract PWS, Section 2.2.2, "Implementing Category A Work at CMMI Level
3 on Day One," states, "On Day One \[contractor name omitted\] will
provide ED with well-documented, institutionalized, well-evaluated CMMI
Level 3 processes."

The COR stated that she did not have enough time to monitor the
contractor's compliance with all contract terms and conditions. In
addition, the COR stated that this was her first experience in
monitoring a performance-based contract and that training was not
provided to OCIO staff in this area. However, CMMI performance was
reviewed and reported on by the IV&V and EDNet contractors during
performance of the contract. The IV&V contractor's initial analysis of
this area was provided to the Department in March 2006, and the EDNet
contractor had reported on this area in weekly status reports as early
as August 2005. Information was readily available showing noncompliance
in this area, but the Department did not take formal action to ensure
the required level of service was provided.

Overall, the Department was paying for a quality and level of service it
did not receive. The contract required Category A services to be
provided at CMMI Level 3 from the first day of the contract, but
throughout the entire first year of the contract, the Department was
receiving services that did not meet this requirement. These services
were paid for on a fixed price per unit basis. The Department did not
obtain any consideration from the contractor for providing services
below contract specifications, or otherwise modify the contract to allow
for this departure from requirements. Since providing this level of
service was an evaluation factor in

awarding the contract, this contractor may not have been considered if
it had not provided assurances that it would provide CMMI Level 3
services.

**Issue 2b -- Category C staff provided by the contractor did not always
meet the requirements of the labor categories for which the Department
was charged.**

The EDNet IV&V contractor reviewed the resumes of Category C staff to
determine whether the staff met the requirements for the labor category
in which they were placed. The IV&V results were provided to the
Department on March 2, 2006. The requirements for each labor category
were defined in the EDNet contractor's technical proposal. It was
determined that 36 of the 79 staff met qualifications (46 percent), 26
partially met qualifications (33 percent), and 17 did not meet
qualifications (22 percent). See Attachment 4 for numbers of staff in
each category reviewed, and the IV&V findings.

Category C tasks are provided on a time and materials basis under the
contract. The PWS Section I.C, "Category C (Sustaining Time and
Materials Support)," states, "The contractor shall provide qualified
labor on a time and materials basis for continuous staffing of services
whose processes will be managed by ED . . .."

[]{.mark}

FAR Subpart 16.601 states,

(a) *Description*. A time-and-materials contract provides for acquiring
    supplies or services on the basis of -- (1) Direct labor hours at
    specified fixed hourly rates that include wages, overhead, general
    and administrative expenses, and profit. . ..

(b) *Application*. . . .(1) *Government surveillance*. A
    time-and-materials contract provides no positive profit incentive to
    the contractor for cost control or labor efficiency. Therefore,
    appropriate Government surveillance of contractor performance is
    required to give reasonable assurance that efficient methods and
    effective cost controls are being used.

The CO stated that the contractor retained staff from the prior
contractor on the advice of the Department, but that those staff were
not always the best qualified. The CO added that the contractor needed
to get people here who knew how to do the job and how to do it well. The
COR stated that she did not have enough time to monitor the contractor's
compliance with all contract terms and conditions. In addition, the COR
stated that this was her first experience in monitoring a
performance-based contract and that training was not provided to OCIO
staff in this area. Another OCIO staff member stated that Department
personnel raised the issue of staff quality several times with the
contractor, but no real solutions were provided.

Even though the staff recommended by the Department may not have been
the best qualified, the contractor should not have placed the staff in a
labor category for which they did not qualify, and then bill the
Department based on the rate for that labor category. A waiver process
was established for instances when staff were placed into a labor
category for which they were not completely qualified. The Department
had approved waivers for six staff members; five that partially met
qualifications and one that did not meet qualifications. Waivers had not
been provided for the other 37 staff members that did not meet or only
partially met requirements.

The COR stated the results of the IV&V were provided to the EDNet
contractor and the contractor was asked to address the issue. The COR
stated the contractor updated some of the resumes in question, but was
uncertain as to whether all issues had been addressed or whether the
updated resumes met requirements. She also stated no discussion was held
to recoup funds from the contractor for the charges at labor categories
for which the staff did not qualify. According to the COR, this issue
was not as high a priority as other issues under the contract, so it was
not fully pursued.

[]{.mark}

The EDNet contractor's corrective action plan dated October 6, 2006,
stated the following:

> . . . \[C\]ertain parts of staffing process were under \[the EDNet
> contractor's\] control but that \[the Department's\] involvement
> impacts \[the EDNet contractor's\] ability to fill vacancies within 10
> days as outlined in the Performance Work Statement. . ..

The EDNet contractor's corrective action plan included a proposed change
to the contract. Subsequently, the contract was modified effective,
October 27, 2006, to state,

> The Contractor shall present to the COR a qualified candidate within
> ten (10) business days of a vacancy occurring. A qualified candidate
> is one whose education and experience conforms to the education and
> experience requirements of the labor category.

As a result, the Department did not receive the level of service for
which it contracted. For 16 of 17 Category C staff who did not meet
qualifications and who did not receive waivers, invoices from inception
of the contract through June 30, 2006, show that 16,840 hours and
\$1,277,079 were billed for these staff. An additional 29,141 hours and
\$2,275,797 were billed for the 21 of 26 staff who only partially met
qualifications and who did not receive waivers. While the entire amount
of these payments would not necessarily be inappropriate, had the staff
been billed at labor categories for which they did qualify, payments by
the Department would have been significantly less. Even a difference in
labor category rates of \$5/hour would result in savings of \$229,905
for the period for the staff that did not meet or only partially met
qualifications.

**Issue 2c -- The EDNet asset management system did not meet all
contract requirements and the system delivered includes duplication of
information also maintained by the Office of Management.**

We found that specific tasks outlined in the EDNet PWS were also
assigned to other Department organizations in previously existing
guidance. In addition, the AMS developed by the EDNet contractor was not
provided timely and did not meet all contract requirements. We also
found that the information being tracked by the EDNet contractor
duplicates some information included in OM's AMS.

The EDNet PWS states, "Asset management is intended to provide ED with a
complete an \[sic\] accurate count of its IT inventory, warranties, and
is to be used in its IT refresh cycle."

Specific tasks listed in the PWS under asset management include:

-   Tracking, reporting, and disposal, as required, of resources and
    general-purpose computer assets and maintenance agreements of the
    assets. Managing the disposal and disbursements of IT equipment to
    schools and disbursement of IT equipment meeting the minimum EDNet
    specifications for the Department of Education's PCs R Us.

-   Operating and maintaining a central processing and secure storage
    facility for surplus IT equipment.

-   Accountability for all IT assets and participate in the annual
    physical inventory and reconciliation process.

-   Establishing, updating, and maintaining an asset inventory database.

-   Tracking all ED assets (location, asset ID, serial number, finances)
    and ensuring service contracts are in force as needed to meet SLAs.

-   Developing and documenting asset management policies and procedures.

Office of Management Handbook OM-05 (Handbook), *Property Management
Manual,* dated December 31, 2002, includes all of the above items as
responsibilities of OM's Property Management and Inventory Team (PMIT),
with the exception of the tasks related to the PCs R Us program and
equipment service contracts, that are the responsibility of OCIO. The
Handbook provides policies and procedures for the management and
administration of the Department's plant, property and equipment, and
for establishing and maintaining a compliant and effective property
control system. The Department has assigned overall management of its
equipment to Facilities Services in OM. Contractor staff is also used to
provide IT property management tasks in OM.

Department officials stated at the time the requirements for the EDNet
contract were being developed, OM was seeking a new asset management
software solution. OM selected a vendor and began working to implement
the software for its AMS in Fiscal Year (FY) 2005. The new OM AMS was
implemented in October 2005. The EDNet contract was awarded April 26,
2005.

According to Department officials, the EDNet asset management tasks were
not combined with OM's asset management responsibilities due to
differing concepts of how the EDNet asset information would be used and
the intention to use the EDNet AMS to support the contractor's billing.
Department officials also reported there was reluctance on the part of
OM PMIT staff to combine efforts with the EDNet asset management system
or to use one system to validate the other.

The EDNet contractor stated that originally the EDNet AMS was to be
provided upon transition from the prior contractor, or as of July 1,
2005. The contractor stated that since the originally intended AMS could
not be utilized, the delivery date for the AMS was revised by OCIO to
March 2006, and then revised again to October 2006. None of these
changes to deliverable dates or the scope of the work under this task
were formally made to the contract through a modification.

Also, since the originally intended AMS could not be utilized, the
contractor received initial information from the OM AMS to serve as an
interim solution while the new system was being developed. As result,
the current EDNet's AMS and OM's AMS contain similar information related
to IT equipment.

The EDNet AMS was included in the fixed price portion of the contract.
Since the scope of effort and delivery date of the product were never
formally changed, and an adjustment in contract price was not received,
the Department has paid for services it did not fully receive.

Using the OM AMS to validate information in the EDNet AMS would help
ensure the Department is not being billed for items that are not in use.
Due to past problems noted with the OM AMS, the Department is currently
undergoing an inventory to validate all OM AMS information and provide
an accurate inventory. The Department is also obtaining the services of
a database administrator to expand the fields used in the OM AMS.

**Issue 2d -- The Management Information Dashboard was not provided
timely and did not meet all contract requirements.**

The Dashboard was not provided timely and did not meet all contract
requirements. The Dashboard was intended to function as an automated
data collection tool to assist Department managers in making efficient
and effective use of their resources. The Dashboard was intended to
provide items such as: a daily snapshot of how the Department and the
contractor are delivering IT operations; historical information; and
trend analysis and correlation. We found that the Dashboard was provided
six months after its original due date and was hosted on the
contractor's system, at the direction of a Department official, which
could create difficulty for the Department in enforcing its ownership
rights. The Department rejected the delivered Dashboard because it did
not meet all contract requirements, so the Department still lacks a
fully functional tool for monitoring SLA performance.

FAR Subpart 43.102(a) states,

> Only contracting officers acting within the scope of their authority
> are empowered to execute contract modifications on behalf of the
> Government. Other Government personnel shall not -- (1) Execute
> contract modifications; (2) Act in such a manner as to cause the
> contractor to believe that they have authority to bind the Government;
> or (3) Direct or encourage the contractor to perform work that should
> be the subject of a contract modification.

OCFO Directive 2-108, *Contract Monitoring for Program Officials*
(Directive), effective September 16, 2004, and updated March 30, 2006,
Section VI.E.3, states the COR,

> Provides technical direction to contractors as necessary and
> appropriate, depending on the type and terms of the contract. The
> individual named in the contract as the COR is the sole person (other
> than the CO) with the authority to provide technical direction.
>
> Section VII.H of the Directive states technical direction should be
> kept to a minimum, must not have the effect of making decisions
> reserved for the CO, and must be in writing and on file.

EDNet contract Section G.1.B states,

> Neither the COR nor any other government employee, except the
> Contracting Officer, is authorized to make any commitments or
> otherwise obligate the Department of Education or authorize any
> changes that affect contract price, terms or conditions. . ..

The EDNet contract PWS, Section III, "Scope of Work," under "Expected
Investment in Tools," states,

It is understood that ED will own intellectual property to all data,
software and the licensing of these tools. These tools include: (1)
Management Information Dashboard ....

The problems with the Dashboard occurred because OCIO staff
inappropriately provided direction to the contractor on development of
the Dashboard without formalizing those changes through contract
modifications. The CO stated that he had not received any requests from
OCIO to change the requirements for the Dashboard.

[Delay in Delivery of the Dashboard]{.underline}

The Department delayed development and implementation of the Dashboard.
The EDNet contractor's technical proposal, Section 9.1, incorporated
into the PWS for the EDNet contract, included development of a Dashboard
to collect information on the SLAs to provide management with the
necessary metrics to run operations. The technical proposal states the
initial production of the Dashboard was to be provided within one month
of transition, and complete production within six months of transition.
This schedule would have provided initial production by August 1, 2005,
and complete production by January 1, 2006.

Instead, the former OCIO Program Manager communicated to the EDNet
contractor in an email dated June 14, 2005, that the Department would
provide the specifications for the Dashboard. In a letter to the
Department dated March 21, 2006, the contractor stated that it was,

> . . . \[R\]eady and capable of delivering to its initial production
> version of the Management Dashboard on the proposed delivery date.
> However, per the request of \[the OCIO Program Manager\], we were
> instructed to delay initial production of the Management Dashboard
> until the Department provided its own specifications.

OCIO engaged the IV&V contractor to develop specifications for the
Dashboard, and the IV&V contractor provided the specifications to the
EDNet contractor on December 29, 2005. The EDNet contractor provided
initial production of the Dashboard on January 26, 2006, and complete
production on June 29, 2006, six months after the deliverable was
originally due under the PWS. Changes in the scope of the work (from the
EDNet contractor developing the specifications, to the Department
providing the specifications through the IV&V contractor) and due date
were not formalized in the contract.

[Hosting and Ownership of the Dashboard]{.underline}

An email from the former Deputy CIO for the Department to the EDNet
contractor's Program Manager, dated November 29, 2005, permitted the
contractor to host the Dashboard on its own system and expressed the
view that the Department may be relinquishing its ownership of all data
and resulting hardware and software from the development of the
Dashboard. The email states,

Please use your server and your \[software\] sw . . . to accomplish this
goal.  Please work with \[the IV&V contractor\] for the design of the
\[application\] app -- please show them the dashboard you have already
developed for other \[organizations\] org to include \[Office of Federal
Student Aid\] FSA.  I realize this means that we don't own this
\[hardware\] hw or sw or any customizations you make -- and if you are
not the vendor on this contract that this sw, hw and customization for a
dashboard will leave with you.

The COR stated she was unsure why this change was made and why the
dashboard could not use a Department server. She stated that she was
often excluded from meetings between the Department's program manager
and the contractor.

Due to delays in development of the Dashboard, the Department did not
timely receive a critical tool for monitoring contractor performance
with respect to the SLAs. Because the Department engaged another
contractor to develop the specifications for the Dashboard, additional
costs were incurred. Finally, because the Department provided direction
to the contractor that allowed it to host and develop the Dashboard on
its own servers, the Department may have relinquished its rights to own
the tools it is paying for the contractor to develop. The server for the
dashboard is being warehoused at the contractor's facility in Virginia,
and the transmission of data may not be adequately protected outside the
Department's systems.

**[Recommendations]{.underline}**

We recommend that the Chief of Staff ensure that the Chief Information
Officer and the Chief Financial Officer take action to:

1.  Ensure the contractor provides Category A services at CMMI Level 3,
    or modify the contract, including a reduction in price for lesser
    quality services, to permit the contractor to provide services at
    the lower levels.

2.  Ensure the Category C staff meet the qualifications for the labor
    categories in which they are charged.

3.  Determine the labor categories that the staff identified by the IV&V
    as not meeting or partially meeting qualifications should have been
    placed in, and calculate the amounts that should have been charged
    for their services from the beginning of the contract to date.
    Instruct the contractor to adjust a future invoice to reclaim the
    amounts improperly paid.

4.  Ensure that OCIO staff do not provide direction to the contractor
    that changes the scope, delivery dates, or other contract terms and
    conditions without going through the CO.

5.  As appropriate, formalize the changes that have been made and seek
    equitable adjustments for any concessions made on the part of the
    Department.

6.  Reinforce to the contractor its responsibility in ensuring that any
    direction provided that impacts the scope, delivery dates, or other
    terms and conditions, is provided by the CO prior to making any
    changes.

7.  Work with OM to develop and implement a plan to combine or
    coordinate the EDNet and OM AMSs to reduce duplication of effort and
    data. In the interim, match items billed under the EDNet contract to
    OM's AMS to ensure only appropriate items are being billed.

8.  Confirm that ownership of the Dashboard remains with the Department.
    If possible, take action to transition the Dashboard to the
    Department's systems so that the Department will not lose control of
    this tool in the event of a change in contractors.

9.  Evaluate the security of information passing through the Dashboard,
    outside the Department's systems, to ensure sensitive data is not
    vulnerable. Take appropriate actions to ensure the security of this
    data.

10. Provide training to the COR, and other involved OCIO staff and
    managers, in monitoring performance-based contracts.

**Department Comments**

The Department concurred with the finding and recommendations.

[]{#__RefHeading___Toc156030076 .anchor}FINDING NO. 3 -- Contract
Administration Practices Were Not Effective.

**[Summary]{.underline}**

The contract administration practices followed for the EDNet contract
were not effective in ensuring that the contractor complied with the
terms and conditions of the contract. Specifically, we found,

a.  A contract monitoring plan was not developed,

b.  Deliverables under the contract were not routinely reviewed and
    recommendations were not made to the CO for acceptance or rejection,

c.  The COR did not prepare and submit to the CO written evaluations of
    contractor-submitted reports, and

d.  Contract files did not include complete records of actions taken
    under the contract.

This occurred because the Department did not ensure appropriate
resources were assigned to manage and oversee contract administration,
and because Department staff did not comply with Federal regulations and
Department policies and procedures.

As a result, the Department did not have complete documentation of the
contractor's performance and its monitoring of the contract. The
Department lacks assurance that the contractor fully complied with the
terms and conditions of the contract and provided all services and
deliverables required and for which the Department paid. In the event of
a future dispute, the Department's position would be weakened by this
lack of documentation. Also, by not enforcing some contract terms, the
Department may be setting a precedent that could make it more difficult
to enforce other contract terms.

Office of Federal Procurement Policy (OFPP), Policy Letter 93-1,
*Management Oversight of Service Contracting,* reissued May 18, 1994
(Policy Letter), ". . . establishes Government-wide policy, assigns
responsibilities, and provides guiding principles for Executive
Departments and agencies in managing the acquisition and use of
services."

Section 7c of the Policy Letter states,

> When contracting for services, in particular for highly specialized or
> technical services, agencies should ensure that a sufficient number of
> trained and experienced officials are available within the agency to
> manage and oversee the contract administration function. This
> especially applies to such services as management and professional
> support, studies, analyses, and evaluations, and engineering and
> technical support. Agency officials need to make sound judgments on .
> . . whether the contractor is performing according to the contract
> terms and conditions . . .. Agency officials must also provide an
> enhanced degree of management controls and oversight when contracting
> for functions that closely support the performance of inherently
> Government functions.

[]{.mark}

[]{.mark}

FAR Subpart 37.5 implements OFPP Policy Letter 93-1. Subpart 37.503
states, "The agency head or designee should ensure that . . . (d)
Strategies are developed and necessary staff training is initiated to
ensure effective implementation of the policies in 37.102." Subpart
37.102 is entitled, "Policy," and provides requirements for using
performance-based acquisition for services. FAR Subpart 37.102(h)
states,

Agencies shall ensure that sufficiently trained and experienced
officials are available within the agency to manage and oversee the
contract administration function.

**Issue 3a -- A contract monitoring plan was not developed for the EDNet
contract.**

A contract monitoring plan (CMP) was not developed for the EDNet
contract. The EDNet contract was awarded on April 26, 2005, but as of
September 2006, a CMP had not yet been developed. Currently, the COR is
assisted in monitoring the contract by another OCIO staff member on a
part-time basis, OCIO managers, and by the IV&V contractor. However, an
overall plan to coordinate the activities of Department staff and
contractors was not developed to ensure the contract was appropriately
monitored.

OCFO Financial Management and Accountability Procedure (Procedure)
CO-111, *Writing and Implementing a Contract Monitoring Plan,* effective
May 31, 2005, states,

> Contracts and Acquisition Management (CAM) policy is that every
> contract must include a Contract Monitoring Plan (CMP) describing the
> steps the Government will take to monitor contractor performance.
>
> The procedure requires the CO, Contract Specialist (CS), or COR to
> write a plan detailing how contract performance will be monitored
> throughout the life of the contract. It further requires acquisition
> team members to meet to review the CMP.

The CMP is to be written before contract award. However, in an email to
all OCFO CAM staff on December 16, 2005, CAM management extended the
requirement to all existing contracts, stating that by January 31, 2006,
CAM staff were required to write a CMP for each active contract or task
order.

The CO stated that a CMP was developed as a deliverable under the IV&V
contract. The CO believed the IV&V CMP covered all monitoring activities
for Department and contractor staff. However, we found the IV&V plan did
not include the responsibilities of the Department staff and was limited
to only the IV&V contractor's responsibilities. The IV&V CMP did not
cover 7 of 12 activities outlined in the Procedure -- specifically,
accepting/rejecting deliverables, entering receipts for deliverables,
reviewing invoices, paying invoices, communicating with contractors,
processing modifications, and administering subcontracting requirements.

In addition, the IV&V contract was not awarded until September 30, 2005,
and the IV&V CMP was not established until December 5, 2005. From the
award of the EDNet contract, effective May 1, 2005, seven months elapsed
prior to development of the IV&V CMP. For that time period, no plan was
in effect for monitoring the EDNet contract.

Without assessing risk and developing a CMP to address the risks in the
EDNet contract, the Department lacks assurance that the contractor is
adhering to the requirements of the contract, and that the Department is
receiving intended products and services.

**Issue 3b -- Deliverables under the contract were not routinely
reviewed and recommendations were not made to the CO for acceptance or
rejection.**

Documentation of deliverable receipt and acceptance was not found in
either the COR or CO files. There was no supporting documentation to
show if all deliverables were received, received timely, and met
contractual requirements.

We asked the COR to provide a copy of any deliverable tracking schedules
she used to ensure the contractor provided all required deliverables.
The COR had not developed such a schedule, but provided a schedule
prepared by the contractor that summarized deliverable status. In July
2006, the IV&V contractor developed a compliance matrix of deliverables
required under the contract. This matrix included 1,948 items in 19
deliverable categories. OCIO staff provided us with this matrix for 5
deliverable categories that included 817 items. We judgmentally selected
32 items (4 percent) from this list to determine whether the
deliverables were received timely and met contract requirements.

While the COR provided us with all of her electronic files and emails,
we could not determine which items represented final deliverables, or
when the documents were received (see further discussion of contract
file documentation in Issue 3d of this section). We then asked the COR
to provide us with copies of the documents received for these 32 items,
along with information to show the date received and acceptance by the
Department, if any. According to the COR, 24 of the 32 deliverables
requested (75 percent) were not delivered. Of the eight that were
delivered, four were related to the Management Dashboard and Firewall
Assessment, which were rejected by the Department for not meeting
contract requirements. Included as not delivered were three deliverables
originally developed by the former EDNet contractor that the current
contractor had not updated or revised as required.

FAR Subpart 46.501 states,

> Acceptance constitutes acknowledgment that the supplies or services
> conform with applicable contract quality and quantity
> requirements....Supplies or services shall ordinarily not be accepted
> before completion of Government contract quality assurance
> actions....Acceptance shall ordinarily be evidenced by execution of an
> acceptance certificate on an inspection or receiving report form or
> commercial shipping document/packing list.

FAR Subpart 46.502 states,

> Acceptance of supplies or services is the responsibility of the
> contracting officer.

Section VI.E.5-6 of the Directive states, the COR:

Reviews and makes timely recommendations to the CO as to the approval,
disapproval, or other action to take concerning a contractor's
submission of (or failure to submit) payment requests, deliverables,
interim or final progress and financial reports, or any other
requirements of the contract. . ..

> Immediately reports contractor performance problems to the CO/CS.

Section VII.A.4 of the Directive states,

> The CO relies heavily on the COR to collect monitoring information and
> in make \[sic\] related analyses and recommendations for
> administrative action. This information and analysis must be fully
> documented and reported promptly to the CO to protect the
> Government\'s interests, and to ensure that the program office will
> have the facts necessary to make informed decisions about the contract
> and the program in general.

Section VII.N.1-3 of the Directive states that only a CO can formally
accept or reject deliverables, and that it is the CO's responsibility to
take formal action to reject the deliverable. Formal action can include
written notification to the contractor and rejection of deliverable
payment, if applicable.

OCFO Procedure CO-8, *Procedure for Receiving Goods and Services in the
Contracts and Purchasing Support System (CPSS)*, dated March 15, 2003,
includes the following definitions:

> Acceptance - acknowledgement by the government that goods and services
> received conform with the requirements of the contract. The
> contracting officer is responsible for acceptance of goods and
> services. The COR inspects and recommends acceptance.
>
> Constructive Acceptance - A concept in which acceptance is deemed to
> have occurred on the 7th day after the contractor delivered goods or
> performed the services, unless a disagreement over quantity, quality,
> or contractor compliance with a contract requirement exists.

The May 12, 2005, memorandum from the CO appointing the COR to the EDNet
contract, signed by both parties on May 20, 2005, includes the following
as the COR's responsibility:

> Monitor the contractor\'s performance to ensure compliance with the
> technical requirements of the contract including inspection and
> testing of deliverables and evaluation of reports. Recommend final
> acceptance or rejection to the CO.

The COR stated that she was not aware of any analyses performed to
determine whether deliverables met contract requirements. She stated she
feels overwhelmed with contract monitoring responsibilities and does not
have enough time to both review invoices and ensure the contractor is
complying with contract terms. The CO stated he is copied on
deliverables provided to the COR, and the COR provides acceptance
information to him only on an exception basis. If the COR does not
provide direction, the CO assumes the deliverables meet requirements.
The CO stated after seven days, constructive acceptance of deliverables
occurs.

Without formal review and acceptance, the Department cannot ensure the
contractor is meeting requirements and that inappropriate deliverables
have been constructively accepted. If deliverables do not meet
requirements, the Department may not be receiving the products and
services paid for under the contract. Inadequate inspection and lack of
formal acceptance or rejection of deliverables may set a precedent of
not enforcing contract terms, making it more difficult for the
Department to enforce other contract terms. Also, there is no assurance
that all the deliverables are delivered as required by the contact. As
seen from our limited review of 32 deliverables, the contractor had not
provided 24. The Department is not getting the products and services for
which it is paying under the contract.

**Issue 3c -- The COR did not prepare and submit written evaluations of
contractor-submitted reports.**

The COR did not prepare and submit written evaluations of
contractor-submitted reports as required. We identified 43 different
reports to be submitted monthly, quarterly, or semi-annually under the
contract. The COR stated communication with the CO related to progress
reports is rarely made and the evaluation, if any, is informal. As a
result, this tool for contract monitoring is not being effectively used.

FAR Subpart 42.1101 states,

> Production surveillance is a function of contract administration used
> to determine contractor progress and to identify any factors that may
> delay performance. Production surveillance involves Government review
> and analysis of -- (a) Contractor performance plans, schedules,
> controls, and industrial processes; and (b) The contractor's actual
> performance under them.

FAR Subpart 42.1106(b) states that contract administration offices shall
review and verify the accuracy of contractor reports and advise the
contracting officer of any required action.

[]{.mark}

Section VII.G.2.a.ii of the Directive states,

[]{.mark}

Understand and Evaluate the Performance Reports -- The COR must read
promptly all progress reports submitted by the contractor. Failure to
read the reports negates their considerable value in keeping the
Government up to date.\
\
If a report's language is vague or unclear, the COR should ask the
contractor for clarification. The contractor may be trying to \'gloss
over\' a problem. If the technical content of the report lies outside
the COR's expertise, a technical specialist within the Department should
be called in to interpret the report.

> The COR must make a written evaluation of each report. Depending on
> the type of contract and relative importance of the report, the
> evaluation might be either rigorous or reasonably informal.

The May 12, 2005, memorandum from the CO appointing the COR to the EDNet
contract, and signed by both parties on May 20, 2005, includes the
following as the COR's responsibility,

> Review progress and financial reports, invoices, vouchers, and
> recommend approval or disapproval by the CO.

The COR stated written evaluations of contractor-submitted reports are
done through the CPSS when she approves invoices. However, we reviewed
the information in CPSS and found that there were no comments for 12 of
31 (39 percent) invoices reviewed. Of the 19 invoices that included
comments, 15 did not identify the author of the comments. None of the
comments identified the reports reviewed and conclusions of the review.
This information does not satisfy the requirements of a written
evaluation. The CO stated the COR contacts him via email as incidents
arise, but does not provide him with written evaluations of the
contractor's reports. This also does not meet the requirements of the
Directive to provide a written evaluation of each report.

Detailed review of contractor-submitted reports, whether by the COR or
other staff with results provided to the COR, provide the Department
with an effective monitoring tool and potentially early detection of
developing problem areas. Feedback to the CO is critical to ensure the
CO is kept informed of any developing issues.

**Issue 3d -- Contract files did not include complete records of actions
taken under the contract.**

Contract files did not adequately document all significant actions and
monitoring. Specifically we noted,

-   Neither the COR nor CO files contained documentation of deliverable
    receipt and acceptance,

-   Technical direction provided was not documented,

-   Significant discussions with the contractor regarding billing
    accuracy were not documented, and

-   Electronic data maintained by the COR was not easily identifiable
    and readily retrievable.

[Documentation of Deliverable Receipt and Acceptance]{.underline}

[]{.mark}

As stated above under Issue 3b, the COR did not maintain documentation
of deliverable receipt and acceptance. As discussed below, the COR
maintained all email correspondence and files received, but we could not
determine from this information which items represented final
deliverables and when the items were received.

[]{.mark}

[]{.mark}

[Documentation of Technical Direction Provided]{.underline}

As discussed in Finding 2, Department staff provided verbal or technical
direction to the contractor with respect to tasks under the contract,
primarily the asset management system and the Management Dashboard. This
direction was not fully documented in contract records, and the CO was
not formally informed of changes that were being made.

## Significant Discussions with the Contractor Regarding Billings

The COR reported she experienced difficulties with the accuracy of
contractor billings during the early months of the contract. She stated
she had numerous discussions with less senior representatives of the
contractor, but eventually spoke with a vice president to resolve the
problem. The COR's files did not have documentation of that discussion.

## Electronic Data Maintained by the COR

[]{.mark}

We also noted that the COR's file organization could be improved. A
great deal of the COR's information was maintained electronically. While
this is an acceptable format per the FAR, it was difficult to identify
deliverables, reports, and other information. The COR provided us with
three CDs, which included 5,088 email messages and 662 other files that
although organized to some extent in subfolders by subject areas, did
not indicate which items under subfolders represented the final versions
of the documents. For example the files under the "deliverable"
subfolder did not indicate which files were the final versions of
deliverables provided or when the deliverables were received. In
addition, the information was stored on the COR's email archives and
personal folder in the network drive, limiting access to information
that may be needed by others, should the COR be unavailable when
information is required.

FAR Subpart 4.801 states,

\(a\) The head of each office performing contracting, contract
administration, or paying functions shall establish files containing the
records of all contractual actions.

> \(b\) The documentation in the files...shall be sufficient to
> constitute a complete history of the transaction for the purpose of --
> (1) Providing a complete background as a basis for informed decisions
> at each step in the acquisition process; (2) Supporting actions taken;
> (3) Providing information for reviews and investigations; and (4)
> Furnishing essential facts in the event of litigation or congressional
> inquiries. . ..

FAR Subpart 4.802 states,

> \(c\) Files must be maintained at organizational levels that ensure --
> (1) Effective documentation of contract actions; (2) Ready
> accessibility to principal users; (3) Minimal establishment of
> duplicate and working files; (4) The safeguarding of classified
> documents; and (5) Conformance with agency regulations for file
> location and maintenance.
>
> \(f\) Agencies may retain contract files in any medium (paper,
> electronic, microfilm, etc.) or any combination of media, as long as
> the requirements of this subpart are satisfied.

[]{.mark}

Section VII.K of the Directive states,

> 1\. The purpose of detailed record-keeping is to build a complete
> history of each project so that information is not lost or forgotten,
> and so that others -- e.g., one's supervisor, a new COR assigned to
> the project, an auditor or perhaps a court of law -- can get a clear
> picture of what has occurred during the life of the contract. (If a
> dispute occurs, it could be several years between the event and its
> resolution. The COR and the program office file could be called upon
> at a very late date.) Files and records should be maintained in an
> orderly fashion with an index noting documents contained in files.
>
> 2\. The COR should document [every]{.underline} significant action
> taken or conversation held in the course of monitoring or
> administering a contract. The judgment of what is significant is left
> to the COR although too much documentation is usually better than not
> enough. (Emphasis in original.)
>
> 3\. Any monitoring action or conversation, which discloses that the
> contractor is either failing to perform as required or is failing to
> make sufficient progress, must be documented....
>
> 4\. When the COR judges actions or conversations worthy of
> documentation, such documentation must be placed in the program office
> contract file, and a copy must be sent to the CO for entry into the
> official file. . ..
>
> 6\. Any substantive instruction or recommendation made by the CO to
> the COR must also be made a part of the official file. Such
> documentation is necessary to prevent confusion concerning "apparent
> authority" (see [Section
> VII.B.2](../../../../../../../../C:/Documents%20and%20Settings/christina.galligan/My%20Documents/Contract%20Monitoring/#_B._Standards_of))
> and guard against misunderstandings among the CO, COR and contractor.
>
> []{.mark}

Section VII.Q of the Directive states,

> 1\. The Government's record for a contract is maintained primarily in
> two places: In the program office, and in the contracting office.
>
> 2\. The program office file, maintained by the COR, should contain all
> information needed by the COR to carry out his or her contract
> monitoring and managing responsibilities....
>
> 3\. The file maintained by the CO is the Government's "official file"
> and must contain all information having even the slightest bearing on
> the obligations of the two parties to the contract and their
> performance against those obligations.

The Department did not ensure that a CMP was developed, or that the CMP
developed by the IV&V contractor included the responsibilities of all
parties. The Department had not established a process to formally
document deliverable receipt, review, and acceptance. The COR stated
that she felt overwhelmed by her contract monitoring responsibilities
and did not have time to ensure the contractor met all contract
requirements.

[]{.mark}

Without an organized approach to contract monitoring, through
development of a CMP that covers all responsibilities, the Department
cannot ensure that all deliverables were received, on time, and met
contract requirements. The Department's interests in the event of a
dispute may be harmed without complete documentation of products
provided by the contractor and the Department's determination of the
acceptability of those products.

Without sufficient documentation of OCIO's decisions and actions on the
contract, the Department's interests may not be sufficiently protected.
Lack of documentation in the event of a dispute or future action may
harm the Department's position. Also, without sufficiently organized
contract documentation, whether in hard copy or electronic, available to
all parties that need it, the Department may be unable to or experience
difficulty in identifying, retrieving, and reviewing data should the
need arise.

**[Recommendations]{.underline}**

We recommend that the Chief of Staff ensure that the Chief Information
Officer and the Chief Financial Officer take action to:

1.  Immediately develop and implement a contract monitoring plan for the
    EDNet contract to include the roles and responsibilities of all
    involved OCIO staff, OCFO/CAM staff, and the IV&V contractor.
    Conduct a meeting with all involved parties to ensure that all are
    familiar with their responsibilities, as required by the OCFO
    procedure.

2.  Ensure that appropriate resources, whether Department staff, IV&V,
    and/or other contract assistance, are assigned so that
    contract-monitoring responsibilities can be effectively
    accomplished.

3.  Establish a process to track receipt of deliverables, review the
    deliverables for compliance with contract requirements, and provide
    written recommendation to the CO

> for acceptance or rejection of deliverables. Ensure that constructive
> acceptance does not occur and harm the Department's interests under
> the contract.

4.  Ensure that written evaluations of contractor submitted reports are
    performed and provided to the CO.

5.  Periodically determine the reliability of performance reports
    provided by the contractor by verifying the information contained in
    the reports.

6.  Establish and implement a process to ensure that all significant
    actions and monitoring under the contract are adequately documented,
    organized, and accessible. Ensure that CAM and program office files
    contain complete information, whether in electronic format or hard
    copy, to constitute a complete history of the contract that is
    accessible to all involved parties.

**Department Comments**

The Department concurred with the finding and recommendations.

[]{#__RefHeading___Toc156030077 .anchor}OBJECTIVE, SCOPE, AND
METHODOLOGY

The objective of the audit was to determine the effectiveness of the
Department's management of the EDNet contract. To accomplish our
objective, we performed a review of internal control applicable to the
Department's management of the EDNet contract. We reviewed applicable
laws and regulations and Department policies and procedures. We
conducted interviews with Department officials to gain an understanding
of the contract management process and specific management of the EDNet
contract. We also conducted interviews with the EDNet contractor and the
IV&V contractor to obtain information about various aspects of contract
performance.

We reviewed the EDNet contract, statement of work, and business and
technical proposals submitted by the contractor. We also reviewed the
IV&V contract. We evaluated information in the COR's files, including
electronic data and email, and in the official contract file maintained
by CAM. We performed in-depth reviews of two significant deliverables --
the Management Information Dashboard and the EDNet Asset Management
System -- and evaluated reports provided to the Department by the
contractor and IV&V.

The scope of our review included analysis of information from the
competition of the EDNet contract, from January 6, 2005 through April
26, 2005, and analysis of performance and deliverables applicable to the
EDNet contract during the period May 1, 2005, through October 6, 2006.
The original listing of contract deliverables from the IV&V Compliance
Matrix (CMX) included a total of 1,948 deliverables from 19 deliverable
categories. OCIO provided OIG with the CMX for 817 deliverables in five
major categories. We judgmentally selected a sample of 32 deliverables
from this list, based on criticality level in the EDNet contract, to
review the Department's process for receiving and accepting/rejecting
deliverables. Judgmental sampling was used since results would not be
projected as separate deliverable requirements varied significantly and
were not comparable.

The scope of our review also included contract payments made during the
period May 1, 2005 through July 31, 2006. We extracted the listing of
payments from the Financial Management Support System (FMSS), a
component of the Education Centralized Automated Processing System
(EDCAPS). We included all 31 invoices in our review for the scope
period, totaling \$20,609,890 in payments made by the Department.

To complete our review, we relied on computer-processed data obtained
from EDCAPS/FMSS representing contract payments from May 1, 2005 through
July 31, 2006. This data was also recorded in the Department's CPSS and
compared by OIG staff to the payments in EDCAPS/FMSS. We also verified
the completeness and accuracy of the data by reviewing EDNet contractor
invoices in the CO and COR files to validate the payment amounts
recorded in EDCAPS/FMSS and CPSS. Based on our testing, we determined
the listing of contract payments was complete for the purposes of our
audit.

The fieldwork for our audit was conducted at Department offices in
Washington, DC, during the period July 2006 through November 2006. We
held an exit conference with OCFO and OCIO

staff on November 15, 2006. Our audit was performed in accordance with
generally accepted government auditing standards appropriate to the
scope of the review described above.

**Attachment 1: Original Service Level Agreements**

+---+-------------------------+--------+---------+---------+-------+
| # | # Description           | **     | **Re    | **      | # In  |
|   |                         | Disinc | porting | Maximum | clude |
| S |                         | entive | Freq    | Award   | d as  |
| L |                         | L      | uency** | Point   | Modif |
| A |                         | evel** |         | Dedu    | ied S |
|   |                         |        |         | ction** | LA \# |
+===+=========================+========+=========+=========+=======+
| 1 | Customer Satisfaction   | No     | Daily   | N/A     | N/A   |
|   | (Touch Survey)          | ne[^3] |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | Customer Satisfaction   | 5      | Weekly  | N/A     | N/A   |
|   | (Manager Survey)        | points |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Customer Satisfaction   | 20     | Semi-A  | N/A     | N/A   |
|   | (Independent Survey)    | points | nnually |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 4 | Acceptable Time to      | None   | Daily   | N/A     | 10    |
|   | Notify ED of a Security |        |         |         |       |
|   | Event                   |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 5 | Acceptable Time to      | None   | Daily   | N/A     | 11    |
|   | Respond to CERT         |        |         |         |       |
|   | \[Computer Emergency    |        |         |         |       |
|   | Readiness Team\] /      |        |         |         |       |
|   | FedCIRC \[Federal       |        |         |         |       |
|   | Computer Incident       |        |         |         |       |
|   | Reporting Center\]Alert |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 6 | Acceptable Time to      | High   | Daily   | 108.0   | 11    |
|   | Complete CERT / FedCIRC |        |         |         |       |
|   | Tasks                   |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 7 | Acceptable Time from    | High   | Weekly  | 33.6    | 7     |
|   | Anti-Virus Update       |        |         |         |       |
|   | Release to Full         |        |         |         |       |
|   | Distribution            |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 8 | Acceptable Time to      | Low    | Weekly  | 9.6     | 8     |
|   | Release Virus           |        |         |         |       |
|   | Definitions Update      |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 9 | Number of Virus         | High   | Weekly  | 28.8    | 12    |
|   | Penetrations to the     |        |         |         |       |
|   | Network That Cause a    |        |         |         |       |
|   | Network Outage          |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 1 | Acceptable Time to      | None   | Monthly | N/A     | None  |
| 0 | Communicate a Major     |        |         |         |       |
|   | Emergency Event         |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 1 | Asset Management --     | None   | Weekly  | N/A     | None  |
| 1 | Hardware and/or         |        |         |         |       |
|   | Software Inventory      |        |         |         |       |
|   | Accuracy                |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 1 | Production Server       | None   | Monthly | N/A     | 1     |
| 2 | Availability -          |        |         |         |       |
|   | Criticality 1           |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 1 | Production Server       | Low    | Weekly  | 1,66    | 1     |
| 3 | Availability -          |        |         | 0.8[^4] |       |
|   | Criticality 2           |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 1 | Production Server       | Medium | Weekly  | 1,166.4 | 1     |
| 4 | Availability -          |        |         |         |       |
|   | Criticality 3           |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 1 | Production Server       | High   | Weekly  | 5,299.2 | 1     |
| 5 | Availability -          |        |         |         |       |
|   | Criticality 4           |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 1 | Messaging Server        | High   | Monthly | 54      | 3-5   |
| 6 | Availability            |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 1 | System Restoration in   | V      | Daily   | 25,308  | 13    |
| 7 | Non-Disaster Recovery   | arious |         |         |       |
|   | Situations.             |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 1 | Department Authorized   | High   | Monthly | 10.8    | None  |
| 8 | Disaster Recovery       |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 1 | Timeliness of Backups   | Low    | Monthly | 3.6     | 9     |
| 9 |                         |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | Timeliness of Desktop   | Medium | Weekly  | 14.4    | 14    |
| 0 | System Installation     |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | Timeliness of Move, Add | Medium | Monthly | 3.6     | 15    |
| 1 | or Change Software or   |        |         |         |       |
|   | Hardware a Desktop      |        |         |         |       |
|   | System Upon Appropriate |        |         |         |       |
|   | Department Request      |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | Timeliness of           | None   | Weekly  | N/A     | 16    |
| 2 | Establishment of        |        |         |         |       |
|   | Individual User         |        |         |         |       |
|   | Accounts                |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | Timeliness of EDNet     | None   | Weekly  | N/A     | 16    |
| 3 | Password Reset          |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | File Restoration        | None   | Weekly  | N/A     | 16    |
| 4 | (Non-Disaster           |        |         |         |       |
|   | Recovery).              |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | Timeliness of Disabling | None   | Weekly  | N/A     | 16    |
| 5 | User Accounts           |        |         |         |       |
|   | (Including E-mail).     |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | Timeliness of Priority  | High   | Weekly  | 28.8    | 17    |
| 6 | 1 Customer Support      |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | Timeliness of Priority  | Medium | Weekly  | 14.4    | 18    |
| 7 | 2 Customer Support      |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | Timeliness of Priority  | Low    | Weekly  | 9.6     | 19    |
| 8 | 3 Customer Support      |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 2 | Timeliness of Helpdesk  | None   | Daily   | N/A     | 20    |
| 9 | Call Answer             |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Helpdesk Call           | None   | Daily   | N/A     | 21    |
| 0 | Abandonment Rate        |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Helpdesk First Call     | High   | Daily   | 108     | 22    |
| 1 | Resolution              |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Follow-On Calls due to  | High   | Daily   | 108     | 23    |
| 2 | Problem Repeated after  |        |         |         |       |
|   | Initial Fix Failed      |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Call Center             | None   | Daily   | N/A     | None  |
| 3 | Availability            |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Category B Service      | High   | Weekly  | 28.8    | None  |
| 4 | Variance to Budget      |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Category B Variance to  | High   | Weekly  | 28.8    | None  |
| 5 | Schedule                |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Category B Deliverable  | None   | Monthly | N/A     | None  |
| 6 | Quality                 |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Move, Add, Change or    | None   | Daily   | N/A     | 24    |
| 7 | Delete Telecom --       |        |         |         |       |
|   | Software                |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Move, Add, Change or    | None   | Daily   | N/A     | None  |
| 8 | Delete Telecom --       |        |         |         |       |
|   | Hardware and Wire       |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 3 | Network Availability    | High   | Daily   | 108     | 2     |
| 9 |                         |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 4 | Response Time --        | High   | Daily   | 108     | None  |
| 0 | Network                 |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 4 | Variance to Schedule    | None   | Weekly  | N/A     | None  |
| 1 | (Quality Assurance)     |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 4 | Mean Time to Repair --  | None   | Weekly  | N/A     | None  |
| 2 | Applications            |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
| 4 | Availability -- Custom  | None   | Daily   | N/A     | None  |
| 3 | Reports                 |        |         |         |       |
+---+-------------------------+--------+---------+---------+-------+
|   | **Total**               |        |         | **3     |       |
|   |                         |        |         | 4,243** |       |
+---+-------------------------+--------+---------+---------+-------+

Attachment 2: Modified Service Level Agreements

+----+------------------------------+---------+---------+-------------+
| #  | #                            | **Disin | **Re    | **Maximum   |
|    |                              | centive | porting | Award Point |
| #  | # Description                | Level** | Freq    | Deduction** |
|  S |                              |         | uency** |             |
| LA |                              |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 1  | Server Availability          | High    | Weekly  | 5           |
+----+------------------------------+---------+---------+-------------+
| 2  | Network Availability         | High    | Weekly  | 5           |
+----+------------------------------+---------+---------+-------------+
| 3  | Outlook Availability         | Medium  | Weekly  | 3           |
+----+------------------------------+---------+---------+-------------+
| 4  | Blackberry Availability      | Medium  | Weekly  | 3           |
+----+------------------------------+---------+---------+-------------+
| 5  | Unity Availability           | Medium  | Weekly  | 2           |
+----+------------------------------+---------+---------+-------------+
| 6  | Qualified Personnel          | Medium  | Daily   | 5           |
|    | Availability                 |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 7  | Anti-Virus Update (Software  | High    | Weekly  | 2           |
|    | Version)                     |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 8  | Virus File Release           | Low     | Weekly  | 3           |
|    | (Signature Files)            |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 9  | Timeliness of Backups        | Low     | Weekly  | 1           |
+----+------------------------------+---------+---------+-------------+
| 10 | Timeliness of Notification   | High    | Daily   | 3           |
|    | of Issue & Security Events   |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 11 | Response to CERT/FedCIRC     | High    | Weekly  | 1           |
|    | Alerts & Tasks               |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 12 | Number of Virus Penetrations | High    | Weekly  | 1           |
|    | to the Network That Cause a  |         |         |             |
|    | Network Outage               |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 13 | Service Restoration          | Medium  | Daily   | 1           |
|    | (Non-Disaster Recovery)      |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 14 | Timeliness of Desktop System | Medium  | Weekly  | 1           |
|    | Installation                 |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 15 | Timeliness of Move, Add or   | Low     | Weekly  | 1           |
|    | Change Software or Hardware  |         |         |             |
|    | a Desktop System Upon        |         |         |             |
|    | Appropriate Department       |         |         |             |
|    | Request                      |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 16 | Maintain User Accounts       | Low     | Weekly  | 1           |
|    | (Email, Password Resets,     |         |         |             |
|    | Disable User Accts, File     |         |         |             |
|    | Restorations)                |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 17 | Priority 1 Customer Support  | High    | Weekly  | 1           |
+----+------------------------------+---------+---------+-------------+
| 18 | Priority 2 Customer Support  | Low     | Weekly  | 1           |
+----+------------------------------+---------+---------+-------------+
| 19 | Priority 3 Customer Support  | Low     | Weekly  | 1           |
+----+------------------------------+---------+---------+-------------+
| 20 | Helpdesk Call Answer         | Low     | Daily   | 1           |
+----+------------------------------+---------+---------+-------------+
| 21 | Helpdesk Call Abandonment    | Low     | Daily   | 1           |
|    | Rate                         |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 22 | Helpdesk First Call          | Medium  | Daily   | 1           |
|    | Resolution                   |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 23 | Follow-On Calls due to       | High    | Daily   | 3           |
|    | Problem Repeated after       |         |         |             |
|    | Initial Fix Failed           |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 24 | Move, Add, Change or Delete  | Low     | Weekly  | 1           |
|    | Telecom - Software           |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 25 | Timely Production of         | Medium  | Weekly  | 1           |
|    | Required Reports             |         |         |             |
+----+------------------------------+---------+---------+-------------+
| 26 | Complete and Accurate        | Low     | Daily   | 1           |
|    | Process Documentation        |         |         |             |
+----+------------------------------+---------+---------+-------------+
|    | **Total**                    |         |         | **50**      |
+----+------------------------------+---------+---------+-------------+

###  Attachment 3: Capability Maturity Model Integration (CMMI) Levels

  -----------------------------------------------------------------------
  CMMI      Definition
  Levels    
  --------- -------------------------------------------------------------
  Level 0   ***Unaware***. No awareness of a need to improve IT service
            delivery.

  Level 1   ***Initial Phase***. IT service delivery is characterized as
            ad hoc, and occasionally even chaotic. Few processes are
            defined, and success depends on individual effort and
            heroics.

  Level 2   ***Repeatable***. Basic service management processes are
            established. The necessary discipline is in place to repeat
            earlier successes on similar services with similar service
            levels.

  Level 3   ***Defined***. The IT service processes are documented,
            standardized, and integrated into standard service processes.
            All services are delivered using approved, tailored versions
            of the organization's standard service processes.

  Level 4   ***Quantifiable Managed***. Detailed measurements of the IT
            service delivery process and service quality are collected.
            Both the service processes and the delivered services are
            quantitatively understood and controlled.

  Level 5   ***Optimization***. Continuous process improvement is enabled
            by quantitative feedback from the processes and from piloting
            innovative ideas and technologies.
  -----------------------------------------------------------------------

**Attachment 4: Category C Staff**

**Labor Categories and Qualifications**

+----------------------------------+--------+--------+--------+--------+
| ## Labor Categories              | *      | *      | **Par  | # N    |
|                                  | *Total | *Quali | tially | ot Qua |
|                                  | Revi   | fied** | Quali  | lified |
|                                  | ewed** |        | fied** |        |
+----------------------------------+--------+--------+--------+--------+
| Application Engineer             | 1      |        | 1      |        |
+----------------------------------+--------+--------+--------+--------+
| Application Engineer -- Senior   | 1      |        | 1      |        |
+----------------------------------+--------+--------+--------+--------+
| Business Analyst -- Senior       | 2      | 1      |        | 1      |
+----------------------------------+--------+--------+--------+--------+
| Communication Analyst --         | 3      |        | 2      | 1      |
| Intermediate                     |        |        |        |        |
+----------------------------------+--------+--------+--------+--------+
| Communication Analyst -- Senior  | 10     | 2      | 7      | 1      |
+----------------------------------+--------+--------+--------+--------+
| Communication Installer --       | 1      | 1      |        |        |
| Senior                           |        |        |        |        |
+----------------------------------+--------+--------+--------+--------+
| Computer System Installation     | 1      |        |        | 1      |
| Specialist                       |        |        |        |        |
+----------------------------------+--------+--------+--------+--------+
| Data Communication Manager       | 3      | 2      |        | 1      |
+----------------------------------+--------+--------+--------+--------+
| Database Management Specialist   | 1      |        |        | 1      |
+----------------------------------+--------+--------+--------+--------+
| Database Management Specialist   | 1      |        | 1      |        |
| -- Senior                        |        |        |        |        |
+----------------------------------+--------+--------+--------+--------+
| Engineering Aide                 | 3      | 2      |        | 1      |
+----------------------------------+--------+--------+--------+--------+
| Enterprise Resource Planning     | 3      | 3      |        |        |
| Analyst/Designer                 |        |        |        |        |
+----------------------------------+--------+--------+--------+--------+
| Functional Analyst               | 3      | 2      | 1      |        |
+----------------------------------+--------+--------+--------+--------+
| Functional Analyst -- Senior     | 6      | 1      | 2      | 3      |
+----------------------------------+--------+--------+--------+--------+
| Hardware Installation Technician | 1      |        |        | 1      |
| -- Senior                        |        |        |        |        |
+----------------------------------+--------+--------+--------+--------+
| INFOSEC \[Information Security\] | 1      | 1      |        |        |
| Systems Specialist -- Senior     |        |        |        |        |
+----------------------------------+--------+--------+--------+--------+
| INFOSEC Systems Technical        | 2      | 2      |        |        |
| Specialist                       |        |        |        |        |
+----------------------------------+--------+--------+--------+--------+
| Information Engineer -- Senior   | 2      | 2      |        |        |
+----------------------------------+--------+--------+--------+--------+
| Learning Architect               | 1      |        | 1      |        |
+----------------------------------+--------+--------+--------+--------+
| Network Engineer                 | 1      |        | 1      |        |
+----------------------------------+--------+--------+--------+--------+
| Network Engineer -- Senior       | 2      | 2      |        |        |
+----------------------------------+--------+--------+--------+--------+
| Principal Business Process       | 1      |        | 1      |        |
| Reengineering Specialist         |        |        |        |        |
+----------------------------------+--------+--------+--------+--------+
| Principal Information Engineer   | 7      | 7      |        |        |
+----------------------------------+--------+--------+--------+--------+
| Principal System Architect       | 7      | 3      | 4      |        |
+----------------------------------+--------+--------+--------+--------+
| Project Analyst 1                | 3      |        | 2      | 1      |
+----------------------------------+--------+--------+--------+--------+
| Project Managers                 | 6      | 1      | 1      | 4      |
+----------------------------------+--------+--------+--------+--------+
| Quality Assurance Manager        | 1      | 1      |        |        |
+----------------------------------+--------+--------+--------+--------+
| User Services Specialist V       | 1      |        |        | 1      |
+----------------------------------+--------+--------+--------+--------+
| Voice Communication Manager      | 2      | 1      | 1      |        |
+----------------------------------+--------+--------+--------+--------+
| Web Designer                     | 2      | 2      |        |        |
+----------------------------------+--------+--------+--------+--------+
| Totals                           | **79** | **36** | **26** | **17** |
+----------------------------------+--------+--------+--------+--------+
| **Percentage of Total**          |        | *      | *      | *      |
|                                  |        | *46%** | *33%** | *22%** |
+----------------------------------+--------+--------+--------+--------+

**Attachment 5: List of Acronyms**

  -------------------------------------------------------------------------
  **Acronym**   **Definition**
  ------------- -----------------------------------------------------------
  ALO           Audit Liaison Officer

  AMS           Asset Management System

  CAM           Contracts and Acquisitions Management

  CAP           Corrective Action Plan

  CERT          Computer Emergency Readiness Team

  CFO           Chief Financial Officer

  CIO           Chief Information Officer

  CMMI          Capability Maturity Model Integration

  CMP           Contract Monitoring Plan

  CMX           Compliance Matrix

  CO            Contracting Officer

  COR           Contracting Officer's Representative

  CPSS          Contract and Purchasing Support System

  CS            Contract Specialist

  ED            Department of Education

  EDCAPS        Education Centralized Automated Processing System

  EDNet         Education Network

  FAR           Federal Acquisition Regulation

  FedCIRC       Federal Computer Incident Reporting Center

  FMSS          Financial Management Support System

  FSA           Office of Federal Student Aid

  IT            Information Technology

  IV&V          Independent Verification and Validation

  OCFO          Office of the Chief Financial Officer

  OCIO          Office of the Chief Information Officer

  OFPP          Office of Federal Procurement Policy

  OIG           Office of Inspector General

  OM            Office of Management

  PM            Project Manager

  PMIT          Property Management and Inventory Team

  PWS           Performance Work Statement

  SLAs          Service Level Agreements

  SP            Service Provider
  -------------------------------------------------------------------------

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

Attachment 6: Department Response to Draft Report

March 26, 2007

TO: Michele Weaver-Dugan, Director

Operations Internal Audit Team

> Office of Inspector General

FROM: David L. Dunn

Chief of Staff

SUBJECT: Response to Draft Audit Report

> *The Department's Management of the EDNet Contract*
>
> Control Number ED-OIG/A19G0009

Thank you for your draft audit report, *The Department's Management of
the EDNet Contract*, ED-OIG/A19G0009, dated January 18, 2007. I have
reviewed the draft audit report and take no exception to the objectives,
scope, methodology or findings of the report. I concur with the report's
nineteen recommendations. Attached is a proposed Corrective Action Plan
submitted on behalf of the Office of the Chief Information Officer and
the Office of the Chief Financial Officer for implementing the OIG
recommendations.

## Please note that the corrective actions contain sensitive procurement information that must be protected from unauthorized access. {#please-note-that-the-corrective-actions-contain-sensitive-procurement-information-that-must-be-protected-from-unauthorized-access. .Style1}

If you have any questions regarding this response, please contact Brian
Burns, Deputy Chief Information Officer, at (202) 245-6642.

Attachment -- Proposed Corrective Action Plan

cc: Chief Financial Officer

Chief Information Officer

Director, Contracts and Acquisitions Management

[]{.mark}

[^1]: SLAs are agreements that set expectations between the service
    provider and the customer. They describe the products or services to
    be delivered, the single point of contact for end-user problems, and
    metrics by which the effectiveness of the process is monitored and
    approved.

[^2]: See Attachment 1 for a complete list of SLAs measured during the
    base year of the contract.

[^3]: Although identified as SLAs, the surveys identified as numbers one
    through three were not considered SLAs for scoring purposes. They
    were considered to be part of the second set of 50 points in the
    Annual Performance Review Plan.

[^4]: Items numbered 13 through 17 had considerably higher possible
    award deductions because they measured performance on an individual
    server basis within each category.