Telecommunications: GSA Has Accumulated Adequate Funding for Transition to New Contracts but Needs Cost Estimation Policy
Released: 2007-03-26T00:00:00

Letter

Background

As part of its mission of providing federal agencies with acquisition services and solutions at best value, GSA’s technology programs offer agencies options to acquire needed telecommunications services. An option chosen by more than 135 agencies is the FTS2001 program, which consists of two large governmentwide telecommunications contracts—one awarded to Sprint in December 1998 and the other to MCI in January 1999—and FTS2001 crossover contracts.

GSA is planning to replace the FTS2001 contracts, FTS2001 crossover contracts, and separate wireless contracts with a new set of contracts. Collectively known as the Networx program, these new contracts are to provide governmentwide telecommunications services through two indefinite-delivery/indefinite-quantity acquisitions—Networx Universal and Networx Enterprise. The Universal acquisition is expected to satisfy the requirements for a full range of national and international network services and, according to GSA, to ensure the continuity of broad-ranging services with global geographic coverage rendered under expiring contracts. The Enterprise acquisition is expected to offer agencies leading-edge services and solutions with less extensive geographic and service requirements than Universal. The services required in these contracts focus on Internet-based offerings and related security and management services.

GSA expects the transition to begin when Networx Universal awards are made in March 2007 and to continue until fiscal year 2010. Because the FTS2001 contracts with Sprint Nextel Corporation and Verizon Business expired in December 2006 and January 2007, respectively, GSA and the incumbent vendors negotiated separate sole-source contracts that essentially extend the terms of the FTS2001 contracts for 42 months. These sole-source contracts were awarded to ensure uninterrupted service and allow agencies adequate time to complete the transition to Networx.

GSA is working with representatives of federal agencies to prepare for the upcoming transition to Networx, both directly and through the Interagency Management Council (IMC), a group of senior federal information resource officials who advise GSA on issues related to telecommunications contracts. The IMC has worked with GSA to document lessons learned from the transition to FTS2001 that began in the late 1990s, GSA’s most recent governmentwide telecommunications transition. One important lesson learned was that GSA and agency plans for funding transition expenses should be determined early to allow agencies to gauge the impact of transition expenses on their budgets. Specifically, the lessons-learned document recommended that guidelines be established to allow agencies to complete the financial planning required to ensure that the resources needed for transition were available. This led the IMC and GSA to develop a Taxonomy and Allocation of Transition Costs document that identified which Networx transition costs would be borne by GSA and which would be borne by transitioning agencies.

This taxonomy document indicated that GSA will incur or reimburse agencies, including: GSA contractor support costs to, for example, aid in planning for the transition and oversight of Networx contractors. GSA officials also indicated that contractor support will be used to develop a methodology for tracking transition progress and the establishment of a transition coordination center; and certain costs incurred by agencies during transition.

In 2004, following the development of the taxonomy document, GSA generated an estimate of its costs for the transition. GSA’s methodology for its estimate was to: develop assumptions for the estimate, define and develop a baseline for the estimate based on experiences and lessons learned from the previous transition, determine the network growth as well as the total business volume projections for fiscal year 2006 based on historical traffic and cost trends, define the transition cost elements, define and develop a formula for calculating an estimate for each cost element, and design and develop the estimated transition cost model for sensitivity analysis.

Using this methodology, GSA estimated that it would need a total of $151.5 million for a 30-month transition, most of which would be used to reimburse agencies’ transition costs. Table 1 below details the transition costs GSA expects to pay.

GSA planned to pay for its costs and the reimbursement of certain agency costs using its Information Technology (IT) Fund. The IT Fund was a full- cost recovery revolving fund whereby GSA fully recovered all costs of its technology programs and its operations via estimated fee rates. The fees, which are charged to agencies for the use of GSA contracts cover the direct costs of its operations—such as the development and management of contract vehicles—and indirect costs associated with its headquarters, such as support for the Offices of the Chief Information Officer and the Chief Financial Officer. The IT Fund allowed GSA to stabilize rates for its services when expenses varied and was used to provide funding for the previous transition to FTS2001. The IT Fund also contained a working capital reserve that was used to offset losses due to fluctuations in business volumes and other unexpected contingencies. At each fiscal year- end, the uncommitted balance of funds remaining was to be transferred to the general fund of the treasury as miscellaneous receipts.

Recently, a new law changed the structure of the IT Fund and GSA’s organization. On October 6, 2006, the General Services Administration Modernization Act combined the IT Fund with another GSA revolving fund—the General Supply Fund—to create the Acquisition Services Fund. This legislation, which also merged GSA’s technology and supply programs, made all capital assets and balances remaining in the IT and General Supply Funds available for the purposes of the Acquisition Services Fund. According to GSA, the Federal Acquisition Service will increase agency savings, enhance GSA’s capability to meet customer requirements for excellence, and improve internal efficiencies.

GSA’s Analysis Was Not Sound

The analysis GSA used to derive its estimate of $151.5 million was not sound because it was not sufficiently accurate, comprehensive, documented, or validated. The analysis was not sufficiently accurate because it is largely based on the assumption that agencies will transition 76 percent of the services acquired under the current FTS2001 contracts to a different provider under Networx—an intentionally conservative scenario that GSA program officials believe is unlikely to occur. Further, the analysis has not been updated after a nearly 2-year delay in the contract award. While GSA appears to have included all pertinent costs, it may have double-counted a cost, calling into question the comprehensiveness of its analysis. In addition, GSA did not document significant assumptions and data. Finally, GSA’s analysis was not validated by an independent cost estimate nor was an uncertainty analysis performed that would allow GSA to quantify the level of confidence it has in its estimate. These weaknesses can be attributed in part to the lack of a cost estimation policy that would help to ensure that such estimates are developed using best practices. While an intentionally conservative approach minimizes the risk that GSA would have inadequate funds to pay for committed transition costs, it increases the risk that GSA will retain excess funds that could be used for other purposes.

GSA’s Analysis Was Not Sufficiently Accurate

Estimates are accurate when they are not overly conservative, based on an assessment of the most likely costs, and adjusted properly for inflation.

Best practices further dictate that as schedules change, cost estimates should be revised to provide management with insight into the current program status, effective control of the program, and the ability to balance resources and the budget.

The analysis GSA used to derive the estimate, however, was not sufficiently accurate. First, the estimate’s main cost driver is based on the assumption that agencies will transition 76 percent of the services acquired under the current FTS2001 contracts to a different provider under Networx—60 percent due to agencies being forced to change providers and 16 percent due to voluntary changes. However, according to program officials, the 76 percent “transition traffic factor” is intentionally conservative and represents a worst-case scenario that is unlikely to occur.

The 76 percent transition traffic factor is also overly conservative when compared with the previous transition. Then, approximately 60 percent of services were shifted from an incumbent to a different provider under FTS2001, the bulk being forced to change providers when an incumbent providing more than half of the services (AT&T) was not awarded an FTS2001 contract. This forced shift happened in part because GSA limited the FTS2001 awards to only two vendors. In contrast, GSA has placed no such limit on the number of Networx vendors, and therefore, all incumbent FTS2001 vendors could potentially be awarded a Networx contract. Further, because the most-used FTS2001 incumbent (Verizon Business) provides approximately 50 percent of the services under the current FTS2001 contracts, the assumption of a 60 percent forced shift would only be realized if no awards were made to Verizon Business and at least one other incumbent. Finally, GSA’s assumption of a 16 percent voluntary shift is significantly higher than the 3 percent that voluntarily changed providers during the previous transition.

Program officials could not identify any basis for the assumption that voluntary changes will reach 16 percent. However, the officials stated that the percentage of voluntary changes could be higher because agencies will likely have more options when selecting vendors and because of improved guidance on federal requirements regarding the fair opportunity process, which is intended to give each awardee an equal opportunity to compete for agencies’ telecommunications services requirements based on agency-established selection criteria.

Second, the analysis has not been updated to reflect schedule changes. When the estimate was developed in 2004, GSA expected to award the first Networx Universal contracts in mid-2005. Since that time, delays have pushed the time frame back almost 2 years. Universal awards are now expected in March 2007. GSA’s analysis has not been adjusted to reflect additional inflation during this period, which could increase the estimates total by as much as $9 million.

Finally, GSA has not revised its analysis to reflect an assessment of most likely costs using currently available information. The analysis assumes that reimbursable agency transition costs would be greater than during the previous transition due to growth in the volume of services ordered. While GSA estimated that service levels would grow 60 percent over the previous transition by the time of contract award, as of August 2006, service levels had actually grown by 55.9 percent and are now expected to remain stable. The overestimation of service growth added approximately $1.7 million to the estimate.

GSA’s Analysis May Not Be Comprehensive

Estimates are comprehensive when their level of detail ensures that all pertinent costs are included and no costs are double-counted. It is important to ensure the completeness, consistency, and realism of the information contained in the cost estimate.

GSA appears to have included all pertinent costs in its analysis; however, according to officials, a specific agency cost valued at $4.3 million may have been double-counted. For several of the transition costs estimated, GSA based its calculations on the actual charges incurred during the previous transition. However, officials indicated that a specific agency cost that is estimated separately in its current estimate may have already been included in a more general category of its previous costs. Despite this uncertainty, GSA calculated the separate total for this specific agency cost based on current service levels and added it to its estimate. As a result, it is likely this specific agency cost is double-counted and therefore overstated.

GSA’s Analysis Was Not Adequately Documented

Cost estimates are well-documented when they can be easily repeated or updated and can be traced to original sources through auditing. Rigorous documentation increases the credibility of an estimate and helps support an organization’s decision-making process. The documentation should explicitly identify the primary methods, calculations, results, rationales or assumptions, and sources of the data used to generate each cost element.

GSA provided us documentation of its methodology, the calculations it used to derive each cost element, results, and many of the previous transition costs. However, it did not document significant assumptions. Specifically, GSA did not document the rationale behind its 76 percent transition traffic factor or why it used a 30-month time period for the transition—two key assumptions of its analysis.

GSA also did not provide documentation of certain data sources. Specifically, program officials could not provide supporting data used to estimate an agency transition cost valued at $4.7 million. In addition, GSA could not document the data sources used to estimate costs for contractor support in planning and implementing the transition. While many costs in its estimate are based on the charges incurred during the previous transition, GSA officials stated that it was not appropriate to use previous costs as a basis for the contractor cost element. These officials explained that unlike the previous transition, GSA would not provide agencies with on-site contractor support. Officials made this decision, in part, because the 2½ years of transition planning that has occurred to date is expected to result in better preparation by agencies and the ability for them to facilitate their transitions without direct assistance from GSA or its contractors. Instead of basing their projection of contractor costs on prior charges, program officials told us that GSA management decided that contractor support costs should not exceed $35 million. Program officials could not provide any data or analysis to support this decision.

GSA’s Analysis Was Not Validated

Estimates are adequately validated when they have been cross-checked with an independent cost estimate, and when a level of uncertainty associated with the estimate is identified. An independent cost estimate provides the estimator with an unbiased test of the reasonableness of the estimate and reduces the cost risk associated with the project by demonstrating that alternate methods generate similar results. In performing an uncertainty analysis, an entity examines the effects of varying multiple elements and, as a result, is able to express a level of confidence in its estimate.

GSA did not validate its analysis against an independent cost estimate or perform an uncertainty analysis. GSA program officials could not provide a rationale of why these activities were not performed.

Quantifiable Weaknesses Are Small, but Effect of Estimate’s Intentionally Conservative Assumption Could Be Significant

The cumulative effect of the quantifiable weaknesses we identified in GSA’s analysis is relatively small, resulting in an underestimation of $3 million, or roughly 2 percent of the total $151.5 million estimate (as shown in table 2). The underestimation of costs, related to inflation during the extended delay in making award, was offset by the overestimation of service growth and the possible double-counting of a specific agency transition cost.

In contrast, if the actual level of services transitioning to a different vendor is lower than the 76 percent transition traffic factor assumed by GSA, the effect will be greater. Because this factor is the primary cost driver in the estimate, significant changes in transitioning traffic result in similarly significant changes in total costs. This effect can be illustrated using GSA’s transition cost estimate model (see fig. 1). For example, if a transition traffic factor of 66 percent is used (GSA’s assumption of 16 percent for a voluntary transition plus a forced shift of 50 percent that would occur if Verizon Business is not awarded a Networx contract), the total estimated cost of the transition falls to about $136 million. Similarly, if all incumbents are awarded Networx contracts and only GSA’s assumption of a 16 percent voluntary transition occurs, the total estimated cost of the transition is reduced to approximately $40 million. GSA officials explained that they used a risk-averse transition traffic factor to minimize the possibility of underestimating costs.

GSA Does Not Have a Policy to Ensure That Cost Estimates Are Sound

The weaknesses in GSA’s analysis can be attributed in part to the lack of a policy requiring cost estimates to be developed using best practices. Officials from the Offices of the Chief Acquisition Officer, the Chief Information Officer, and the program office’s Controller confirmed that GSA does not have centralized policy or guidance on cost estimation. Instead, the officials that prepared the estimate stated that they based their work loosely on best practices learned as a result of past experiences and were comfortable with the estimate. Officials believe there is no reason to revise their estimate to address the issues we raised in this report because their purpose in producing it was to minimize the risk of underestimating costs. While GSA’s approach minimized the risk of having inadequate funds to fulfill its commitments, it increased the risk that GSA will retain excess funds that could be used for other purposes.

GSA Has Accumulated Adequate Funding for the Transition

GSA has accumulated adequate funding to support its anticipated commitments related to the Networx transition. As of fiscal year-end 2006, GSA accumulated approximately $142 million in a reserve dedicated to costs associated with the transition from FTS2001 to Networx. The uncertainty analysis we performed on GSA’s cost estimate indicates that the funding GSA has already accumulated will most likely be adequate to pay for expected transition costs. By varying the transition traffic factor, our analysis indicates that the $142 million already retained should be adequate to cover expected expenses 96 percent of the time. Figure 2 illustrates the probability that a particular level of funding will be adequate to account for the total actual costs incurred. For example, the $151.5 million estimate represents a confidence level of 100 percent: there is almost no chance that costs will exceed the estimate. The full detail of our methodology is detailed in appendix II.

The merger of the IT and General Supply Funds provides GSA with additional flexibility, further reducing the need to retain the entire amount of the estimate. As discussed, legislation combined the two funds into an Acquisition Services Fund, making their capital assets and balances available for the purposes of the new Federal Acquisition Service. The legislation also allows the Federal Acquisition Service to establish a reserve to retain surplus revenues from GSA’s technology and service programs specifically for the purpose of offsetting losses and other unexpected contingencies. Further, at the end of each year, the statute requires GSA to return to the Treasury any funds not expended or held in a working capital reserve. During fiscal year 2006, GSA’s technology programs experienced losses, but its supply programs reported overall positive earnings of approximately $126 million. If operations continue in this fashion, excess revenue will be available in the combined fund to offset losses or account for contingencies, such as the Networx transition, be retained within the Acquisition Services Fund, or be returned to the Treasury.

With Networx contracts scheduled to be awarded starting in March 2007, GSA will soon be in a position to reassess its main assumption, the transition traffic factor, and the resulting level of funding needed to meet anticipated commitments. Unless GSA revises its estimate, it risks unnecessarily retaining funds that could be reallocated to other agency priorities or returned to the Treasury.

Conclusions

While GSA achieved its goal of minimizing the risk that it would have inadequate funds to pay its transition commitments, the analysis used to develop the transition cost estimate was not sound because it was not sufficiently accurate, comprehensive, documented, or validated. The weaknesses can be attributed in part to a lack of a policy at GSA to ensure that such estimates are developed using best practices and make the best use of agency resources. While the quantifiable effect of the weaknesses in accuracy and comprehensiveness is small, the effect of potential inaccuracies resulting from the intentional use of an overly conservative assumption about a main cost driver could be more significant. Without the use of a cost estimation policy that reflects best practices, GSA could continue to produce similarly unsound estimates, increasing the risk that it will unnecessarily retain funds that could be reallocated for other purposes.

Despite the weaknesses in its analysis, GSA has accumulated adequate funding to support its anticipated commitments related to the Networx transition. Our analysis indicates that it is highly unlikely that GSA will need more than the $142 million it has already accumulated. In addition, the merger of two revolving funds gives it increased flexibility in meeting costs. Once Networx contracts are awarded beginning in March 2007, GSA will be able to forecast the number of forced transitions more accurately, and, if necessary, reduce the amount of funding it plans to accumulate in the future or free already accumulated funds for other purposes. This reassessment will also be an opportunity for GSA to address the other weaknesses in its analysis that resulted from its deviation from best practices.

Recommendations

To improve GSA’s program management, we are making two recommendations. First, to ensure that future cost estimates are sound and can be used as a reliable basis for decisions, we recommend that the GSA Administrator establish a policy for cost estimation efforts at GSA. Specifically, this policy should reflect best practices by requiring that estimates are: accurate (not overly conservative, based on an assessment of the most likely costs, and adjusted properly for inflation); comprehensive (their level of detail ensures that all pertinent costs are included and no costs are double-counted); well-documented (can be easily repeated or updated and can be traced to original sources through auditing); and validated (they have been cross-checked with an independent cost estimate and a level of uncertainty associated with the estimate has been identified.)

Second, to ensure the most efficient use of federal funds, we recommend that the Administrator revise the transition cost estimate following the award of contracts under the Networx program. Specifically, this revision should reflect best practices, include a more precise transition traffic factor, and address the overestimation of service growth, the possible double-counting of a nonrecurring charge, and the effects of inflation during the extended delay in making awards. If the results of this new estimate indicate that the full $151.5 million is not needed to reasonably support the transition effort, GSA should reallocate any excess funds for other purposes allowable within the Acquisition Services Fund or return them to the Treasury.

Agency Comments and Our Evaluation

In written comments on a draft of this report, the GSA Administrator concurred with our recommendations and emphasized the importance of supporting a successful governmentwide telecommunications transition. To address our recommendations, she stated that GSA will issue improved policy guidance on cost estimating and review and adjust the cost estimate as additional information becomes available.

The Administrator also commented that GSA’s transition cost estimate was within 2 percent of our analysis using the same assumptions. However, while we identified only $3 million in quantifiable errors (roughly 2 percent of GSA’s total estimate), our report also states that if the extent of transitioning services is significantly lower than GSA’s intentionally conservative assumption, the actual costs of the transition could be considerably less than GSA’s estimate. Specifically, if all incumbents are awarded Networx contracts and only GSA’s assumption of a 16 percent voluntary transition occurs, the total cost of the transition could be reduced to approximately $40 million—over $110 million less than GSA’s estimate.

In addition, the Administrator stated that GSA does not concur with the entirety of the draft report. She presented two main objections. First, she questioned whether our reported findings were balanced given the facts and results presented. We clarified our report to ensure that our findings better reflect the information we discussed. Second, she raised a concern that we incorrectly suggested comparability between the pending Networx transition and the prior transition. We maintain that the two transitions are comparable, particularly because GSA’s analysis is based, in part, on the results of the previous transition. Specifically, GSA’s analysis relied on experiences and lessons learned from the previous transition to establish costs for the current estimate.

Appendix III provides the full text of GSA’s comments. GSA also provided technical comments that have been incorporated in this report, as appropriate.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the GSA Administrator and interested congressional committees. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

Should you or your offices have any questions about matters discussed in this report, please contact me at (202) 512-6240 or by e-mail at koontzl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine (1) the soundness of the analysis the General Services Administration (GSA) used to derive the estimate of funding that would be required for the transition and (2) whether GSA will have accumulated adequate funding to pay for its transition management costs.

To determine the soundness of GSA’s analysis, we conducted an intensive search of over 250 source documents of both government and industry literature for examples of best practices in the field of cost estimation. This included literature on cost estimation from the Society of Cost Estimating and Analysis, the Department of the U.S. Army, the Office of the Secretary of Defense, the National Aeronautics and Space Administration, and the Department of Energy. This indicated that high quality, reliable cost estimates are: accurate (not overly conservative, based on an assessment of the most likely costs, and adjusted properly for inflation); comprehensive (their level of detail ensures that all pertinent costs are included and no costs are double-counted); well-documented (can be easily repeated or updated and can be traced to original sources through auditing); and validated (they have been cross-checked with an independent cost estimate and a level of uncertainty associated with the estimate has been identified).

To determine the extent to which GSA followed these practices, we analyzed documentation supporting the transition estimate, documentation provided by GSA on the previous transition estimate, and the Networx Request for Proposals. We also interviewed GSA Networx program managers and attended a GSA sponsored transition conference and meetings of the Interagency Management Council Transition Working Group.

To address whether GSA has accumulated adequate funding to pay for its transition management costs, we obtained and analyzed Cost and Capital Requirements Plans for the Information Technology Fund submitted to the Office of Management and Budget and legislation for GSA’s Information Technology, General Supply, and Acquisition Services Funds. In addition, to determine the extent of funding held in GSA’s related accounts, we analyzed financial statements for GSA’s technology and service programs. To verify the reliability of these records, we obtained and analyzed the results of the most recent GSA financial audits and audit reports from GSA’s Inspector General, and we interviewed GSA’s independent financial auditor regarding the quality control procedures in place. The independent auditor did not specifically review the dollar amounts in the Information Technology or General Supply Funds for accuracy but did test the controls in place for compliance with laws and regulations. This auditor stated that there were no reportable findings associated with either fund, and it was reasonable to assume that these accounts were fairly stated. As a result, we determined the data were sufficiently reliable for the purposes of this report. We also interviewed officials from the Office of Management and Budget and officials from GSA’s technology and service programs, Office of the Chief Acquisition Officer, Office of the Chief Financial Officer, and Office of the Chief Information Officer.

To assess the adequacy of the level of funding already accumulated by GSA, we performed an uncertainty analysis for GSA’s estimate using a Monte Carlo simulation. A Monte Carlo simulation provides a perspective on the potential variability of the cost estimate should the facts, circumstances, or assumptions change. We chose to vary only the transition traffic factor because it is the main driver of the costs in GSA’s estimate. To carry out this simulation, we identified a minimum, maximum, and median value for the transition traffic factor based on information received from GSA.

We conducted our work between May 2006 and January 2007 in accordance with generally accepted government auditing standards.

Appendix II: Uncertainty Analysis for GSA’s Transition Estimate

An uncertainty analysis provides decision makers with a perspective on the potential variability of the estimate should the facts, circumstances, and assumptions change. By examining the effects of varying the estimates elements, a degree of uncertainty about the estimate can be expressed, possibly as an estimated range or qualified by some factor of confidence. For example, an estimate that produces a 100 percent confidence level indicates that, based on the methodology used to create that estimate, there is almost no chance that costs will exceed the estimate.

We performed our uncertainty analysis on GSA’s estimate using a Monte Carlo simulation. We chose to vary only the transition traffic factor because it is the main driver of the costs in GSA’s estimate. To carry out this analysis, we identified a minimum, maximum, and median value for the transition traffic factor, based on information received from GSA.

The transition traffic factor represents the possible percentage of services under FTS2001 that may transition to a different provider under the Networx contracts. For this factor, we chose a minimum of 3 percent, which represents the voluntary shift to a different vendor that occurred during GSA’s previous transition to FTS2001. For the maximum, we used 76 percent, as this value was chosen by GSA officials as a worst-case scenario in an effort to mitigate the risks of underestimating costs. The median of these two numbers is 39.5 percent. Table 3 shows the variations of each factor used in our uncertainty analysis.

Appendix III: Comments from the GSA Administrator

Appendix IV: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contact named above, James R. Sweetman, Jr., Assistant Director; Jamey A. Collins; Neil J. Doherty; Jennifer K. Echard; Wilfred B. Holloway; Ethan J. Iczkovitz; Frank Maguire; Karen A. Richey; Glenn D. Slocum; and Amos A. Tevelow made key contributions to this report.