Agricultural Conservation: USDA Should Improve Its Methods for Estimating Technical Assistance Costs
Released: 2005-01-03T00:00:00

Letter

Background

In 1994, the Congress established NRCS and gave it jurisdiction over programs of the former Soil Conservation Service, as well as other USDA financial or technical assistance programs for natural resource conservation and rural development. With more than 12,000 employees nationwide—about three-fourths located in its state offices and 2,500 field offices—NRCS focuses primarily on private and other nonfederal lands. NRCS staff regularly work in partnership with state, local, and private entities, using the same case files and technical assistance tools. NRCS’s primary partners are state conservation agencies and the approximately 3,000 conservation districts nationwide. These conservation districts are units of local government organized to support local conservation efforts with their own programs and staff.

When asked to do so by farmers and others, NRCS staff assess farmers’ needs—a process that generally involves traveling to the site. NRCS staff work with the landowner to develop a conservation plan that describes the strategy to be used, the schedule of activities, and estimated costs. In some instances, plans are revised several times until the landowner selects a final alternative. If the landowner applies to implement the conservation plan under a Farm Bill program and the land meets the eligibility criteria and is accepted, NRCS develops a contract. After a contract is developed and signed, NRCS staff complete paperwork for payments to the landowner. NRCS staff assist with installation of practices, for example, by surveying land, providing practice standards and specifications, and ensuring contractors have carried out the terms of the contract. In addition, NRCS continues to document activities throughout the life of the contract, which may be years or decades. Staff also periodically certify that the participant is complying with the contract terms, depending on the program requirements.

In 1998, the Chief of NRCS called for a new agencywide effort to improve NRCS’s accountability by providing better information and analyses on how the agency uses its resources and what it achieves with its funds. As part of these efforts, NRCS has taken steps to estimate its future technical assistance costs and budgets, as described in figure 1.

NRCS’s model for estimating the cost of programs consists of an Excel spreadsheet that makes program-by-program calculations using such data as time per task estimates and salary benefits and support costs, as well as assumptions about the average length of a contract and the proportion of work that will be performed during each year of the contract. Figure 2 illustrates the information used in NRCS’s cost of programs model.

Time Per Task Estimates

NRCS state offices have overall responsibility for developing the time per task estimates used in the model. NRCS divided the country into 218 areas and assigned teams of about 7 to 12 NRCS and partner staff to develop time per task estimates for the work to be done in each area. According to NRCS’s process for developing time per task estimates, these teams first define the typical work in their area—that is, the sizes and types of land they work with and the conservation practices they plan for this land. For example, a Texas team determined that one type of typical conservation work in its area was planting grass cover on a 150-acre dairy farm. Using this description, the team discussed and agreed on estimates of time required for each of the 29 tasks associated with performing this work, such as the time it would take them to design a grass-planting plan. Generally, after developing the time per task estimates, the team leader submits them to the state time per task leader to review and input into an NRCS database. The methods of review the state offices use include comparing the areas’ estimates with each other. If the state leader encounters substantial unexplained differences among estimates, he or she generally contacts the team leader for an explanation and to have the estimate changed, if appropriate. The state leader then submits final time per task estimates to NRCS headquarters, which also reviews them. If questions arise, headquarters may ask the state office to perform additional review, and make changes, if warranted.

Since the quality of data entered into a model affects the estimates it produces, NRCS has worked to increase the reliability of its time per task estimates by developing new estimates on three separate occasions and asking for staff feedback on the quality of the estimating process. In addition, NRCS plans to update its time per task estimates again because the tasks and the time required to perform them have changed, along with program requirements and policies, over the past 5 years.

Salary Costs

NRCS’s cost model also includes data on average salary costs, which are based, in part, on data from NRCS’s time and attendance system. NRCS staff enter hours that they work by program, and by the activities associated with those programs, biweekly in the time and attendance system. For example, they might record working 20 hours on CRP and 20 hours on WRP. For the 20 CRP hours, they may record 10 hours to determine land eligibility for multiple landowners and 10 hours working with landowners on conservation plans. These data are reviewed first by supervisors and then at the state office level.

NRCS’s Technical Assistance Cost Estimates Differ from Actual Costs Reported by NRCS

NRCS tested cost estimates from its model for fiscal years 2002 and 2003 by comparing them with the agency’s actual costs. Our analysis of these comparisons shows that program-by-program estimates from NRCS’s model vary considerably from the agency’s actual costs as shown in table 1. These results do not meet the agency’s goal of achieving a difference of no more than 10 percent between estimates and reported costs.

For fiscal year 2002, the CRP estimate was closest to the actual costs—it was 5 percent lower than the actual costs reported. Of the remaining program estimates for 2002, three were higher than the actual cost data by 48 percent to 302 percent, and four were lower by 19 percent to 36 percent. Altogether, NRCS estimated that its technical assistance costs for eight of the Farm Bill conservation programs would be about $254 million, 19 percent higher than its actual costs of about $213 million. NRCS did not estimate costs for two programs that had not yet been implemented.

For fiscal year 2003, the estimate for the Agricultural Management Assistance Program, a program that provides financial assistance to producers in 15 states to, among other things, construct or improve irrigation structures and plant trees, was closest to the reported costs— it was 9 percent greater than the actual costs. Of the remaining program estimates for 2003, six were higher than the actual cost data by 17 percent to 50 percent, and three were lower by 16 percent to 60 percent. Altogether, NRCS estimated its technical assistance costs for 10 Farm Bill conservation programs would be about $295 million, about 15 percent higher than its actual costs of about $257 million. This estimate was 4 percent closer to total actual costs than in fiscal year 2002.

For the three largest programs—the Environmental Quality Incentives Program, CRP, and WRP—the estimates varied from the actual cost data somewhat less in fiscal year 2003 than 2002. In fiscal year 2002, the estimates had a spread of 70 percent—from a 22 percent underestimate to a 48 percent overestimate. In fiscal year 2003, the estimates had a spread of 38 percent—from a 16 percent underestimate to a 22 percent overestimate.

Differences in Estimated and Actual Costs Have Several Causes

We identified several reasons for the differences between NRCS’s estimated and actual costs. First, for fiscal years 2002 and 2003, the model estimated costs based on the assumption that programs would be fully funded in the beginning of the fiscal year. This did not happen, however, because the 2002 Farm Bill was enacted later than expected and because USDA operated under a continuing resolution for a good portion of fiscal year 2003. According to NRCS officials, as a result, less technical assistance work was performed than the estimates reflected. Second, NRCS’s model includes costs for work performed by partners’ staff and paid for by the partner organizations, while the actual cost data generally contains only the costs for the work of NRCS’s staff. Third, NRCS’s model uses some data that are based on inaccurate assumptions. This is likely to have contributed to differences between estimates and actual costs reported by NRCS.

Actual Timing of NRCS Work Differed from Timing Assumed When Estimating Costs

In several instances, NRCS performed technical assistance work at different times than NRCS originally assumed it would when estimating technical assistance costs. First, NRCS estimates have assumed that full funding would be available for new contracts at the start of the fiscal year, but in practice, this has not occurred. For example, in 2002, the Farm Bill was enacted in May, later than NRCS expected, and OMB apportioned funds to USDA for implementing the Farm Bill programs in July—about three quarters of the way through the fiscal year. Since only 2-1/2 months of the fiscal year remained, different work—and in some cases, less work— was performed under the Farm Bill than NRCS had anticipated, according to NRCS officials. The general 2002 sign-up period for CRP contracts did not start until August 2002, limiting the amount of work performed on new CRP contracts. Moreover, in fiscal year 2003, USDA operated under a continuing resolution until receiving fiscal year 2003 appropriations in February and an OMB apportionment in March—about half way through the fiscal year. While NRCS can adjust its assumptions, it is not possible to eliminate uncertainties and variances related to the timing of funding approvals that cause differences between the estimated and actual program costs. NRCS officials said that they have been studying the model’s estimates and modifying some assumptions about workloads to improve the estimates.

NRCS Model Included Costs for Work Performed by Partners

Because NRCS’s partners’ efforts are a relatively important part of overall technical assistance efforts, NRCS has included its partners’ costs in its model so that it is in a position to estimate total technical assistance costs to carry out the programs. For example, according to 1999 NRCS staff estimates, the most recent available, NRCS’s partners were responsible for about 17 percent of total CRP costs and 15 percent of total WRP costs that year. Using those percentages, we calculated that NRCS’s partners might have added about $15 million to these two programs’ fiscal year 2002 technical assistance costs.

To further illustrate the effects of including partners’ costs, we conducted two comparisons. First, we compared NRCS’s fiscal year 2002 technical assistance cost estimates, which include partners’ costs, for two programs—CRP and WRP—with the costs reported by NRCS. We then made the same comparison but with partners’ costs excluded. (See table 2.) The results show that the differences between the estimated and actual costs increase when partner costs are excluded. For the CRP program, for example, the difference between the estimated and reported costs increases from a 5 percent underestimate to a 22 percent underestimate when partner costs are excluded. For WRP, the difference between the estimated and reported costs also increases from a 22 percent underestimate to a 34 percent underestimate when partner costs are excluded.

However, NRCS officials believe that the differences between their estimates, which include partners’ costs, and the reported actual costs are less significant because their partner costs have decreased—they could not say by how much. This may not be the case, however. An NRCS briefing document reported in 2000 that NRCS expected its partners to perform more of its workload than in 1999. Similarly, most of the officials we spoke with from conservation districts—which are key partner organizations— reported that conservation districts have increased the amount of technical assistance work they have performed in the past few years. Finally, partner’s costs should not be included when NRCS reports its costs, and when it compares its estimated and actual costs. In commenting on a draft of this report, NRCS officials noted that it is important to include partner costs when estimating the full costs of the programs. They also stated that estimated costs are used for budgetary purposes and actual costs are the only costs charged to the government and used in final reports.

Time Per Task Data Used in the Model Are Based on Inaccurate Assumptions

NRCS’s estimates of technical assistance costs for 10 Farm Bill conservation programs are developed, in part, using time per task data that are based on inaccurate assumptions. First, NRCS’s basic assumption used for developing its estimates—descriptions of typical technical assistance work—oversimplifies the situations field staff encounter to the extent that the resulting estimates do not accurately represent the time it takes staff to do the work. Second, NRCS’s time per task estimates are based on some assumptions that do not reflect actual workplace conditions. Finally, some NRCS staff who provide technical assistance are uncertain about how to allocate the time they spend traveling to and from field locations among NRCS’s programs.

Descriptions of Typical Work

NRCS uses descriptions of typical work as the basis for its time per task estimates. NRCS guidance tells teams to develop descriptions of their typical work, but does not tell them how to do this for areas in which conditions are diverse. In the absence of more specific guidance, staff have encountered difficulty with the concept of typical work. According to one team leader, for example, the guidance does not tell teams how to determine what is typical when there are significantly varying land sizes, types of farms, or conservation practices, and several team leaders told us that such variations made identifying typical work difficult or impossible. For example, a North Dakota team leader said that the conservation work in his area varies considerably, ranging from installing pipelines to creating ponds to building fences. Moreover, a Wisconsin team leader told us that at least three typical descriptions would be needed to represent the variety of farms in his area: one each for dairy, vegetable, and cranberry farms. Other NRCS staff commented that it may not be possible to describe what is typical for western rangeland, where operations vary from hundreds of acres to tens of thousands of acres. Finally, another team leader told us that NRCS’s guidance is vague, and as a result, his team interpreted the term “typical” to mean average, while another understood it to be the median or middle value. NRCS staff raised these concerns in 2000, and NRCS officials are aware of the difficulties staff have encountered in describing their typical work. According to these officials, the scope of this estimating problem is nationwide and resolving this concern is important for making more accurate estimates. NRCS is considering whether it is possible to resolve difficulties staff face in describing typical work by enlarging the number of areas for which estimates are developed or collecting data for more than one “typical” unit per area, and thereby reducing the extent of diversity within each. Additional information will be needed to determine whether such an approach will succeed.

Assumptions about the Workplace

NRCS has directed its staff to base its time per task estimates on three assumptions that we believe do not reflect actual workplace conditions. First, estimating staff are to assume that all NRCS staff are fully trained. This is not the case, however. About 10 percent of current NRCS staff were hired between 2001 and 2004. According to one NRCS official, staff need 1 to 1.5 years to become able to independently perform most technical assistance for CRP and 3 to 5 years for WRP. Because not all staff are fully trained, assuming that they are is likely to inappropriately lower the time per task estimates. The second assumption is that staff are not interrupted during their workday. Under normal conditions, staff regularly experience interruptions that decrease productivity. Assuming that this is not the case is also likely to contribute to inappropriately lowering the time per task estimates. A third assumption, which would likely lead to raising the time per task estimates, is that NRCS staff completely follow NRCS’s policies, procedures, and guidance in performing work. Actually, however, staff sometimes take shortcuts that do not comply with all policies, procedures, and guidance—thereby completing tasks faster than expected.

In contrast, NRCS’s reported actual costs are based on actual work conditions. That is, these costs reflect the additional time taken by new and partially trained staff, the added time caused by interruptions that staff regularly face, and the timesaving shortcuts that staff sometimes take. Although we could not determine the precise effects of these assumptions, some information is available to indicate that they warrant reexamination. For example, NRCS staff reported that by using shortcuts, their CRP and WRP work took 24 percent and 31 percent less time, respectively, than the time they had estimated in their workload analysis. NRCS officials said that they have adjusted the model to take into account some policy and procedural changes that reduce the workloads of NRCS staff and added that they would also reconsider the assumptions that we identified.

Allocation of Travel Time

NRCS staff also reported some confusion about how to allocate their travel time. The guidance directs staff to divide travel time, which constitutes an important portion of field staff work time, among different program activities when necessary. Staff usually drive to meet with landowners and view land, often traveling to distant locations and working on several program activities with several farmers during a single trip. When this occurs, they must determine how much travel time they should assign to each program. The guidance states that travel time should be “prorated” among different program activities but does not explain how to do this. This lack of guidance results in reporting inconsistencies. For example, staff often visit several farms on a single trip making it difficult to determine how to prorate this time among multiple program activities. NRCS officials said that they are aware of this problem but have not yet developed a solution.

Conclusions

While we recognize that only 2 years of comparative cost data are available and that NRCS has been striving to improve its technical assistance cost estimates, NRCS’s cost estimates differ enough from actual costs reported by NRCS to be of concern to those who use these estimates. NRCS’s overall technical assistance cost estimate for 10 Farm Bill conservation programs for fiscal year 2003 is closer to the reported cost than the estimate was for fiscal year 2002, but too much variation is evident on a program-by-program basis in both years. Until improvements in NRCS’s technical assistance cost estimating are demonstrated through tests of the model’s results, we believe NRCS cost estimates should be used with caution. Without identifying costs incurred by its partners when assessing the reasonableness of the estimates made by its model, NRCS cannot ensure the validity of its cost comparisons. Also, unless NRCS modifies its assumptions to better reflect actual workplace conditions, its technical assistance cost estimates will not be as precise as they could be. Finally, without pilot testing its plans for improving descriptions of typical work or other changes in data development, NRCS cannot be assured that its investment in its next nationwide workload analysis will be well spent. As NRCS improves the quality of its workload analysis, including its time per task estimates, and the assumptions used in the model, we believe more accurate technical assistance cost estimates will be developed. Moreover, when these improvements have been made, NRCS will be in a better position to evaluate the overall quality of its estimating. Further testing in the years ahead may well be needed to gain a better understanding of the causes of variations in the program-by-program cost estimates.

Recommendations

To improve the accuracy, and therefore the usefulness of NRCS’s program cost estimating, we are recommending that the Secretary of Agriculture direct the Chief of NRCS to take the following three actions: clearly identify nonreimbursable costs incurred by NRCS’s partners when presenting estimates of NRCS’s costs, ensuring that its model’s estimates are comparable with actual data; change the assumptions used for developing time per task data for the model so that they better reflect actual work conditions; and pilot test the feasibility of proposed changes in the development of the time per task data, including changes in development of typical work descriptions in several diverse areas of the country before proceeding with another nationwide workload analysis.

Agency Comments and Our Evaluation

We provided a draft of this report to USDA for review and comment. We received oral comments from NRCS, which are summarized below. We also received technical comments, which are incorporated in this report as appropriate. NRCS accepted our findings and said they would develop actions in response to our recommendations. The agency stated that the report provides the basis for updating the agency’s workload analysis, more accurately estimating partner contributions to NRCS’s programs, and making other necessary adjustments to the assumptions in cost estimates. NRCS also stated that our report rightly points out that some assumptions used in estimation were inaccurate, but that portions of the report had an unnecessarily, negative tone. They noted for example that the estimates would have been closer to actual costs if funding had been available at the beginning of fiscal years 2002 and 2003. We agree that earlier funding would have likely helped close the gap between estimated and actual costs. NRCS also stated that partner contributions could be excluded from the model’s estimates, but that NRCS wanted to acknowledge the full cost of programs using partner costs. We agree that partners’ costs can and should be excluded from the model’s estimated costs when cost estimates are used for budgetary purposes or for comparison with actual costs. Lastly, NRCS commented that we found no problems with the logic of the model. We disagree. The model’s inclusion of costs that the agency did not incur, such as partners’ costs, is inappropriate when comparing estimates to NRCS’s actual cost data.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this letter. We will then send copies to interested congressional committees, the Secretary of Agriculture, the Chief of NRCS, and other interested parties. We will also make copies available to others on request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please call me at (202) 512-3841. Key contributors to this report are listed in appendix II.

Objectives, Scope, and Methodology

Our objectives were to (1) review the technical assistance cost estimates produced by the model and (2) identify the sources of differences that may occur between the estimates and NRCS’s reported costs.

To review the estimates, we assessed the differences between the NRCS’s model results compared with the actual costs reported by NRCS. We compared NRCS’s fiscal years 2002 and 2003 technical assistance cost estimates by program with the actual costs reported by NRCS, the only years for which these two sets of costs were available. To identify sources of difference between these costs, we assessed assumptions and data used in the cost estimates. To assess NRCS’s cost model, we analyzed the model and related documentation to determine whether the model appeared to be a reasonable method for estimating program costs. In addition, we checked whether the model formulas, contained in a MS Excel file, used the appropriate data, and we reviewed the formulas to ensure that they were logical. In addition, we replicated the model formulas using the proper data and ensured that the resulting figures matched those shown in the model. We interviewed NRCS officials responsible for developing the cost model to gain an understanding of the model and its development. These officials were staffed in NRCS’s Budget Planning and Analysis Division, Operations Management and Oversight Division, and in field offices.

We conducted sensitivity analyses to illustrate the possible importance of different variables in the model. These sensitivity analyses were conducted using Monte Carlo simulation, which uses random numbers to measure the effects of uncertainty on model output—in this case, the technical assistance cost estimates. Our analysis was based on general assumptions about the probability distributions characterizing some of the variables in the model.

Also, we interviewed and requested documentation from 20 officials of state government agencies and conservation district associations to assess whether the nonreimbursed contributions of conservation districts increased or decreased in the past several years. We did so for each of the 10 states with the most CRP and WRP contracts. The 10 states are Illinois, Iowa, Kansas, Louisiana, Minnesota, Mississippi, Missouri, New York, North Dakota, and Wisconsin. Of the 20 officials we contacted, 13 responded to our request and provided some information about partners’ contributions.

To assess NRCS’s means for ensuring the reliability of the data used in the cost model, we traced the time per task and other data to their respective sources, which included agency reports and databases. Since information regarding data sources was not always readily available in writing, we met with NRCS officials who described and provided the sources of the model’s data. Once we identified the source, we verified that the data had been correctly transferred from the source to the model. In addition, we performed limited reliability tests, primarily tests for omitted entries and outliers, of the available source data. Furthermore, to obtain NRCS’s field staff views on the reliability of the time per task estimates and time and attendance data used in the model, we used a semistructured interview guide to interview all 10 officials leading estimate development efforts in the 10 states that had the most CRP and WRP contracts in 2002. The 10 states had over half of NRCS’s total CRP contracts and over 40 percent of its WRP contracts. Using another semistructured interview guide, we interviewed 10 randomly selected NRCS team leaders (out of 44 leaders) who each led a team developing time per task estimates in one area in each of the 10 states. However, we could not assess the quality of NRCS’s reviews of its workload analysis at state offices because NRCS state officials retained insufficient documentation of the reasons for changes in data made during their reviews.

To assess the reliability of other data used in NRCS’s model, we reviewed NRCS’s method for developing overhead cost data and salary cost data, which relies on the time and attendance system. Overall, we noted that USDA obtained an unqualified opinion on its financial management activities in fiscal years 2002 and 2003. This opinion covers, in part, the salary cost data that NRCS relies on when reporting its costs. In addition, we reviewed NRCS’s development of overhead cost information that is based on Office of Management and Budget budget object classifications, which include such costs as rent, utilities, equipment, and supplies. In addition, we reviewed an NRCS draft report about the quality of NRCS time and attendance data. That 2001 NRCS draft report found that about half of NRCS field offices had deficiencies in documenting their use of time, but the report did not provide sufficient detail to reveal the precise extent of the problems. Since then, NRCS has implemented corrective actions, according to agency officials, and has verified on a limited basis that improvement has occurred.

Our issues with the reliability of data are discussed throughout the report. We performed our work between September 2003 and October 2004 in accordance with generally accepted government auditing standards.

GAO Contacts and Staff Acknowledgments

GAO Contacts

Acknowledgments

In addition to the individuals named above, Charles W. Bausell Jr., Nancy Crothers, Anne E. Dilger, Beverly A. Peterson, Lynne M. Dunlay, and Judy K. Pagano made key contributions.