Public Transportation: FTA's Triennial Review Program Has Improved, But Assessments of Grantees' Performance Could Be Enhanced
Released: 2009-07-30T00:00:00

Letter

Background

As one of 10 administrations within the Department of Transportation, FTA provides financial assistance to transit agencies to develop new systems and improve, maintain, and operate existing systems in all 50 states, the District of Columbia, and some U.S. territories. FTA maintains its headquarters in Washington, D.C., with 10 regional offices throughout the continental United States, and employs about 500 of the approximately 60,000 employees of the department. Figure 1 summarizes key information about FTA’s 10 regions, including the total amount and funds for the Urbanized Area Formula Program.

In 1994, FTA laid out a common framework for regional and headquarters staff to administer oversight reviews in FTA Order 5400.1. That order, which remains in effect today, describes the process for conducting various oversight reviews and taking corrective actions. In addition, the order established an internal working group called the Oversight Review Council to develop oversight review schedules, coordinate review schedules to avoid unnecessary duplication and overlap, and achieve a comprehensive, uniform, and cohesive approach, among other things.

The triennial review provides FTA with a valuable opportunity to routinely evaluate the use of federal funds for about 600 urban area grantees at least once every three years—or about 200 grantees each year. It covers many areas, such as procurement, financial management, and drug and alcohol programs. Appendix II contains FTA’s descriptions of all 23 triennial review areas. FTA also uses a similar, general review to oversee grantees that receive rural area formula funds—a state management review.

In addition to the more general triennial and state management reviews, FTA conducts special oversight reviews with a narrowed subject matter focus, such as procurement system reviews, which help ensure that grantees (especially 30 grantees that receive large amounts of federal transit funds) are purchasing buses and trains using competitive bidding practices, as well as special civil rights reviews, which help ensure that grantees are, among other things, providing public transportation to the disabled and adhering to equal employment opportunity practices. Grantees also experience non-FTA reviews, such as single audits that examine grantees’ financial statements, inspector general audits, and various state audits. Table 1 describes FTA’s primary oversight reviews.

The triennial review covers 23 triennial review areas, but given the need to review all of these areas, the time constraints (estimated at about 130 hours for each review), and in order to reduce the oversight burden upon grantees, FTA does not conduct an in-depth document review or inspection. For example, FTA normally samples no more than 15 financial records during site visits, conducts a spot check of facility and maintenance records, and may only inspect a few buses and subway cars during the review. However, according to FTA officials, when an area of concern or deficiency is noted, FTA will expand the review to examine these areas more in-depth. Figure 2 summarizes the major steps in the triennial review process.

The triennial review also gives FTA an opportunity to provide technical assistance to individual grantees—assistance specifically aimed to help individual grantees resolve specific deficient findings resulting from triennial reviews. FTA also helps grantees prepare for the triennial review by holding annual triennial review workshops throughout the country and offering formal training courses focused on specialized subject areas through the National Transit Institute—a training institute funded by an FTA grant.

While officials from both FTA headquarters and regional offices are ultimately responsible for implementing the triennial reviews, they have increasingly relied on contractors since the program was created in the early 1980s. After we reported in 1992 that FTA had not fully utilized contractors for oversight activities, it has, since 1996, used contractors to conduct all triennial reviews. FTA, through its Office of Program Management, awarded multiple-year, firm, fixed-price contracts totaling about $4 million each year to four contractors to conduct the fiscal years’ 2008 through 2012 triennial reviews.

To assist contractors with the reviews, FTA developed the Triennial Review Contractors’ Guide. FTA has expanded the contractors’ guide from its original 29 pages to over 200 pages in 2008. The contractors’ guide provides the questions to ask grantees for each of the 23 triennial review areas, a list of grantees’ documents that contractors are to examine, and a method that contractors are to use for determining whether grantees are meeting federal requirements.

The contractors’ guide assists contractors with the decision-making process about what constitutes a deficient finding by providing condition statements for the 23 triennial review areas. For example, according to the contractors’ guide, if the grantee has included the lobbying prohibitions clause (that is, the prohibition from using federal funds to lobby for additional federal funds) in its procurement solicitations, then the grantee is not deficient; otherwise, the grantee is deficient. The contractors’ guide also contains suggested corrective actions if a grantee is found deficient, (e.g., the grantee must certify to FTA that it will have subgrantees, contractors, and subcontractors comply with the lobbying prohibitions requirement.) In another example, if FTA conducted a special procurement system review (see table 1) in the prior two fiscal years or if one is scheduled for the year of the grantee’s triennial review, then a review of the procurement area is not necessary and a notation is made in the triennial review report as “not reviewed.” Subsequently, contractors enter predetermined, deficiency codes into OTRAK that correspond to explanations for any deficient findings, herein called findings. Once entered, FTA regional offices track and monitor the finding until the grantee addresses it. Table 2 shows the four categories FTA uses to document triennial review results.

FTA’s Triennial Review Data Show Many Grantees Are Not Making Progress Toward Meeting Federal Requirements, and FTA Could Do More to Improve Grantees’ Performance

From fiscal years 2000 through 2008, triennial review data show many grantees are not making progress toward meeting federal requirements, as indicated by not consistently reducing grantees’ findings with federal requirements. While FTA uses several tools to help grantees meet more requirements, especially in areas with the greatest number of findings, it could do more to improve grantees’ performance by identifying the underlying causes and by objectively noting the severity of the finding.

Over Two-Thirds of the 424 Grantees We Analyzed Did Not Consistently Meet More Federal Requirements From Fiscal Years 2000 through 2008, But Practices Used by Some Grantees Could Improve Other Grantees’ Performance

Our analysis of OTRAK data shows that 51 percent of 424 grantees had mixed results in terms of meeting more triennial review requirements from fiscal years 2000 through 2008, as demonstrated by both increases and decreases in their numbers of findings over three consecutive reviews. Over the same time period, 17 percent of grantees consistently met fewer requirements from one review to the next, as indicated by a consistent increase in the numbers of findings in their triennial reviews. Thirty-one percent of grantees consistently met more federal requirements over time—one of the goals of the triennial review—as demonstrated by a consistent decrease in their numbers of findings over three consecutive reviews. Finally, 1 percent of grantees met the same number of requirements in all three reviews. Figure 3 summarizes the performance in terms of meeting federal requirements for 424 of the approximately 600 current urban area formula fund grantees, as measured by the change in the number of findings in three triennial reviews from fiscal years 2000 fiscal years 2000 through 2008. through 2008.

While comparing the change in the numbers of findings over time does not show improvement across all grantees, a different analysis of OTRAK data shows that some grantees averaged fewer than two findings per triennial review, which we identify for this analysis as high-performing grantees. Figure 4 illustrates the range in the average number of findings after three triennial reviews for each of the 424 grantees from fiscal years 2000 s 2000 through 2008. through 2008.

This analysis of OTRAK data shows that about 31 of 424 grantees averaged fewer than 2 of 229 possible findings per triennial review. Twenty-six of these grantees provide transportation services to small or medium-sized urban communities, located in various parts of the country.

Three of the 10 grantees we interviewed were high-performing grantees. Executives representing these three grantees implemented some management practices that they believe contributed to their high performances. One management practice included linking the job descriptions of pertinent individuals to the triennial review—a practice that grantee executives said contributed to a culture of accountability. Specifically, the controller’s and grants administrator’s job descriptions for the large grantee explicitly required these individuals to be responsible for the triennial review. An assistant director of public works, responsible for managing the transit system of a small grantee, was required to help implement FTA requirements, among other things. In addition, according to the chief executive officer of the medium grantee, the officer set verbal job expectations with managers of zero findings, which the officer believed contributed to the high performance in the triennial review.

Another practice implemented by 2 of the 3 high-performing grantees was to initiate external reviews by independent consultants to assess their operations, which contributed to, or is expected to sustain, their high performance. Executives from a large grantee we interviewed attributed their high performance to an external review conducted several years ago that focused on the grantee’s readiness for the triennial review. Despite having zero findings in their last two consecutive triennial reviews, executives from the medium grantee recently completed an external maintenance review that they believe should help them achieve their goal of operating an efficient transit system and that is necessary to remain good stewards of public funding.

FTA officials recognize the importance of sharing best practices among grantees. As such, it advocates a grantee mentoring system where regional offices, within resource and budget restraints, initiate, encourage and direct grantees to team with recognized grantee leaders in order to improve grantee performance. FTA officials also indicated that they address mentoring during their triennial review workshops.

FTA Helps Grantees Meet More Requirements, Especially in Five Review Areas with Most Findings; However, FTA Does Not Identify the Underlying Causes of Findings

When we analyzed OTRAK data from fiscal years 2000 through 2008 by area, we found that the 424 grantees had the greatest numbers of findings in five review areas—procurement, drug and alcohol program, maintenance, technical, and satisfactory continuing control areas. These five areas accounted for about 54 percent of all findings over the last nine years. This may not be unexpected since we also found a positive relationship between the number of questions asked for certain areas and the number of findings. Specifically, grantees tended to have higher numbers of findings in those areas where the contractors’ guide requires contractors to ask more questions when compared to other areas. For example, for fiscal year 2008, contractors asked grantees 20 questions in the procurement area, as compared to 2 or 3 questions in other areas, such as school bus and half-fare, which had far fewer findings. Figure 5 shows the percentage of triennial review findings for the top 5 areas, compared to the other 18 areas for the same 424 grantees over the nine-year period.

FTA has also analyzed OTRAK data on past triennial reviews, identified similar areas with the greatest number of findings, and provided resources at a national level to help grantees address the findings in these areas. According to several end of year reports, FTA analysis of deficiency codes related to the procurement area showed grantees lacking justification for non-competitive awards, lacking open competition, having no contract administration system, and having no written record of procurement history. For the drug and alcohol area, deficiency codes pointed to grantees having policies that were out-of-date or lacked required elements, and conducting random drug and alcohol tests below the federally required level. To help grantees improve their performance in areas with the greatest number of findings, FTA has provided an increasing amount of technical assistance to individual grantees; workshops that focus on, among other things, procurement related matters; and training. In its 2003 end of year report, FTA noted that a decline in findings from fiscal year 2000 to 2003 indicated the effectiveness of its efforts to improve grantees’ performance.

We could not definitively determine whether, or to what degree, FTA’s technical assistance, workshops, and training improved grantees’ performance because while workshops increased from 2005 to 2008, so did the total number of findings. Other factors could have contributed to a reduction of grantees’ findings over the last nine years. The reduction in findings could have been caused by contractors reporting that they did not review certain areas for some grantees that had recently experienced a special oversight review or by FTA conducting a pilot study in 2005 to streamline the triennial review process, which also resulted in some areas not being completely reviewed. Further, grantees, as well as FTA headquarters and regional officials, cite a shift over the years in how FTA conducts the triennial reviews—from a “gotcha” auditing approach to a customer-service approach, helping grantees quickly resolve findings during the reviews and lessening the need to report the findings.

Although FTA analyzes deficiency codes in OTRAK data to determine problem areas, these codes alone do not describe the underlying causes for the findings. Generally accepted government auditing standards recommend that federal agencies, such as FTA, identify, to the extent possible, the underlying cause in developing findings as this provides persuasive evidence on the factor(s) responsible for the problem. Deficiency codes pointing to grantees lacking open competition or not having a written record of their procurement history, for example, is not an underlying cause, as grantees’ ability to meet procurement requirements may be caused by grantees not having sufficient or knowledgeable staff to initiate a competitive contracting process. If FTA would identify these as common underlying causes, it could prescribe, as part of the necessary corrective action, that grantees attend specific procurement training courses. Knowing the common underlying causes of grantees’ findings could provide FTA with valuable information to help many grantees find solutions to correct their findings and improve grantees’ long-term performance. FTA officials stated that during the workshops, they plan to emphasize ways to identify systemic deficiencies.

Officials expressed concern about their authority to use oversight funds for the purpose of undertaking additional efforts to help the grantees meet or exceed requirements by identifying the common underlying causes for less than satisfactory levels of performance. Officials told us that they have the authority to use oversight funds for providing individual grantees with technical assistance—assistance necessary to satisfactorily address specific findings resulting from the grantee’s triennial review. However, they acknowledged that they can help grantees develop capabilities to improve their long-term performance in other ways, such as National Transit Institute courses, that do not require using oversight funds.

FTA Plans to Improve the Data It Collects on the Severity of Findings

While FTA collects data on the numbers, the areas, and the reasons for the findings by the deficiency codes in OTRAK, the information it collects on severity of findings is not objective. OTRAK requires that contractors and FTA regional staff enter a level of severity—”high,” “medium,” and “low”— for each finding. However, individuals entering severity data into OTRAK labeled almost 98 percent of findings as medium, which makes it difficult for FTA to prioritize technical assistance in problem areas. According to FTA officials, contractors or regional staff made judgments about the level of severity because FTA has not developed definitions from which to make this judgment. This could mean the information on severity is not consistent across the country. FTA officials said that they plan to improve their data collection efforts for the fiscal year 2010 triennial reviews by establishing definitions for the three levels of severity, which could provide more meaningful findings data and add significant value to the information FTA collects to target its technical assistance for the purposes of improving grantees’ performance.

The Triennial Review Program Uses Some Strong Management Practices, but Can Be Improved in a Few Areas

Strengths of the triennial review include a well-defined process, the increased use of an electronic tracking system to monitor grantees’ performance, increased timeliness in issuing triennial review reports, and the willingness of FTA staff to seek out opportunities for continuous improvement. However, we found a few areas where FTA could improve the triennial review process by ensuring and documenting grantees receive a complete review at least once every three years and by coordinating the schedules of multiple oversight reviews in the same fiscal year.

The Triennial Review Program Has A Well- Defined Process and Continues to Improve

One strength of the triennial review program is that the process is well defined. FTA Order 5400.1 and the Triennial Review Contractors’ Guide detail the roles and responsibilities for those involved in the triennial review, including FTA headquarters, regional offices, contractors, and grantees. These documents outline the process, identify the questions that contractors are to ask grantees, specify the types of documents that grantees are to provide to FTA, and describe any follow up required of regional staff. Preparing policies and procedures that outline individual responsibilities meets an important element of strong federal grant accountability best practices.

Another strength is an increased use of an information system to track and monitor grantees. In 1998, we criticized FTA for not making full use of its information system because its regional staff had not been uniformly updating the system and its headquarters staff was not enforcing its use as a management tool. FTA upgraded OTRAK in 2003 from a local-area computer network to a Web-based system, which allows headquarters and regions to share information and track oversight activities more efficiently. FTA has continued to incrementally update this system. In addition, the contractors’ guide directs contractors to input the findings from each review into OTRAK and all regional staff and contractors we spoke with reported using OTRAK. Performance information, such as that recorded in OTRAK, helps agencies identify, prioritize, and manage potential at-risk grantees.

FTA has strengthened its triennial review reporting process by issuing more timely triennial review reports than in previous years. In 1998, we criticized FTA for taking an average of 182 days after the site visit to process its 1995 and 1996 triennial review reports and recommended that FTA follow its established 90-day time frame. FTA subsequently established a performance measure in 2001 of issuing 95 percent of triennial review reports within 30 days of completing a review. FTA has significantly improved its performance in issuing reports in recent years. FTA reported it issued between 82 to 94 percent of triennial review reports within 30 days of completing the review from fiscal year 2004 through 2008. During that same time period, only 2 of 902 reports were issued more than 60 days after the review.

Finally, FTA has looked for ways to continuously improve by contracting with Booz Allen Hamilton to examine its oversight review processes and recommend ways to improve them. In January 2008, Booz Allen Hamilton issued its final report, outlining four areas of improvement: program planning, program execution, program evaluation, and change management. We found that FTA is taking steps to respond to recommendations in these areas, including developing a strategic approach for prioritizing each of the Booz Allen Hamilton recommendations and developing a Web-based, master calendar feature in OTRAK that is expected to improve the visibility of future oversight review schedules to assist both FTA headquarters and regional staff in coordinating the oversight review schedules.

FTA Could Better Ensure and Document that Grantees Receive a Complete Review at Least Once Every Three Years

FTA is required to conduct a complete review and evaluation of grantees’ adherence with statutory and administrative requirements at least once every three years. In reviewing 198 fiscal year 2008 triennial review reports, we identified, but could not confirm, 33 instances where FTA may not have completely reviewed and evaluated grantees in all areas. Specifically, contractors recorded in the 2008 triennial review reports that certain areas were “not reviewed” because FTA had conducted special oversight reviews on these areas in the prior two years—fiscal years 2006 or 2007. For example, contractors recorded in the triennial review reports of ten grantees that the drug and alcohol program area was “not reviewed” during the 2008 triennial review because these grantees had received special drug and alcohol program reviews in fiscal year 2006. Because the area was not reviewed during the 2008 triennial review, these grantees may not be reviewed in this area for 5 years (this area was last reviewed during the 2006 special oversight review and will not be reviewed again until the 2011 triennial review). While FTA’s contractors’ guide describes the flexibility to not review certain areas, FTA regional office staff provides triennial review contractors with the approval to not review certain areas when a special oversight review has been conducted in the prior two years or is scheduled in the fiscal year of the triennial review.

FTA officials indicate that their practice is to update its review and evaluation of an area having an oversight review in the prior two years. However, the guidance does not always instruct contractors to document this update in the triennial review reports to help FTA ensure it meets the statutory requirement of a complete review at least once every three years. As an example, the contractors’ guide describes such a process for reviewing and evaluating the planning area, telling contractors to review previous planning certification review reports during the triennial review desk review, verify the status of any findings, and determine if corrective actions have been implemented on schedule. The contractors’ guide does not provide similar guidance for other areas, including the drug and alcohol program area, and we did not see—nor could FTA provide us with— any such documentation in the 2008 triennial review reports. In commenting on our draft report, FTA said it would revise applicable guidance, including the contractors’ guide for fiscal year 2010, to avoid ambiguity and ensure consistency within the triennial review program.

FTA headquarters officials indicated that regional officials and contractors discuss all 23 areas during the triennial review process and identify concerns about the grantees. Specifically, headquarters officials explained that regional staff and contractors discuss the results of special oversight reviews conducted in the prior two years, such as any outstanding findings from the prior reviews, and reach consensus on the extent to which a grantee would be reviewed in the related area during the triennial review, ranging from an area not being reviewed at all to a more detailed review in the area, depending on the situation. However, officials acknowledged that the results of those discussions are not always documented in the triennial review reports, especially in regards to the extent to which areas are to be reviewed or not reviewed during the triennial review. Because of this lack of documentation, we could not confirm in the 33 instances noted above whether or to what extent all 23 areas were reviewed and evaluated.

By allowing flexibility in the contractors’ guide to not review certain areas, FTA may not be ensuring that all areas are completely reviewed and evaluated at least once every three years, as required by statute. In addition, because grantees can and do experience findings during triennial reviews, despite having special oversight reviews in the prior two years, updating previously conducted special oversight reviews is important. We identified seven instances where grantees had received special oversight reviews in the financial management and civil rights areas in fiscal years 2006 and 2007, yet during the 2008 triennial review, findings in these areas resurfaced. Finally, proper evaluation and documentation provides future reviewers with a more transparent audit trail of findings and gives FTA managers the confidence that grantees were completely reviewed and evaluated in all 23 areas at least once every three years. Officials plan to refine and use OTRAK to document, among other things, the results of special oversight reviews that occurred in the years between triennial reviews, which would help ensure grantees adhere to requirements.

Uncoordinated Scheduling of Multiple Oversight Reviews May Unduly Burden Some Grantees

We found a limited number of instances in which grantees may have been unduly burdened by uncoordinated scheduling of multiple oversight reviews in the same fiscal year. We found that 23 of 198 grantees (12 percent) receiving a triennial review in fiscal year 2008 also received one or more special oversight reviews in the same year. Twenty-four of 183 grantees (13 percent) received multiple reviews in fiscal year 2007, including one medium sized grantee that underwent a special procurement system review in January 2007, a drug and alcohol program review in February 2007, and a triennial review in June 2007. The same grantee also underwent two special civil rights reviews—an Americans with Disabilities review in October 2007 and a Disadvantaged Business Enterprises review in November of 2007.

One grantee with whom we spoke said that although the triennial review is beneficial, the multiple oversight reviews it received in the same fiscal year were burdensome, redundant and delayed project schedules and cost the grantee about $18,000 to $20,000. Four other grantees provided information showing they spent about 100 to 300 hours preparing for and hosting 2- to 3-day triennial reviews involving sometimes over 20 individuals and many departments. Some grantees also said that special oversight reviews continue for two or more weeks and require even more grantee time and attention.

Representatives from the American Public Transportation Association—an association that represents many grantees—also said that two or more FTA oversight reviews in the same year could be burdensome. According to the association representatives and grantees with whom we spoke, multiple reviews in the same year are especially burdensome for smaller grantees, whose employees tend to be responsible for multiple review areas. For example, a small grantee told us that the triennial review alone taxed its two employees who were responsible for addressing many of the areas in the triennial review. Large grantees, on the other hand, typically have several employees in a division responsible for one triennial review area, so the burden of multiple reviews may be less taxing for them than for small grantees.

The decision-making body responsible for developing and coordinating FTA’s oversight review schedules is the Oversight Review Council; but, according to FTA officials, the Oversight Review Council is not fulfilling these responsibilities, as it was inactive on and off in recent years because of management changes and priorities. FTA Order 5400.1 requires the Oversight Review Council to develop and coordinate various oversight review schedules to avoid unnecessary duplication and overlap, among other things. The Council was reinstituted in 2007 after management changes, and continues to redefine its roles and responsibilities. Further, FTA is aware of the burden oversight reviews place on grantees and works to limit this burden.

Without the Oversight Review Council taking responsibility for developing and coordinating oversight review schedules, FTA offices developed their own oversight processes, which, in some instances, could lead to grantees receiving a triennial review and a special review in the same fiscal year. For example: FTA’s goal is to conduct special procurement reviews every three years on the 30 grantees that receive large amounts of federal transit funds.

The Offices of Civil Rights and Safety and Security conduct their own special oversight reviews, including reviews of grantees that have had civil rights complaints filed against them. According to officials from the Civil Rights office, after notifying the regions and others of the schedules, the regions are to notify headquarters about any scheduling conflicts.

The drug and alcohol review schedules are sometimes made without informing regional staff because this information is sensitive.

Other FTA officials who conduct special oversight reviews enter their scheduled review date of grantees in OTRAK. However, regional staff said they are sometimes not aware of special review scheduling.

In receiving urban area formula funds, grantees agree to the congressionally-mandated triennial review at least once every three years. Some grantees are subject to other FTA oversight reviews. Having an effective body to coordinate the strategic planning of special oversight reviews, in coordination with the triennial review, could help increase FTA’s oversight coverage of grantees, while at the same time, minimize the undue burden of uncoordinated multiple oversight reviews on some grantees. FTA’s new Web-based, master calendar feature in OTRAK is expected to improve the visibility of future oversight review schedules.

FTA’s Two Triennial Review Performance Measures Meet Some, But Not All Key Attributes of Successful Performance Measures

FTA established two performance measures to assess timeliness of steps in the triennial review program: (1) 80 percent of triennial review findings are to be closed within 30 days of their due date (“Close Findings Timely”); and (2) 95 percent of the final triennial review reports are to be issued within 30 days of the completion of the review (“Issue Reports Timely”). While these measures meet some, they do not meet all key attributes of successful measures, as described in our past work on the Government Performance and Results Act of 1993. Further, FTA lacks performance measures that assess other core elements of FTA’s triennial review program, including the outcome and quality of the reviews. FTA officials recognize the limits of their current performance measures and have begun efforts to enhance them. Table 3 summarizes the 9 key attributes we used to analyze FTA’s performance measures.

Figure 6 compares FTA’s triennial review performance measures to these key attributes, showing that they meet some but not all attributes. Both measures meet 4 of 9 attributes; that is, they link to agency goals, provide a measurable target, are clear, and have limited overlap with other measures. The Issue Reports Timely measure meets 2 additional attributes by demonstrating objectivity and reliability.

Linkage. We found FTA’s two triennial review performance measures link throughout the organization—from FTA regional offices, through headquarters, and to the department level. Both performance measures aligned with the Department of Transportation’s strategic goal of organizational excellence in the FTA Annual Performance Plan, FTA regional business and strategic plans, and individual performance appraisals. In addition, FTA officials—including officials in the Performance Management Division, the Office of Program Management, and all 10 regional offices—confirmed in interviews that performance goals were communicated across the agency. Such communication is important to ensure FTA officials understand what the organization is trying to achieve and the goals it seeks to reach.

Measurable target. Both performance measures also have measurable numeric targets—given in percentages—that allow FTA officials to easily assess whether overall goals and objectives were achieved. Numeric targets facilitate assessments of whether objectives were achieved because comparisons can be easily made between projected performance and actual results. FTA reports meeting the Close Findings Timely measure in fiscal years 2004 through 2007; and in fiscal year 2008, closed 74 percent of findings within 30 days. Although FTA reports never meeting the Issue Reports Timely measure in fiscal years 2004 through 2008, it has greatly improved the timeliness with which it issues triennial review reports. From fiscal years 2004 through 2008, FTA nearly met its goals in 4 of the 5 years.

Clarity. We found the performance measures are clearly stated and each has a definition that is consistent with the methodology used to calculate it. The Issue Reports Timely formula is the percentage of reports issued within 30 days of the final conduct of the review, and the Close Findings Timely formula is the number of findings closed within 30 days of the due date, divided by the sum of the number of findings closed plus the number of findings open that are 30 days past the due date. Clear measures provide transparency and allow managers to accurately assess FTA’s performance in issuing triennial review reports and closing findings in a timely manner.

Limited overlap. Both program performance measures have unique names, definitions, and do not overlap. One measure evaluates the timeliness of issuing reports, while the other assesses the timeliness of closing findings. The presence of distinct performance measures prevent managers from sifting through unnecessary or redundant information.

Objectivity. We found the Issue Reports Timely measure is objective because the data elements used to calculate it are free from significant bias. However, our analysis of the Close Findings Timely measure found that its objectivity is limited because decisions to close findings, establish due dates for when grantees must address findings, and extend, if necessary, any due dates, is done largely at the discretion of regional staff, in consultation with grantees, and, when appropriate, headquarters officials. This presents opportunities to influence this measure because regional staff may use different criteria for when to close or extend deadlines. While FTA officials recently met to discuss a mandatory process to uniformly close findings across all FTA regions, the current lack of criteria reduces the transparency of FTA’s performance in closing findings. For example, an FTA official stated that the Closing Findings Timely performance measure could result in the unintended consequence of prematurely closing findings. The annual performance appraisals and resulting compensation for regional staff, including the Regional Administrators, are based, in part, on meeting the Close Finding Timely performance measure. While we did not find widespread instances when findings were prematurely closed, an official from the civil rights office stated there have been cases when regional staff closed triennial review findings in OTRAK and requested that the civil rights office monitor the grantees’ progress in resolving them.

Reliability. We found the Issue Reports Timely measure is reliable, as FTA has standard procedures for collecting data, and based on our examination of individual triennial review reports, accurately stated the number of reports issued in the most recent end of year report for fiscal year 2008. However, our analysis of data used to calculate the Close Finding Timely performance measure found data inaccuracies and, therefore, the measure does not meet the key attribute of reliability. After examining closed findings data we found several instances where OTRAK reported findings open more than seven years after they were due for closure. FTA officials familiar with the database said that these inaccuracies resulted from FTA regional staff not entering data into OTRAK after the transition to the new Web-based information system in 2003 and said, but could not document, that the findings were likely closed. However, we also found OTRAK data entry errors from fiscal years 2005 through 2007—after FTA had transitioned to the Web-based system—where the database showed almost 200 findings were open more then one year after they were due for closure. Headquarters officials said, but could not document, that these findings are inaccurately listed as open in OTRAK and were likely closed at the regional level. Officials further stated they plan to re-examine the regional process for closing findings and issue guidance to improve this process. Finally, the 2008 Booz Allen Hamilton report noted instances of missing data in OTRAK because some regions were using stand alone spreadsheets instead of fully using the database to report closing findings. Officials acknowledged that entering closed findings data into the database can be problematic, especially for inexperienced regional staff not familiar with OTRAK. Officials believe that a 2007 decision to tie regional staff performance appraisals to data entered into OTRAK should provide an incentive to ensure future data are reliable. In addition, officials said they are emphasizing the importance of recording data into OTRAK to regional offices and contractors. Without reliable data, FTA managers cannot be confident in the extent to which performance goals have been achieved, including the timeliness in closing findings.

Core program activities. Both triennial review performance measures do not meet the triennial review program’s core program activity, evaluating whether grantees meet federal requirements over time. While FTA’s existing performance measure of Close Findings Timely may increase adherence to federal requirements by encouraging grantees to address findings, it does not provide an overall program level assessment of grantees’ performance over time. In addition, FTA measures annual grantee compliance information in Triennial Review Program end-of-year reports, but has not developed performance measures that set targets to evaluate progress. Therefore, FTA does not assess whether the triennial review program is increasing grantees’ ability to continuously meet federal requirements. FTA could measure the core program activity of the triennial review program in various ways, such as assessing grantees’ performance at the end of each review year or assessing grantees’ long term performance, as our previous analysis of findings data shows. See figure 3.

Governmentwide priorities. FTA’s current performance measures partially address governmentwide priorities by ensuring that FTA reports on its timeliness in conducting reviews and in addressing corrective actions. However, the measures do not evaluate other governmentwide priorities, including the outcome of the reviews—the extent to which grantees are meeting federal requirements, or the quality of the triennial reviews.

Outcome. The performance measures assess the output of the triennial review program—the closing of findings and issuing of final reports— but, not the outcome—grantees’ ability to meet federal requirements. As previously stated in the discussion of core program activities, outcome performance measures—such as grantees’ performance over time, or annual progress against a measurable target—can help evaluate the results of the triennial review programs. Without such a measure, FTA is limited in the extent to which it can effectively ascertain and document the extent to which the triennial review program is improving grantees’ performance in meeting federal requirements and FTA managers have limited information on program performance, making the possibility of achieving program goals less likely.

Quality. FTA has no performance measures for assessing the quality of the triennial review program. FTA officials help ensure quality by checking triennial review reports to determine whether the contractors made reasonable findings for grantees, determining what actions grantees must take to resolve findings, and attending some site visits to oversee contractors. However, FTA officials are unsure how to develop a measure for evaluating the quality of the triennial review program. In addition, Booz Allen Hamilton reported that FTA headquarters and field staff do not have a common understanding of what constitutes a quality review or how it could be measured. Without common quality standards, FTA cannot evaluate the overall quality of the program nor ensure the results and objectives of the triennial review program are being met. Performance measures related to quality could assess the implementation of triennial reviews, including a way to oversee contractors and test the overall quality of reviews. Many federal agencies periodically review their programs to ensure they meet quality standards. FTA officials recognize the benefits of such performance measures and plan to enhance its current measures and standards to assess the quality and value added of the triennial review program to help ensure that the results and objectives of the triennial review program are being met.

Balance. FTA’s triennial review program performance measures address only one of three governmentwide priorities and, therefore, are not balanced—an attribute that applies to the overall suite of measures, rather than the measures individually. Balanced performance measures allow managers to understand what other triennial review aspects are working well and why. Adding outcome and quality performance measures will help balance the existing timeliness measures and could allow managers to better set triennial review priorities by not overemphasizing one or two priorities at the expense of others. FTA officials recognize current performance measures do not assess some aspects of the triennial review program and are undertaking efforts to enhance them.

Conclusions

Strong FTA oversight of federal surface transportation grant programs helps ensure that grantees meet federal requirements, provide safe public transportation, and safeguard from misuse the billions of dollars that Congress appropriated for surface transportation programs to help America recover from the economic downturn and the billions more that Congress plans to reauthorize for surface transportation. FTA’s efforts over the past decade to improve the congressionally-mandated triennial review—the primary means for overseeing hundreds of grantees—have produced concrete results. For example, FTA now regularly collects oversight data on grantees, issues the vast majority of triennial review reports within 30 days of the review (a significant improvement since our last report in 1998), and provides grantees with technical assistance to address problem areas. Nonetheless, FTA should undertake additional efforts to further improve the triennial review program.

Although one of the goals of the triennial review program is to assist grantees in meeting more federal requirements over time, our analysis of OTRAK data shows many grantees with mixed success in reducing their numbers of findings over three triennial reviews from fiscal years 2000 through 2008; and 17 percent of the grantees we reviewed met fewer federal requirements in their latest triennial review than they did in their two previous reviews. While FTA analyzes apparent problem areas, its analysis does not provide information on the underlying causes of findings and information on the level of severity for each finding. Without conducting further analyses of its OTRAK data and better understanding the underlying causes of grantees’ performance in triennial reviews, FTA may be missing valuable opportunities to increase its effectiveness in helping grantees improve their performance.

The triennial review process could be improved in two respects. First, when a grantee has recently received or is scheduled to receive a special oversight review, the flexibility currently allowed in the contractors’ guide could lead to confusion about how to document that all 23 triennial review areas are completely reviewed at least once every three years. Until FTA revises its guidance to remove the “not reviewed” category, require an updated evaluation of areas reviewed in the prior two years of the triennial review, and describe how to document triennial review reports with the updates, FTA does not provide assurance that grantees are being assessed equally and are receiving a complete review and evaluation at least once every three years, as required. Second, until the Oversight Review Council performs its role as oversight review coordinator or delegates this role to another FTA body, uncoordinated, multiple oversight reviews may not effectively increase FTA’s coverage of grantees and may place undue burden on a few grantees.

Finally, although FTA’s two triennial review performance measures meet some of the key attributes we have previously reported are necessary for a successful program, the Close Findings Timely measure could be improved to minimize the possibility for manipulation and increase the reliability of closed finding data. Without an internal control process to verify whether grantees have satisfactorily resolved findings, FTA cannot determine the validity of reported results for the Close Findings Timely performance measure. Without reliable data, FTA may be unable to determine the validity of reported results and make appropriate decisions. FTA also lacks performance measures for evaluating the outcome and quality of the triennial review program. Such performance measures are essential because they can help decision makers determine areas for improvement and increase the likelihood of achieving program goals, such as increasing the number of grantees that meet federal requirements.

Recommendations for Executive Action

To support the triennial review program’s goal of having grantees consistently meet more federal requirements, the Secretary of Transportation should direct the FTA Administrator: 1. To further analyze and use OTRAK data on the triennial review to help grantees improve and sustain their performance. As part of this analysis, FTA should consider and identify, when necessary: (1) high performing grantees and their management practices, among other things; (2) the areas in which problem findings occur, and the underlying causes of these findings; and (3) the level of severity for each finding. 2. To help ensure that a complete performance review and evaluation of each grantee is conducted at least once every three years, in accordance with the statutory requirement, revise the contractors’ guide to remove the “not reviewed” category, require an updated evaluation of areas reviewed in the prior two years of the triennial review, and describe how contractors should document the triennial review reports with updates to other reviews, such as special oversight reviews conducted in the prior two years. 3. To strengthen the triennial review process by ensuring that the Oversight Review Council, or another body, carries out the responsibilities specified in FTA Order 5400.1 to be actively involved in coordinating oversight review schedules in order to increase FTA’s oversight coverage of grantees while, at the same time, minimizing the undue burden of multiple oversight reviews on grantees. 4. To improve the objectivity and reliability of the “Close Findings Timely” performance measure by: developing an internal controls process to verify grantees have satisfactorily resolved findings before closing them; continuing to emphasize the use of OTRAK across all regions and developing a process to close out triennial reviews in OTRAK upon grantees’ completion of corrective actions. 5. To meet standards set forth in previous GAO work, based on the Government Performance and Results Act of 1993, and develop performance measures to assess: the outcomes of the triennial review program, such as a method for evaluating improvements in grantee performance in meeting more federal requirements over time; and the quality of the triennial review through improved contractor oversight, testing, or inspection.

In addition, FTA should ensure additional outcome and quality performance measures are balanced with the existing measures for assessing the timeliness of the triennial review program so that one priority is not emphasized at the expense of others.

Agency Comments

The Department of Transportation reviewed and commented on a draft of this report. The department generally agreed with the report’s findings and recommendations. It also suggested that we provide more information on FTA’s efforts to review grantees at least once every three years and coordinate its oversight reviews, which we added, as appropriate. The Department also provided a technical correction, which we have incorporated.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to appropriate congressional committees, to the Secretary of Transportation, and to appropriate FTA officials. The report will also be available at no charge on the GAO Web site at www.gao.gov.

If you have any questions about this report, please contact me at (202) 512- 2834 or herrp@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

To determine the extent to which the triennial reviews indicate grantees met federal requirements from fiscal years 2000 through 2008, we reviewed documentation on the Federal Transit Administration’s (FTA) Oversight Tracking System (OTRAK)—the official electronic information system for tracking and monitoring oversight activities. Documents included a system description, requirements, and background; reference guides; data entry controls and processes; data dictionary; deficiency codes; and the users’ manual. We requested and obtained OTRAK findings data by grantee and by area from fiscal years 2000 through 2008. Data prior to 2000 was not available. We sorted the data to, among other things, determine the extent to which grantees’ performance in meeting federal requirements increased, decreased, both increased and decreased, or remained constant in three triennial reviews; which grantees averaged fewer than two findings in their triennial reviews; and which areas accounted for the greatest numbers of findings. We did not have direct access to the OTRAK database, but took several steps to assess the reliability of the data. We performed electronic testing to identify missing data, obtained and reviewed the queries that FTA used to generate the data that we requested, and discussed with FTA and its OTRAK contractors the results of the information. Based on these tests, we excluded 150 grantees because they did not meet our criteria, having exactly three triennial reviews in the last nine years and excluded 27 other grantees because, according to FTA officials, they received triennial reviews for multiple triennial review cycles, more than 3 fiscal years, or both as a result of triennial reviews that were either canceled or closed without findings.

To address the strengths and weaknesses of the triennial review process, we reviewed federal guidance for improving grant accountability, federal internal control standards, GAO reports on performance measures, FTA Order 5400.1, triennial review reports of numerous grantees, and a 2008 Booz Allen Hamilton report of FTA’s oversight programs. We also reviewed and compared schedules for the triennial and other special oversight reviews by grantee from fiscal years 2006 through 2008 to determine which grantees experienced both a triennial review and at least one special oversight review in fiscal year 2008. In addition, we examined specific areas reviewed or not reviewed during fiscal years 2007 and 2008 triennial review reports and compared the areas to completed or planned special oversight reviews. We attended the triennial review of the Los Angeles County Metropolitan Transit Authority in September 2008. We also attended a FTA triennial workshop for new grantees in Kansas City, Missouri in July 2008.

We selected 10 grantees with a range of findings, from various FTA regions, having special oversight reviews in fiscal years 2006, 2007, or 2008, and from different population sizes—small (50,000-199,999), medium (200,000 to 999,999), or large (1 million or more) urban centers. We interviewed these grantees in order to understand grantees’ practices that contributed to them averaging fewer than 2 findings per review, understand the burden multiple reviews had on grantees’ operations, and understand if the triennial review unduly burdened grantees with different size operations. The following lists the 10 grantees interviewed: Southwest Ohio Regional Transportation Agency, Cincinnati, Ohio Spokane Transit Authority, Spokane, Washington St. Joseph Transit, St. Joseph, Missouri Utah Transit Agency, Salt Lake City, Utah Southeastern Pennsylvania Transportation Authority, Philadelphia, Santa Monica Blue Bus, Santa Monica, California Wichita Transit, Wichita, Kansas Hampton Roads Transit, Hampton, Virginia City of Hattiesburg, Mississippi Santa Clara Valley Transportation Authority, San Jose, California With respect to this and our other reporting objectives, we interviewed officials from the following organizations: FTA’s Associate Administrator for Program Management FTA’s Office of Civil Rights FTA’s Office of Safety and Security FTA’s Office of Program Management FTA’s Office of Performance Management All FTA regional offices All four triennial review contractors (Reid Consulting LLC, CDI/DCI Joint Venture, Advanced Systems Technical and Management Inc., and Interactive Elements LLC)

Booz Allen Hamilton The American Public Transportation Association Community Transportation Association of America.

To determine FTA’s performance measures for the triennial review and to what extent FTA’s performance measures meet nine key attributes, we obtained, reviewed, and analyzed, among other documents: GAO reports describing the importance of establishing effective performance measures, GAO guidance on assessing the strengths and weaknesses of federal agencies’ performance measures and identifying the key attributes of successful performance measures, FTA triennial review end-of-year reports, and FTA performance appraisal templates. We also spoke with FTA management to identify the current performance measures for evaluating the success of the triennial review program, the reasons for them, and when they were established. We then compared FTA’s two performance measures to each of the nine key attributes in order to determine the extent to which and why they either fully met, partially met, or did not meet all attributes.

Appendix II: Description of FTA’s 23 Triennial Review Areas

Appendix II: Description of FTA’s 23 Triennial Review Areas FTA’s Description of the Requirements in Each Area The grantee must be eligible and authorized under state and local law to request, receive, and dispense FTA funds and to execute and administer FTA-funded projects. The authority to take all necessary action and responsibility on behalf of the grantee must be properly delegated and executed.

The grantee must demonstrate the ability to match and manage FTA grant funds, maintain and operate federally-funded facilities and equipment, and conduct an annual independent organization-wide audit in accordance with the provisions of OMB Circular A- 133.

The grantee must be able to implement the Urbanized Area Formula Grant Program of Projects in accordance with the grant application, Master Agreement, and all applicable laws and regulations, using sound management practices.

The grantee must maintain control over real property, facilities, and equipment and ensure that they are used in transit service.

The grantee must keep federally-funded equipment and facilities in good operating order.

FTA grantees will use their own procurement procedures that reflect applicable state and local laws and regulations, provided that the process ensures competitive procurement and that the procedures conform to applicable federal law.

The grantee must comply with the policy of the Department of Transportation that DBEs are ensured nondiscrimination in the award and administration of Department of Transportation-assisted contracts. Grantees also must create a level playing field on which DBEs can compete fairly for Department of Transportation-assisted contracts; ensure that only firms that fully meet eligibility standards are permitted to participate as DBEs; help remove barriers to the participation of DBEs; and assist the development of firms that can compete successfully in the marketplace outside the DBE program.

Per “Buy America” law, federal funds may not be obligated unless steel, iron, and manufactured products used in FTA-funded projects are produced in the United States, unless FTA has granted a waiver, or the product is subject to a general waiver. Rolling stock must have sixty percent domestic content and final assembly must take place in the United States.

To protect the public interest and prevent fraud, waste, and abuse in federal transactions, persons or entities, which by defined events or behavior, potentially threaten the integrity of federally administered programs, are excluded from participating in FTA-assisted programs.

Recipients of federal grants and contracts exceeding $100,000 must certify compliance with restrictions on lobbying before they can receive funds. In addition, grantees are required to impose the lobbying restriction provisions on their contractors.

The grantee must participate in the transportation planning process in accordance with FTA requirements; Safe, Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for Users requirements; and the Metropolitan and Statewide Planning Regulations.

The grantee must ensure that no person in the United States shall, on the grounds of race, color, or national origin, be excluded from participating in, or denied the benefits of, or be subject to discrimination under any program, or activity receiving federal financial assistance.

FTA’s Description of the Requirements in Each Area The grantee is expected to have a written copy of a locally developed process to solicit and consider public comment before raising a fare or carrying out a major reduction of transportation services.

Grantees must ensure that elderly persons and persons with disabilities, or an individual presenting a Medicare card will be charged, during non-peak hours for transportation using or involving a facility or equipment of a project financed under Section 5307, not more than 50 percent of the peak hour fare.

Titles II and III of the Americans with Disabilities Act of 1990 provide that no entity shall discriminate against an individual with a disability in connection with the provision of transportation service. The law sets forth specific requirements for vehicle and facility accessibility and the provision of service, including complementary paratransit service.

FTA grantees are prohibited from using federally-funded equipment and facilities to provide charter service except on an incidental basis and when one or more of applicable exceptions set forth in the charter service regulation applies.

FTA grantees are prohibited from providing exclusive school bus service unless it qualifies under specified exceptions. In no case can federally-funded equipment or facilities be used.

The grantee must collect, record, and report financial and non-financial data in accordance with the Uniform System of Accounts and updated with the National Transit Database Reporting Manual.

Any recipient of Urbanized Area Formula Grant Program funds must annually certify that it is spending at least one percent of such funds for transit security projects or that such expenditures for security systems are not necessary.

FTA grantees are required to maintain a drug-free workplace for all employees and to have an ongoing drug-free awareness program.

Grantees receiving FTA funds under Capital Grant (Section 5309), Urbanized Area Formula Grant (Section 5307), or Non-Urbanized Area Formula Grant (Section 5311). Programs must have a drug and alcohol testing program in place for all safety-sensitive employees.

The grantee must ensure that no person in the United States shall on the grounds of race, color, creed, national origin, sex, age, or disability be excluded from participating in, or denied the benefits of, or be subject to discrimination in employment under any project, program, or activity receiving federal financial assistance from the federal transit laws.

ITS projects funded by the Highway Trust Fund and the Mass Transit Account must conform to the National ITS Architecture, as well as to Department of Transportation adopted ITS Standards.

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact:

Staff Acknowledgments

In addition to the contact named above, Catherine A. Colwell (Assistant Director), Matthew A. Cook, Elizabeth H. Curda, Bert Japikse, Delwen A. Jones, Grant M. Mallie, Steve Martinez, Amanda K. Miller, Joshua H. Ormond, Kelly L. Rubin, and Sandra E. Sokol made key contributions to this report.