Background

FMCSA and state law enforcement agencies in partnership enforce safety  standards for the more than 500,000 interstate motor carriers operating  in the United States. States and, to a lesser extent, FMCSA staff,  perform roadside inspections of vehicles to check for driver and  maintenance violations and then provide the data from those inspections  to FMCSA for analysis and determinations about a carrier’s safety  performance. FMCSA also obtains data from the reports filed by state and  local law enforcement officers when investigating commercial motor  vehicle accidents or regulatory violations. FMCSA provides grants to  states that may be used to offset the costs of conducting roadside  inspections and improve the quality of the crash data the states report to  FMCSA. In addition, FMCSA’s field offices in each state, known as  divisions, have investigators who conduct safety reviews of carriers  identified by state inspection and other data as unsafe or at risk of being  unsafe. Most states augment FMCSA investigators’ efforts by reviewing  carrier operations as well.
Before CSA, FMCSA relied primarily on comprehensive compliance  reviews on-site at carriers to determine whether they were operating  safely. Carriers were selected for these reviews based on safety  assessments generated by FMCSA’s statistical enforcement model— SafeStat—that used data obtained from accident reports and other safety  data supplied by FMCSA’s state partners (see table 1). During these  reviews, an investigator would visit a motor carrier to assess compliance  with safety regulations by interviewing company officials and reviewing  records that pertain to alcohol and drug testing of drivers, insurance  coverage, crashes, driver qualifications, the number of hours a driver has  worked within a certain time period, vehicle maintenance, prior  inspections, and transportation of hazardous materials. FMCSA officials  believe that such comprehensive compliance reviews are an effective  way to assess a carrier’s safety performance. However, compliance  reviews are extremely resource intensive; therefore, only a small  percentage of the motor carrier industry can be evaluated in this manner,  given limited federal and state resources. Annually, for example, FMCSA  and its state partners have conducted compliance reviews of about 3  percent of registered motor carriers. As a result, FMCSA was not able to  evaluate the vast majority of registered motor carriers and most were not  assigned a safety rating.
In 2004, FMCSA began to design and develop CSA, a program to better  target resources toward unsafe carriers, deploy a more comprehensive  array of interventions, and proactively evaluate safety performance based  on data, rather than solely based on compliance reviews. Through  implementation of CSA, FMCSA expects to assess a larger portion of the  motor carrier industry and to increase the emphasis on driver safety.  Additionally, FMCSA expects to use data to identify unsafe carriers and  drivers earlier to address safety problems before crashes occur. In this  way, FMCSA intends to create a culture of compliance, in which officials  and carriers will work together to address safety issues early, and carriers  will have access to information and resources that can help them better  comply with safety regulations. FMCSA officials expect this approach will  more efficiently use FMCSA and its state partners’ resources. FMCSA  expects to significantly reach, or “touch,” more carriers—thus improving  their safety—and ultimately reduce motor carrier crashes, injuries, and  fatalities.
To date, FMCSA has focused its implementation efforts on carriers— examining the safety performance of the company—whether it be a  trucking company with hundreds of vehicles or a small company  operating one or two trucks. FMCSA’s implementation efforts also include  an increased assessment of the safety behavior of the drivers for carriers  selected for intervention. FMCSA also intends to rate or determine the  fitness of all drivers, regardless of whether the carriers they work for are  selected for intervention. The rating would cover such things as whether  the driver was driving while impaired by drugs or alcohol or received  tickets for moving vehicle violations.
SMS—the first oversight activity under CSA—is intended to allow FMCSA  to more accurately assess a carrier’s safety performance. SMS is applied  to safety data obtained primarily from roadside inspections as well as  from crash reports. These data are sorted into six Behavior Analysis and  Safety Improvement Categories (BASIC) that are associated with unsafe  performance according to FMCSA’s analysis. In addition to the six  BASICs, SMS also incorporates data based on a carrier’s crash  involvement (see table 1).
Once the data are sorted into the seven data categories, the SMS  algorithm measures and generates scores for the carrier’s safety  performance in each category. Carriers are placed into peer groups (i.e.,  other carriers with similar numbers of inspections or size) and ranked  according to performance. The rankings determine which carriers are not  operating with optimal safety practices and, therefore, will be prioritized  for intervention. CSA is intended to improve upon SafeStat, which  measured safety in only four safety evaluation areas: driver, vehicle,  safety management, and accident (equivalent to the SMS Crash  Indicator). CSA uses a wider array of safety data to create a more  nuanced understanding of a carrier’s safety performance and presents  that information using more refined categories.
FMCSA has made carriers’ SMS scores available to carriers themselves  as well as to the public, including shippers and insurers. Carriers are  allowed to request reviews of any data they believe are incorrect through  an FMCSA system known as DataQs. These requests for review can  include moving violations reported by state authorities that carriers  believe are invalid or mistakenly attributed to the wrong carrier. FMCSA  forwards each request for review to the state in which the carrier was  cited. States then research the issue, often by contacting the inspector  who conducted the inspection and his or her supervisor. Based on this  research, states decide if the violation is warranted and make changes if  necessary.
All of these safety data are collected and maintained in FMCSA’s existing  Motor Carrier Management Information System (MCMIS). Our previous  work assessed FMCSA data reliability and identified problems with the  quality of crash data reported to FMCSA, including data that were  inaccurate, incomplete, and not reported in a timely manner. FMCSA  has been making efforts to improve crash data quality, including awarding  Safety Data Improvement Program grants to states to improve their crash  data. States’ efforts to improve crash data include expanding electronic  reporting; improving the timeliness, completeness, and accuracy of  reporting; and standardizing police accident report forms.
The second oversight activity under CSA is the introduction of a variety of  interventions for interceding with carriers when their SMS scores indicate  safety deficiencies. The expanded array of interventions available under  CSA offers FMCSA more flexibility and the opportunity to apply  interventions commensurate with a carrier’s safety performance (see  table 2). The new interventions were created to get carriers to improve  behaviors linked to possible crash risk. As a result, these carriers have  the opportunity to take corrective actions to avoid another intervention in  the future. Under CSA, interventions that involve investigations follow a  process known as the Safety Management Cycle which will expand  investigations from simply identifying what violations occurred to  determining why violations exist so that FMCSA can offer more  constructive improvement recommendations.
While some of the interventions, such as Notice of Violation and Notice of  Claim, available under CSA are not new, FMCSA intends to apply them in  a more systematic manner under CSA. For example, according to  FMCSA, the agency only issued a handful of Notices of Violation over the  past 5 years because prior FMCSA information technology systems did  not provide the capacity to issue and track them. Under CSA, Notices of  Violations can be issued in conjunction with Cooperative Safety Plans,  giving carriers a framework in which to address the violations. In another  example, the agency intends to increase its use of the Notice of Claim.
The third oversight activity under CSA is determining a carrier’s fitness to  operate motor vehicles, known as a Safety Fitness Determination.  FMCSA plans to use SMS scores to make a Safety Fitness Determination  to indicate whether a carrier should continue to operate or should be  suspended from operating (i.e., be ordered “out-of-service”). Currently,  FMCSA determines a carrier’s fitness to operate based only on the  outcome of an onsite comprehensive investigation, similar to how it was  done under SafeStat. If a review shows that a motor carrier is unfit to  operate pursuant to governing regulations, FMCSA can issue an Out-of- Service order that prohibits the carrier from operating until the  deficiencies are corrected. However, as part of CSA, FMCSA plans to  initiate a rulemaking that will enable it to use SMS-generated scores to  determine if carriers are unfit to operate. FMCSA has not determined if  the same categories currently used to determine if a carrier is fit to  operate—“satisfactory,” “conditional,” and “unsatisfactory”—will be used,  but it does not plan to increase the number of categories.
In 2008, FMCSA launched an operational-model test (pilot) of the CSA  program in four states and later expanded the pilot to five more states over  30 months through June 2010. During Phase 1, four states (Colorado,  Georgia, Missouri, and New Jersey) tested CSA on carriers with the  exception of those with the poorest SafeStat ratings.  Fifty percent of the  non-excluded carriers in each state were subject to certain aspects of the  CSA model—specifically a subset of the BASICs and the interventions— and the other 50 percent were subject to SafeStat. During Phase 2, the  carriers subject to CSA in those four states, including those excluded from  Phase 1, were then subjected to all of the BASICs and interventions. Later,  FMCSA added Delaware, Kansas, Maryland, Minnesota, and Montana to  the pilot testing, with 100 percent of the carriers in each state subject to all  of the BASICs and interventions. UMTRI analyzed the results of Phase 1 of  the pilot as well as supplementary results from Phase 2 and issued its final  report in August 2011.
In February 2011, we reported that FMCSA obligated more than $30  million for costs related to CSA from fiscal years 2007 through 2010.  FMCSA used these funds to develop the SMS and new interventions,  conduct and evaluate the pilot test, conduct travel and training related to  CSA, and develop information technology related to CSA.

	Some CSA Oversight Activities Are Functional but Others Are Indefinitely Delayed, and Implementation Problems Could Compromise the Program’s Effectiveness

Close to a year after the anticipated completion date, FMCSA has partially  implemented two of the three planned CSA oversight activities—the SMS  and an expanded set of interventions—in all states; however, it still cannot  use CSA safety ratings to (1) use CSA to assess the fitness of motor  carriers or (2) assign safety fitness determinations to individual drivers that  would prohibit them from operating trucks and buses. Although it has been  delayed, FMCSA has begun to implement the CSA oversight activities  directed at carrier safety, including SMS and carrier interventions, such as  Warning Letters and On-site Focused investigations. However, FMCSA  has yet to issue the Notice of Proposed Rulemaking (NPRM), originally  scheduled to be finalized in 2009, that would allow it to use CSA data to get  unsafe carriers off the road. At present, it appears that FMCSA will not be  issuing the rulemaking until later this year at the earliest. Furthermore, in  implementing these CSA oversight activities, FMCSA has experienced  issues that could affect CSA’s effectiveness. However, FMCSA has not  provided comprehensive information to Congress and the public on the  status of CSA as well as the risks associated with these delays and issues,  and how it plans to mitigate those risks. Moreover, FMCSA has only  recently taken steps to separately measure the fitness of drivers to operate  trucks and buses, as research has shown that drivers—not vehicle  problems—cause most carrier crashes. FMCSA has not specified time  frames for developing this component or how it will ultimately be used.

		FMCSA Has Fully or Partially Implemented Some Carrier Oversight Activities but Operational Issues Could Compromise Effectiveness

Although two of CSA’s three planned oversight activities for evaluating  carriers are at least partly implemented and functional to varying degrees,  implementation remains a work in progress. The first CSA oversight  activity—developing SMS—was implemented in December 2010, as  scheduled, and is functional (see table 3). For the second oversight  activity, seven of the nine interventions—five of which are new—are  generally functioning as intended. Two others—Off-site Investigations and  Cooperative Safety Plans—have been delayed indefinitely because the  technology needed to implement them is not yet operational. With respect  to the third planned oversight activity, suspending unfit carriers on the  basis of SMS scores, FMCSA originally intended to finalize the  rulemaking by 2009 but this effort has been delayed; FMCSA now plans  to issue the Notice of Proposed Rulemaking later this year and will not  finalize the rulemaking until 2013. According to FMCSA officials, they  delayed the rulemaking because of needed changes to SMS that arose  during the pilot. In addition, they indicated that FMCSA has a backlog of  other key rulemakings that has affected its ability to complete the CSA  rulemaking.
FMCSA fully implemented the system to measure the performance of  carriers in all safety categories in 2010. This information is provided to  carriers to help them identify and address their own safety issues.  Additionally, FMCSA has made most carriers’ safety data publicly  available since December 2010 (see fig. 1 for a sample screenshot of  carrier information available to the public). Shippers and insurers, among  others, can now use this information to make business decisions.  However, as figure 1 shows, the Crash Indicator score and the Cargo- Related BASIC score are not being made publicly available. Stakeholders  raised concerns that the Crash Indicator includes all crashes, including  those in which the driver was not accountable. FMCSA took an interim  step to make the Crash Indicator score available only to the carrier.  FMCSA plans to contract with the Department of Transportation’s John A.  Volpe National Transportation Systems Center (Volpe) to develop a  system to allow states to determine if a driver is accountable for a  particular crash. FMCSA expects Volpe to begin work on this effort in  January 2012. Specifically, FMCSA intends to allow carriers to request  changes to their violations data by providing a police accident report to  demonstrate that the carrier should not be held accountable for a  particular crash. Similarly, the motor carrier industry raised concerns  about biases created by grouping different types of carriers together for  the Cargo-Related BASIC, specifically grouping open deck carriers (flat  bed carriers) with those that use enclosed trailers. FMCSA agreed with  the industry that these biases may exist and decided not to make the  Cargo-Related BASIC data publicly available. In addition, industry raised  concerns about FMCSA’s original plans to base individual carrier crash  rates on the number of power units, i.e., trucks they operate, as opposed  to the number of vehicle miles traveled. FMCSA agreed that vehicle miles  traveled is a more equitable measure of exposure when determining  crash rates. After considering industry concerns, FMCSA modified the  measurement system to now use a combination of power units and  vehicle miles traveled to analyze crash risk. According to most trucking  association officials we interviewed, FMCSA has been willing to listen to  carriers’ concerns while implementing CSA and, according to several, has  responded by making adjustments.
Another issue that has arisen during the implementation of this part of CSA  is that state enforcement agencies, such as state police or state highway  patrol agencies, have experienced some difficulties handling motor carriers’  requests to review violations data through FMCSA’s DataQs system. In the  months before FMCSA began implementing CSA nationwide, as well as  after FMCSA began implementing CSA, carriers have been requesting  reviews of violations data at a higher rate than in the past and, in some  cases, straining states’ resources. Although carriers previously could  request reviews of violations data through the DataQs system, carriers did  not challenge the data as often because SafeStat focused on only certain  violations. Because CSA uses all violations to determine carriers’ SMS  scores and has made an expanded range of data about the motor carriers’  safety records available to them, carriers have taken a much greater  interest in these data. Specifically, in August 2010, when FMCSA first  made the violations data available for carriers’ review, the number of  requests for review was about 2,600 per month. This number increased to  a high of about 5,000 per month in October 2010, 2 months after FMCSA  made carriers’ BASICs scores available for their review. Although this  number has since decreased to about 3,700 per month by May 2011 and  decreased further to about 3,000 by August 2011, it is still higher than  when FMCSA first made violations data available for carriers’ review.  Specifically, state officials in four of the eight states we visited told us they  have experienced significant increases in the volume of these requests,  which has strained their resources. For example, in Maryland, the volume  of requests for data review has increased from 65 in August 2010 to 122 in  May 2011 before decreasing to 78 by August 2011. To deal with the  increase, the Maryland State Police added another person to handle the  requests. Similarly, in Texas, the number of requests for data review  increased from 195 requests in August 2010 to 285 by May 2011 before  decreasing to 225 by August 2011. To handle the increase, Texas officials  reassigned staff to handle the increased workload but planned to wait  before hiring someone permanently. In addition to the impact on state  resources, state officials in California said the increase in requests could  affect their ability to resolve them within 10 days, FMCSA’s goal for  responding to carriers.  Although the volume of data review requests from  carriers has been declining, it is unclear if this trend will continue as  implementation of CSA progresses.
Trucking associations have raised concerns about how states handle  these requests, as well as about states’ willingness to change violations  data. According to state law enforcement officials, states review the  requests and correct violations that are in error. Officials also indicated  that some requests reflect carriers’ efforts to have as many violations  removed from carriers’ records as possible. In January 2011 FMCSA—in  conjunction with its State Partners—developed and issued a guide to  address issues concerning consistency among states in handling  requests to review violations data.
Thus far, FMCSA has fully implemented seven of the nine interventions  nationwide. Of these seven, three are new—the Warning Letter,  Targeted Roadside Inspection, and Onsite Focused Investigation. The  Notice of Violation, Notice of Claim, Onsite Comprehensive  Investigation, and Operations Out-of-Service Order existed before CSA  and thus were already implemented nationwide. Together, as table 4  shows, these interventions provide a range of benefits. While FMCSA  previously expected to implement two other new interventions—Off-site  Investigations and Cooperative Safety Plans—nationwide by August  2011, it has delayed their implementation in the nonpilot states because it  has not yet finished developing the key technology required to manage  them. This technology, known as Sentri, is part of FMCSA’s ongoing  information technology modernization effort and is intended to provide  FMCSA enforcement and field staff easier access to carrier and driver  information and to help FMCSA and states target unsafe carriers and  drivers. FMCSA officials indicated that, although the agency’s current  legacy systems contain the information investigators need to conduct Off- site Investigations and Cooperative Safety Plans, the systems do not  interact very well. According to FMCSA, one of Sentri’s benefits is that it  will create an environment with a single interface where users can  conduct inquiries, inspections, investigations, and interventions, and  create and review reports. Additionally, Sentri will align information  technology systems with the changes to the investigative processes  resulting from the interventions. FMCSA expects to complete this  technology in April 2012. FMCSA officials indicated that the delays were  due to communication problems between information technology and  program offices—who are customers—as to the data requirements for the  system. Specifically, officials said that program offices needed to better  explain and define requirements so that everyone understands them.  According to FMCSA, its information technology office has put in place  new collaboration and communications methods with the sponsoring  program units. We have reported in the past on the importance of  establishing an agreed-upon set of requirements for customers and  stakeholders. Until FMCSA completes this technology and can fully  implement all of the interventions, it will not be able to reach the  increased number of carriers originally intended.
One issue that could influence the effectiveness of the interventions is  training. As a result of the delay in completing Sentri and the decision to  delay implementing off-site investigations and cooperative safety plans,  FMCSA revised its training plans for nonpilot states. Originally, FMCSA  planned to provide 1 week of classroom training to FMCSA division and  state officials and staff in nonpilot states, as it had done in the pilot states.  Instead, when FMCSA decided to roll out the interventions in a phased  approach, FMCSA division management received 1 day of classroom  training, while other FMCSA division and state investigators received a  series of webinars on the first phase of the roll out. Additionally, FMCSA  and state officials in pilot states are serving as mentors to assist their  counterparts in nonpilot states.
FMCSA and state officials we interviewed in nonpilot states had mixed  opinions on the training. Six FMCSA and state officials in two of the  nonpilot states we visited indicated that, because only certain  interventions were implemented and pilot states were providing  assistance, they felt the training prepared them to implement the  interventions FMCSA initially rolled out. For example, officials in one state  believed that, because the On-site Focused Investigations and  Comprehensive Investigations were similar to the compliance reviews  conducted in the past, they were comfortable with the training they have  received. However, two of the FMCSA officials and one state official in  the nonpilot states we visited felt the training lacked detail and was  insufficient because CSA was still evolving. For example, officials in one  state noted they were not yet conducting On-site Focused Investigations  because they did not feel comfortable with the training they had received  on this intervention. FMCSA officials indicated they were not aware of any  other states that were not conducting On-site Focused Investigations.  However, officials in two states said that while investigators were  conducting On-site Focused Investigations, they were concerned about  how effectively they were being conducted given limited training or  because investigators were not yet comfortable with conducting focused  reviews instead of comprehensive reviews.
FMCSA is taking steps to improve training on interventions. FMCSA  officials acknowledged that the training to date was insufficient and  explained that when they decided to begin implementing CSA in the fall of  2010, they used the webinar approach to provide information quickly to  FMCSA divisions and states. FMCSA provided 2 days of additional  training during the summer of 2011 that consisted of classroom training in  all 50 states and included both management and investigators in FMCSA  divisions and state agencies. This training includes the Safety  Management Cycle approach to interventions involving investigations  which, as noted, FMCSA believes will allow investigators to determine  why violations occur and offer recommendations for improvement.  FMCSA expects that the Safety Management Cycle will be implemented  by the end of 2011. FMCSA officials also indicated that, as they  developed this training, they incorporated the suggestions from  participant evaluations from earlier training classes and agency surveys  from both pilot and nonpilot states.
FMCSA is roughly 2 years behind its original target date for issuing and  completing the rulemaking required to use SMS to determine a carrier’s  fitness to operate. We reported in December 2007 that FMCSA planned  to publish a NPRM for the carrier safety fitness determination in summer  2008 and expected the final rule to be in place in 2009. However,  because of changes to SMS that arose during testing—such as the  change in calculating crash rates—and a backlog of rulemakings for other  FMCSA programs, officials now plan to issue the NPRM late in 2011 and  finalize the rule in 2013. However, the date FMCSA can finalize the rule  could also be delayed. FMCSA officials indicated they do not foresee any  major challenges in meeting the current schedule because they have held  public information sessions since 2008 to inform the motor carrier industry  of the methodology they are considering for the safety fitness  determination. On the other hand, others, such as the National  Transportation Safety Board and the National Private Truck Council,  noted that rulemakings could take much longer. Until the rulemaking is  completed, FMCSA will not realize one of its most important goals for  CSA—enhancing its ability to assign safety fitness determinations to a  significantly greater portion of the motor carrier industry than it currently is  able to do.

		FMCSA Has Had Mixed Success Managing CSA’s Implementation and Has Not Communicated to Congress Its Strategy to Address Identified Weaknesses and Delays

In some areas, FMCSA performed well as it implemented CSA, most  notably, in conducting extensive outreach to carriers. In December 2007,  we reported that communicating needed information to key stakeholders  would be critical to implementing a successful program. According to  trucking association representatives, FMCSA has made considerable  effort to provide information to carriers and associations and, according to  one state trucking association, has probably done as much outreach as  possible, given its resources. FMCSA’s efforts to reach out to carriers and  make them aware of the program, if continued, could help FMCSA  educate carriers about future developments in the program and forestall  problems as it completes implementing the carrier component of CSA  nationwide.  Our interviews with 55 carriers indicated that 23 had learned  about CSA from a variety of sources, including FMCSA’s and states’  outreach efforts and state trucking associations. However, 32 of the  carriers indicated that they were not familiar with CSA. Of these carriers  that had never heard of CSA, 12 were small carriers, 15 were medium  and 5 were large. While the results of our interviews are not  generalizable, they suggest that FMCSA should continue its outreach  efforts.
FMCSA has also been responsive to stakeholder concerns during CSA’s  implementation. In our December 2007 report, we said that controlling  the project by monitoring and providing feedback would be critical to  CSA’s success. Throughout the pilot and implementation, FMCSA has  made changes to CSA based on feedback from carriers and states. As  noted previously, in addition to deciding to not make the Crash Indicator  and Cargo-Related BASIC data public, FMCSA also expanded its basis  for calculating crash rates to include both power units (i.e., trucks) as well  as vehicle miles traveled after stakeholders raised concerns. After  studying the issue, FMCSA determined that including vehicle miles  traveled in addition to power units was a more accurate measure.
Although FMCSA has managed CSA implementation well in these areas,  the agency has experienced some difficulties in others.
FMCSA conducted a workforce analysis study in 2009 to determine the  staffing levels and skill sets necessary to implement CSA. Based on this  study, FMCSA planned to hire additional staff, including staff to support  the expected increase in investigations. For fiscal year 2012, FMCSA has  requested $78 million from Congress to fully implement and integrate  CSA into its operations. Of this request, $61 million is for 696 full time  positions, including salary and benefits, which represents most of  FMCSA’s existing field staff as well as 98 new full-time positions. These  new positions include 30 investigators and 51 program analysts who  would assist intervention managers and investigators throughout  FMCSA’s divisions, among other staff.
Notwithstanding the future of its funding request, FMCSA has not yet fully  determined how it would allocate staff as it moves forward to implement  CSA. FMCSA has not determined which divisions will receive the  additional investigators and program analysts, although small states will  likely share program analysts. FMCSA also has not performed a staffing  analysis to determine how it would reallocate existing staff if it does not  receive the funding in fiscal year 2012 for the new positions. We have  identified key practices for workforce planning, including developing a  process to determine staffing needs and allocate staff among offices and  taking the budgetary process into account. Given the current budgetary  environment, FMCSA officials realize they may not receive all funding  requested and plan to re-examine current staff allocations if FMCSA does  not receive authority for these positions. FMCSA officials have stated that  CSA’s effectiveness would be impacted with less funding because  investigators would not be able to conduct the same number of  interventions and, consequently, FMCSA would not be able to reach as  many carriers as originally expected. However, waiting to determine how  to allocate a lesser number of staff could also delay FMCSA efforts to  continue to implement CSA.
In addition, FMCSA is still adapting to the changes required by the new  interventions. CSA represents a shift to a new paradigm or way of  thinking about safety that requires a cultural change among FMCSA  Division and state staff, which can take time. CSA requires investigators  to change from comprehensively investigating all aspects of a motor  carrier’s operations to focusing only on weaknesses that SMS identifies  (i.e., the on-site focused investigation). During our site visits, FMCSA  division and state staff often reported that they appreciated the  efficiencies gained by using data to identify carriers and areas to focus on  during investigations. However, they also reported that this shift has been  difficult, with some investigators still preferring to conduct comprehensive  investigations. FMCSA officials noted that investigators can expand a  focused review if they see evidence of problems in other areas and that  the efficiency gains FMCSA intends will be negated if investigators  continue to take a comprehensive approach when focused reviews are  warranted. We have reported that major change initiatives and cultural  changes take time to fully implement and take effect. In our 2003 report  on the Architect of the Capitol, for example, we reported that the  experiences of successful major change management initiatives in large  private and public sector organizations suggest that they can often take at  least 5 to 7 years until they are fully implemented and the related cultures  are transformed in a sustainable manner. Additionally, we reported that  fundamental changes in the Architect of the Capitol’s culture will require a  long-term, concerted effort. The same may be true for CSA; much about  CSA is new and, given the nature of this type of cultural transformation, it  may simply take time for staff to adjust to the new paradigm. To address  this issue, FMCSA, among other things, is using the pilot states as  mentors for the states that did not participate in the pilot test, invited  participants from pilot-test states to describe the new process to their  peers in non-pilot states, and has put CSA on the agenda of annual in- service training sessions. Additionally, FMCSA plans to develop a  systematic change management plan.
As we have previously discussed, several steps and issues remain before  FMCSA can fully implement CSA carrier oversight activities. Specifically,  FMCSA has not  completed a key technology to fully implement the interventions and  provided training on interventions yet to be implemented,    developed and issued the NPRM to take action against unfit carriers  based on CSA data,    addressed staffing issues and completed efforts to help staff shift to a  new safety enforcement paradigm.
FMCSA officials acknowledged delays in implementing CSA’s carrier  oversight activities and the need to complete key tasks and address  certain issues. However, they maintain that delays are to be expected  when implementing a major program such as CSA and that, in their  opinion, FMCSA has implemented the bulk of CSA’s oversight activities.  They acknowledged that risks associated with FMCSA’s ability to  complete these items and address budgetary issues could affect their  ability to fully implement CSA, as well as CSA’s effectiveness, and noted  that they track open issues and the associated risks and mitigation  strategies. Although FMCSA officials indicated they have periodically  briefed congressional staff of their progress in developing and  implementing CSA, FMCSA has not developed any type of  comprehensive document that specifically outlines its status,  implementation delays, and other issues that need to be addressed, or  identifies the risks associated with these problems and strategies to  mitigate them.
Our past work has shown that the early identification of risks and  strategies to mitigate them can help avoid negative outcomes when  implementing large-scale projects. For example, in our 2010 report  examining the Federal Railroad Administration’s (FRA) efforts to  implement a Positive Train Control (PTC) system, we reported that  uncertainties about tasks, such as potential delays in developing PTC  components, software, and subsequent testing and implementation of  PTC systems, raise certain risks to successfully completing PTC on  time. Specifically, potential delays in developing PTC components,  software, and subsequent testing and implementation of PTC systems,  raise the risk that railroads will not meet the implementation deadline and  that the safety benefits of PTC will be delayed. We noted that FRA  officials were aware of some of these risks, but said it was too early to  know whether they were significant enough to jeopardize successful  implementation. However, we also noted that, as FRA moves forward  with monitoring railroad’s implementation of PTC, the agency will have  more information regarding the risks to completing PTC on time and  would thus be in a better position to inform Congress and other  stakeholders of the risks and mitigation strategies associated with  implementing the system. Similarly, our 2004 report examining an Amtrak  project to manage improvements to the Northeast Corridor noted that  early identification and assessment of problems would allow for prompt  intervention, increasing the likelihood that corrective action could be taken  to get the project back on track. Risk identification and management are  also essential in the case of CSA, which FMCSA developed with the goal  of significantly improving motor carrier safety. Regularly reporting  information on what steps FMCSA needs to complete in order to  implement CSA—including a timetable—as well as the risks and  mitigation strategies associated with not completing each step or  addressing each issue, would put FMCSA in a better position to respond  to problems when they occur and thus better ensure that FMCSA could  complete CSA’s implementation as planned. This would also provide  Congress and other stakeholders with important information as to  FMCSA’s status in implementing CSA and the associated risks, which  would help Congress make decisions about the program.

		FMCSA Has Made Little Progress on Driver Fitness Ratings

Although the implemented CSA oversight activities have provided FMCSA  additional tools to provide information on drivers and assess their safety  performance, FMCSA has only recently begun steps to develop the  process to separately rate the safety fitness of all drivers under CSA. Since  CSA’s initiation, FMCSA has prioritized implementation of the carrier  oversight activities. FMCSA is seeking to clarify its authority to prohibit  individual drivers, if determined to be unfit based on ratings, from operating  in interstate commerce. FMCSA officials believe that arguably the agency  currently has this authority, but acknowledge that seeking clarification from  Congress would be prudent. FMCSA is seeking this authority as part of the  next surface transportation reauthorization and has provided committees of  Congress technical legislative drafting assistance to this effect.
FMCSA officials also explained they now have access to more information  on drivers than they previously had so that implementing the driver  component is not as critical to CSA’s ability to improve safety as they  believed when designing the program. For example, the Unsafe Driver  BASIC provides additional oversight of drivers and allows FMCSA to  address unsafe driver behaviors by intervening with carriers that employ  unsafe drivers. Other systems also now allow FMCSA to evaluate drivers:    The Driver Safety Measurement System (DSMS) uses safety data  from roadside inspections and crashes to measure drivers’ safety in a  manner similar to that used under SMS and allows FMCSA and state  partners to identify unsafe, or “red flag,” drivers. The red flag driver  investigation process examines drivers receiving certain violations  during the course of motor carrier investigations. However, since  FMCSA has not implemented driver safety fitness determinations, the  agency only uses DSMS internally and for law enforcement purposes.    The Pre-Employment Screening Program (PSP) allows carriers to  view 5 years of individual drivers’ crash data from FMCSA’s MCMIS as  well as 3 years of roadside violation data from MCMIS. Although PSP  provides useful information, it was not intended to be a comparative  tool and thus does not allow carriers to determine how safe or unsafe a  driver is compared to other drivers. Also, participation in PSP is  voluntary; motor carriers must pay a subscription fee for this service.
Nonetheless, including a fitness determination would expand FMCSA’s  oversight by measuring individual driver performance and systematically  identifying unsafe commercial drivers for safety enforcement. It would  allow carriers to determine an individual driver’s safety relative to other  drivers and increase the usage of driver safety data among the motor  carrier industry. FMCSA’s 2005 study of large truck crashes found that  driver behavior is the single largest cause of crashes. FMCSA officials  indicated that they still plan to assess driver fitness as part of CSA but  have not developed a plan or set any timetable for doing so. FMCSA has  also not determined how driver safety determinations will be used or  assessed the safety risk of delayed implementation of them.

	CSA has the Potential to Improve Safety by Identifying More At- Risk Carriers, but the Program’s Full Impact Nationwide Is Unclear

CSA has the potential to identify higher-risk carriers under more precisely  defined areas of safety performance, and FMCSA has an expanded  range of interventions to follow up with them. Collectively, these changes  offer the potential to improve safety. However, not all carriers are  inspected, and larger sized motor carriers are likely to have more  inspections and thus, more likely to be ranked under SMS than smaller  sized motor carriers. Moreover, the technology FMCSA has developed to  select carriers for inspection did not allow inspectors we observed to  quickly determine if a carrier’s past history warranted an inspection.  Instead, they used it to identify what needed to be inspected once a  carrier was already selected for inspection. As a result, some states use  other technologies that incorporate FMCSA’s system, or other methods to  select carriers that may not be systematic. Furthermore, until FMCSA  completes new performance metrics, gauging the extent to which CSA  improves safety will be problematic.

		CSA Has Potential to Improve Safety through Better Use of Data and More Contact with Carriers

To improve safety, CSA makes better use of roadside inspection data in  the following ways:    SMS makes greater use of the data available from roadside  inspections than SafeStat did. Under SafeStat, only out-of-service  violations and selected moving violations were used for estimating  carriers’ scores under the Driver and Vehicle safety evaluation  areas. In SMS, any violation found is used in calculating a carrier  BASIC score. This should help FMCSA to improve overall safety by  allowing it to identify carriers with recurring types of safety violations  that may have been missed under the prior SafeStat system.    SMS allows for more precision in the measurement of safety, since,  as we discussed previously, the BASIC scores and Crash Indicator  measure carrier performance in seven areas, rather than the four  used under SafeStat. For example, CSA measures driver  performance at the motor carrier company level in several categories,  including unsafe driving, fatigued driving, driver fitness, and the use of  controlled substances and alcohol, whereas SafeStat calculated an  overall rating based on all these driver factors combined. This  breakdown not only allows for a more precise determination of motor  carrier safety performance overall but also allows FMCSA to better  identify specific areas of safety shortcomings. For example, CSA can  indicate if a carrier is having a problem with driver fatigue, whereas  SafeStat could not provide this level of detail. Thus, interventions can  be targeted to the specific area of safety concern.     SMS creates percentile ranks for carriers within each BASIC and in the  Crash Indicator, rather than producing just one total summed score, as  SafeStat did. Thus, SMS has the potential to improve safety by  reporting scores on the separate areas of safety problems and making  carriers’ performance in this area explicit. For example, CSA can  indicate that, although a carrier has a relatively poor ranking in the  Cargo-Related BASIC, the carrier has a good ranking in the Unsafe  Driving indicator, thereby enabling FMCSA to focus its interventions on  carrier practices that have the greatest impact on safety.
SMS also allows FMCSA to conduct interventions with a greater number  of motor carriers. SMS identifies about 45,000 motor carriers each month  that exceeded the thresholds in one or more BASICs or the Crash  Indicator. By comparison, under SafeStat, a similar number of about  45,000 carriers per month were identified as exceeding the threshold on  one or more safety evaluation areas to varying degrees, on a scale of A  to G. However, under SafeStat, only those carriers with a SafeStat  rating of A, B, or C were prioritized for SafeStat’s intervention—a full  compliance review—resulting in a smaller percentage of motor carriers  with an identified safety problem receiving the intervention. For example,  during all of fiscal year 2009, 16,512 compliance reviews were carried out  by FMCSA and state partners on motor carriers rated under SafeStat.  Under CSA, any carrier exceeding a threshold in even one BASIC or in  the Crash Indicator will receive an intervention of some type.
The reason FMCSA can contact carriers with a wider range of  violations—including less severe violations—than it did under SafeStat is  that CSA provides a wider range of intervention tools, some of them  requiring few resources to implement. CSA’s range of interventions—from  the resource-intensive On-site Comprehensive Investigation to the  relatively low-resource Warning Letter—provide FMCSA with more tools  for contacting carriers, calibrating the intervention to the severity of the  violation. Under CSA, all carriers newly identified as exceeding the  threshold in one or more safety areas in a given month are subject to  some type of safety intervention by FMCSA, most commonly a Warning  Letter. During the first 6 months of fiscal year 2011, FMCSA sent 19,470  Warning Letters and, along with state partners, conducted 3,190 CSA On- site Focused Investigations in addition to completing 5,684 compliance  reviews through May of 2011, for 28,344 total safety interventions.  Preliminary evidence from the pilot test suggests that even the warning  letters have an effect on safety. Twelve months after receiving only a  Warning Letter, 17 percent of test carriers exceeded at least one SMS  threshold as opposed to 45 percent of the control carriers who did not  receive Warning Letters. Reaching more carriers with enforcement  actions should enable FMCSA to improve safety.

		Not All Carriers Have Enough Inspections to Receive Safety Rankings

While the pilot test suggests that SMS has the potential to improve safety  over the prior SafeStat system, SMS’s ability to calculate BASIC scores  for carriers is dependent upon sufficient roadside inspection data for that  carrier, which are not always available for a significant segment of  carriers. Analysis of the data from the pilot test states found that a  substantial proportion of motor carriers lack sufficient data for ranking in  the six BASICs and Crash Indicator: Specifically, the Fatigued Driving and  Unsafe Driving BASIC both require a minimum of three relevant  inspections and at least one relevant violation for a motor carrier over the  past 24 months; the Vehicle Maintenance, Driver Fitness, and Cargo- Related BASICs each require a minimum of five relevant inspections and  at least one relevant violation over the preceding 24 months. Table 5  shows the percentage of carriers in the pilot test states that have  sufficient data for ranking in a BASIC or the Crash Indicator.
While most large motor carriers have enough data to be considered and  rated under SMS, the majority of smaller carriers do not. For example,  about 48 percent of carriers with 51 to 500 vehicles and about 71 percent  of carriers with 501 or more vehicles have sufficient ranking in the Unsafe  Driving BASIC but only about 1 percent of carriers with 5 or fewer  vehicles do. The majority of companies in operation are small motor  carriers with 5 or fewer vehicles; the lack of sufficient data for ranking on  a BASIC is greatest in this segment of the carrier fleet. Those carriers  with 2 or fewer roadside inspections are only potentially ranked by SMS  through the Controlled Substances/Alcohol BASIC or the Crash  Indicator. Those with 3 to 4 inspections are below the minimum data  sufficiency requirements for the Vehicle Maintenance and Driver Fitness  BASICs. This data limitation will continue to prevent the SMS from  functioning at full capability until efforts to expand roadside inspection  measurement coverage across the motor carrier fleet succeed. In the  meantime, the effect of this data sufficiency limitation is that safety  ranking by SMS is more concentrated among the large sized motor  carriers than it is among the more numerous smaller sized motor carriers.
Based on visits to inspection stations and interviews with inspection  officials in eight states, we found that not all states use methods that  systematically select trucks for roadside inspections, which can limit  CSA’s ability to improve motor carrier safety. FMCSA provides all states  with its Inspection Selection System (ISS-2010) software, designed to  systematically identify carriers with known poor safety performance.  Vehicle selection methods that factor in safety performance offer more  assurance that roadside inspections will ultimately prevent crashes by  focusing resources on higher-risk carriers. The ISS-2010 software also  systematically identifies carriers that have not been ranked in any of the  BASICs by the SMS, so that inspectors can inspect those carriers’  trucks to determine their compliance. Because of the pace at which  trucks move through the scales at inspections stations, however,  inspectors we observed rarely had time to access the ISS-2010 on  FMCSA’s website before deciding which trucks to inspect. Thus,  inspectors mainly used ISS-2010 to obtain information about trucks that  have already been selected for inspection by other means (see below).
Many states use software that allows inspectors to bypass some low-risk  trucks from the inspection station, thus allowing them to select carriers for  inspection from a group with a history of safety problems or unknown  safety performance. These third-party software products incorporate the  ISS-2010 algorithms to allow trucks belonging to carriers with good safety  performance to bypass inspection stations. When these trucks are  allowed to bypass the weigh station, inspection resources can be  expended on carriers with riskier or unknown safety performance,  according to the software. For example, in 30 states, inspectors rely on a  product called PrePass. PrePass incorporates the ISS-2010 selection  algorithms with other proprietary criteria to gauge a carrier’s safety  performance, including crash risk, before its truck enters an inspection  station. Carriers that participate in PrePass receive transponders for their  trucks; weigh stations are fitted with equipment that receives signals from  the transponders. The transponder sends a signal to the inspection  station that alerts inspectors as to whether the participating truck can  bypass the inspection station or if it must come in for an inspection. All  nonparticipating trucks must enter the station when it is open. Inspectors  in states that use such software products then employ a combination of  other methods, some noted below, to select trucks for inspection from  among those that enter the weigh station. FMCSA officials stated that  states are encouraged to use federal roadside inspection grant funds to  purchase technology to assist their inspectors in systematically selecting  trucks for inspection. FMCSA currently does not require states to use  ISS-2010 software or products like PrePass, although it encourages them  to do so.
While some of the selection methods we observed being used at  inspection stations take some aspects of crash risk into account, none are  as systematic as would be the case if inspectors were able to use the  ISS-2010 algorithms for truck selection. Some also may, by chance,  select for inspection the trucks of carriers previously unranked by CSA,  thereby broadening the base of carriers the SMS can potentially rank. We  observed the following selection methods:    Weight as an initial selection factor. All trucks entering a weigh station  will be weighed. If the overall weight or individual weight on a  particular axle exceeds the allowed weight, inspectors put the truck  and driver out of service. This method addresses safety performance  and may by chance select trucks of carriers previously unranked by  the SMS because all trucks must cross the scales. The truck may not  resume its journey until its weight issues are resolved, and the  inspector has discretion to conduct further inspection.    Obvious problems. When inspectors notice obvious problems related  to safety performance on a truck as it moves across the scale—such  as a flat tire, unattached hoses, incorrect or damaged placards, etc.— they may pull the vehicle over for inspection. Many of these problems  could involve safety performance issues and may result in selecting  trucks of carriers previously unranked by the SMS.    Random selection. Inspectors choose trucks for inspection randomly  from among those not put out of service for weight issues. This  method does not gauge crash risk or other aspects of safety  performance but could select trucks of carriers previously unranked by  the SMS.    Local discretion. These methods may focus inspectors’ efforts on  particular types of inspections, carriers, trucks, or loads for a period of  time. Local discretion selection methods can be guided by the  certification level of inspectors available at the station, the training  needs of those inspectors, or news stories about crashes of  particular types of vehicles or loads, among other things. In some  cases, inspectors may focus their efforts on factors that influence  safety, perhaps in response to public opinion about the safety  performance of particular types of vehicles or loads. These methods  could also result in selecting trucks of carriers previously unranked by  the SMS.
All of these methods are limited in identifying higher-risk trucks and  carriers. For example, a truck belonging to a carrier with a history of driver  fatigue issues would not be readily identifiable to an inspector unless a  software product employing the ISS-2010 BASIC-supported algorithm  flagged it.
No inspection selection method can assist weigh station inspectors in  selecting trucks if drivers avoid the weigh station entirely. Our  observations at state inspection stations and discussions with inspection  officials revealed that some drivers attempt to evade roadside inspection  in different ways, allowing some carriers to potentially operate entirely  beyond the scope of CSA. For example, drivers may avoid driving past a  weigh station during its regular hours of operation. Inspection facilities in  many states are open limited hours, and state officials told us there is a  significant level of truck traffic when stations are closed. Because of  physical or staffing constraints at some weigh stations, we observed that  staff may close a station periodically during its standard hours of  operation to relieve crowding or avoid back ups of trucks that could  present a safety hazard on the freeway. State police and other officials in  a number of states also indicated that budgetary constraints may force  them to reduce weigh stations’ hours of operation, decreasing the number  of trucks they can inspect and increasing the travel-time flexibility of  drivers seeking to avoid inspection. State police officials also told us that  some drivers seek to evade inspection by pulling over to the side of the  road until a station closes or by altering their routes to drive around weigh  stations, either on other highways or on smaller roads, sometimes within  sight of staff at the weigh station. Depending on resources available at  the station, troopers may or may not be able to leave the station to stop  drivers whose trucks should be inspected. According to a number of  state inspection officials, when inspectors do inspect trucks of drivers  seeking to avoid inspection, they often find serious safety violations,  reiterating the potential importance of appropriately targeting inspection  resources to road safety.

		Until FMCSA Implements New Performance Measures, Gauging Extent of CSA-related Safety Improvement Will Be Problematic

FMCSA has begun to develop performance measures to assess CSA’s  nationwide performance in improving safety, but has not yet set a  timetable for their completion. Indications that CSA may improve safety  exist. Specifically, the UMTRI evaluation of the pilot test indicates that  CSA’s SMS and new, expanded set of interventions increased FMCSA’s  ability to improve safety in the four pilot states. However, performance  measures are needed to gauge the effectiveness of CSA in improving  safety as it is implemented nationwide.
We have previously reported that agencies need to set quantifiable  outcome-based performance measures for significant agency activities,  such as CSA, to demonstrate how they intend to achieve their program  goals and measure the extent to which they have done so. Performance  measures allow an agency to track its progress in achieving intended  results, which can be particularly important in the implementation stage of  a new program. Performance measures can also help inform  management decision making, such as the need to redirect resources or  shift priorities. In some of our prior work we have recommended that  agencies develop methods to accurately evaluate and measure the  progress of implementation, and develop contingency plans if the agency  does not meet its milestones to complete tasks. In addition,  performance measures can be used by stakeholders, such as state law  enforcement partners, carrier associations, and the public who use the  nation’s highways, to hold FMCSA accountable for results. With  performance measures, FMCSA Divisions and state partners will be able  to set priorities and measure results by state or overall.
FMCSA has been working on developing performance measures for CSA  results and program implementation progress. FMCSA has proposed  several performance measures for CSA, but they have not yet been  approved within the agency. Two of the proposed measures would  assess outcomes of CSA. The first would determine the number of  carriers that received a specific CSA intervention in 1 year and then  showed improvement in the next year. The second would measure the  level of compliance from all inspections in a baseline year before CSA  was implemented (e.g., 2007) and compare that level against compliance  in subsequent years to quantify improvements in compliance across the  entire industry. FMCSA is also considering output measures, such as the  increase in the number of carriers reviewed once off-site and focused  investigations are fully implemented. According to FMCSA officials, these  proposed measures have not yet been approved by the Administrator,  and implementation will depend on accumulating relevant CSA  intervention data once the carrier oversight activities are fully deployed in  2012, as expected. Under this timeline, 2012 would become the baseline  year, which means 2013 would become the first year in which FMCSA  could begin to develop CSA performance targets such as the percentage  of carriers that showed safety improvements after being subject to CSA  interventions.
FMCSA has also begun efforts to track its progress in implementing CSA.  FMCSA has identified the specific steps it has taken to implement CSA,  as well as the states in which the various CSA oversight activities have  been implemented (i.e., pilot states vs. nonpilot states, and, for those  oversight activities that have not been implemented, when FMCSA plans  to implement them). When ultimately developed and implemented, such  measures will help provide CSA managers with information on the status  of CSA implementation and allow them to make adjustments, if  necessary, to meet established timeframes.

	Conclusions

FMCSA’s CSA program has been partly implemented and shows the  potential for improving motor carrier safety. However, key aspects of the  initiative, including using safety data from the new SMS system to take  unsafe carriers and drivers off the road and enforcing other safety  regulations, are indefinitely delayed. In the case of drivers, the plan for  when and how to determine a driver’s fitness to operate vehicles based  on the new measurement system has yet to be developed, and the safety  implications of delayed implementation of drivers’ fitness ratings for  FMCSA’s current goals to improve safety are unclear. FMCSA has also  encountered several problems during implementation, including delays in  developing technology needed for new interventions, and resistance from  staff to shift to a new paradigm of more focused and less time-consuming  reviews of carrier operations. Further, FMCSA has not established a  process for regularly reporting to Congress and the public on CSA’s  status, problems it has encountered in implementing CSA, the risks they  pose to full implementation and its strategy for mitigating these risks. This  type of information is essential to assist Congress in making decisions  about funding or authorizations for the program and assure Congress and  stakeholders that CSA is being successfully implemented. To this end,  FMCSA has made progress in developing performance measures for  determining the extent to which investigative staff are using new CSA  interventions and the safety outcomes of these interventions. However,  until these measures are completed and are being implemented, the  extent of CSA’s effectiveness in improving safety will remain unclear to  FMCSA management, Congress, and the public.

	Recommendations for Executive Action

We recommend that the Secretary of Transportation direct the FMCSA  Administrator to take the following two actions    develop a plan for implementing driver fitness ratings that prioritizes  steps that need to be completed and includes a reasonable timeframe  for completing them. The plan should also address the safety  implications of delayed implementation of driver fitness ratings.  regularly report to Congress on CSA’s status; the problems that  FMCSA has encountered during the implementation of CSA and the  risks they pose to full implementation of CSA; its strategy for  mitigating these risks; and a timetable for fully implementing CSA and  reporting the progress made in developing and implementing CSA  performance measures.

	Agency Comments

We provided a draft of this report to the Department of Transportation for  review and comment. The Department did not agree or disagree with our  recommendations but said it would consider them. The Department  provided technical comments and clarifications, which we incorporated as  appropriate. At a meeting on September 23, 2011, to discuss the  Department’s comments, FMCSA officials confirmed that they intend to  continue implementing the driver fitness ratings. Previously, FMCSA  officials indicated that they were considering implementing these ratings  but had made no final decision. In response to this new information, we  modified the language of our recommendation regarding driver fitness  ratings. Our recommendation originally focused on having FMCSA  determine the safety implications of not fully and expeditiously  implementing the driver fitness ratings and, if it determined that full  implementation was necessary, to then develop an implementation plan.   To reflect that FMCSA has decided to proceed with implementing the  driver fitness ratings, we modified our recommendation to focus instead  on an implementation plan.
We are sending copies of this report to the appropriate congressional  committees; the Secretary of Transportation, the Administrator, Federal  Motor Carrier Safety Administration; and the Director, Office of  Management and Budget. In addition, the report will be available at no  charge on the GAO Web site at http://www.gao.gov.
If you have any questions regarding this report, please contact me at  (202) 512-2834 or at flemings@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix II.

Appendix I: Objectives, Scope, and Methodology

GAO was directed by a 2009 Senate Committee Report, adopted by the  conference committee, to conduct a study as part of the continued  monitoring of the Compliance, Safety, and Accountability program (CSA)  implementation. Specifically, this report addresses (1) the status of the  CSA rollout and what issues, if any, could affect the full and effective  implementation of the program and (2) CSA’s potential to improve safety.
To determine the status of the CSA rollout and challenges that could affect  the full implementation of the program, we analyzed Federal Motor Carrier  Safety Administration (FMCSA) documentation, including information on  FMCSA’s website (www.fmcsa.dot.gov), periodic outreach e-mails from CSA  program officials, and CSA training materials. Additionally, we reviewed  congressional testimony provided by FMCSA’s Administrator. We also  reviewed an evaluation of the pilot test conducted by the University of  Michigan’s Transportation Research Institute (UMTRI). We interviewed  FMCSA and National Transportation Safety Board headquarters’ officials as  well as national representatives of carrier industry associations (see table 6)
We also attended the Commercial Vehicle Safety Alliance (CVSA) annual  conference in September 2010 and interviewed representatives of several  State Partners to discuss CSA implementation, as indicated in table 7.  We also attended two FMCSA-sponsored outreach sessions discussing  different aspects of CSA and carrier motor vehicle safety, one from the  carrier’s perspective.
Additionally, we visited eight states (four that participated in the pilot  program: Georgia, Maryland, Minnesota, and Missouri, and four that did  not: California, Mississippi, Texas, and Utah) to interview FMCSA Division  and State Partner officials as well as industry groups and some carriers.  (See table 8 for criteria we used to select these states.) We collected and  reviewed other CSA implementation and background documentation  during these visits. (See table 9 for agency and industry organizations we  interviewed during state visits.)
To obtain information on motor carriers’ knowledge of and experiences  with CSA, we selected a nongeneralizable random sample of motor  carriers from the Motor Carrier Management Information System  (MCMIS) carrier census file and conducted brief, structured telephone  interviews. We screened the population from which we selected the  sample to remove foreign carriers and those carriers that had not updated  their census (MCS-150) forms with FMCSA in the prior 2 years. We  divided the carriers into three size categories—small, medium, and  large—based on the number of vehicles associated with the company  and randomly selected a group of carriers within each size category to  participate in the structured interviews. During the interviews, we asked  about the interviewees’ knowledge and understanding of CSA;  interviewees were owners, safety managers, or others who would have  knowledge of a carrier’s safety practices and performance. We obtained  responses from 55 motor carriers out of the 270 we attempted to contact.
To determine CSA’s potential to improve safety, we analyzed FMCSA  documents describing the design and function of the Safety Measurement  System (SMS), how the severity of violations were weighted, and other  design documentation as it was released, particularly comparing the SMS  with SafeStat. We also analyzed UMTRI’s pilot test study findings. We  reviewed UMTRI’s statistical methodology and its reliability assessment of  the FMCSA data used for the study and determined that the results of  UMTRI’s pilot evaluation study were sufficiently reliable for our purposes.  We obtained a copy of the May 2011 MCMIS inspection data, upon which  five publicly available Behavior Analysis and Safety Improvement  Categories (BASIC) scores for carriers were based, and analyzed it to  determine the extent to which motor carriers lacked a sufficient number of  roadside inspections for measurement under the BASICs in SMS. We  electronically tested the data for completeness and coding accuracy, and  found it sufficiently reliable for the purposes of our engagement.
We also analyzed the function of FMCSA’s Inspection Selection System  software, which is designed to select trucks for inspection and thereby  guides data collection for the SMS. We did not model the SMS in order to  test its function ourselves, as it was modified several times during the  course of our review. During our state visits, we also visited weigh  stations or other truck inspection sites to interview inspectors about how  they select trucks for inspection and how CSA has affected their work,  and the data they obtain during inspections. We observed truck  inspections during these visits. We also obtained information on crash  data quality by analyzing studies UMTRI conducted on states’ MCMIS  crash data reliability as well as FMCSA’s publicly available crash data  evaluation tools.
We conducted this performance audit from June 2010 to September  2011, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Appendix II: GAO Contact and Staff Acknowledgments

	GAO Contact

Susan Fleming (202 512-2834)

	Staff Acknowledgments

In addition to the individual named above, Ed Laughlin, Assistant Director;  Lynn Filla-Clark, Analyst-in-Charge; Carl Barden; Lauren Calhoun; Alison  Hoenk; Delwen Jones; Elke Kolodinski; Kirsten Lauber, Sara Ann  Moessbauer; Rebecca Rygg, Amy Rosewarne; and Larry Thomas made  key contributions to this report.

Summary:
Why GAO Did This Study

Over 3,600 people in this country died in 2009 as a result of crashes involving large commercial trucks and buses. Until recently the Federal Motor Carrier Safety Administration (FMCSA) and its state partners tracked the safety of motor carriers--companies that own these vehicles--by conducting resource-intensive compliance reviews of a small percentage of carriers. In 2004, FMCSA began its Compliance, Safety, and Accountability (CSA) program. CSA is intended to identify and evaluate carriers and drivers posing high safety risks. FMCSA has focused on three key CSA oversight activities to evaluate carriers: a new Safety Measurement System (SMS) using more roadside inspection and other data to identify at-risk carriers; a wider range of "interventions" to reach more at-risk carriers; and using SMS data to suspend unfit carriers. FMCSA expected to fully implement CSA by late 2010. FMCSA also plans to separately use data to rate drivers' fitness. In this report, GAO assessed: (1) the status of the CSA rollout and issues that could affect it and (2) CSA's potential to improve safety. GAO reviewed CSA plans and data, visited eight states, and interviewed FMCSA, state, and industry officials.

What GAO Found

Close to a year after the anticipated completion date, FMCSA has partially implemented two of the three planned CSA carrier oversight activities--the new SMS and an expanded set of interventions--in all states; however, it still cannot use CSA safety ratings to get unsafe carriers off the road because it has not completed a rulemaking needed to do so. Specifically, (1) FMCSA implemented SMS in 2010, as scheduled, to replace the prior system, known as SafeStat. The system allows FMCSA to evaluate, score and rank the safety of carriers and identify at-risk carriers needing intervention. However, states have had to expend resources to respond to carriers that have requested reviews of inspection violations shown in the system. (2) FMCSA has implemented most of the expanded array of enforcement interventions for at-risk carriers, including issuing warning letters and initiating focused reviews of carriers' safety operations that allow FMCSA to reach more at-risk carriers; however, it has delayed implementation of two interventions--Off-site Investigations and Cooperative Safety Plans--because the technology needed to implement them will not be completed until at least 2012. (3) FMCSA has not yet begun using SMS data to suspend unfit carriers, and is 2 years behind in issuing and completing the rulemaking needed to use these data instead of a time-consuming compliance review. FMCSA expects to finalize the rulemaking in 2013. In addition, FMCSA has had mixed success managing implementation of CSA oversight activities thus far. FMCSA performed well in conducting outreach to carriers and responding to stakeholder concerns, but experienced difficulties in realigning its workforce for CSA and adapting staff to CSA's new safety paradigm. FMCSA has not provided comprehensive information to Congress and the public on the risks associated with either the delayed carrier intervention activities or operational and management issues that arose during implementation and its plans to mitigate these risks; thus Congress may lack information needed to make decisions about CSA. Moreover, FMCSA has taken initial steps to separately measure drivers' fitness to operate trucks and buses by seeking new legislative authority to prohibit unsafe drivers from operating in interstate commerce. However, FMCSA has not specified time frames for developing this measurement, how it will ultimately be used, or whether delaying the implementation will affect safety. It is too early to definitively assess the extent to which CSA will improve truck and bus safety nationwide. Data from a pilot test suggest that SMS and the expanded range of intervention tools provides a more effective means of contacting these carriers and addressing their safety issues. However, CSA's success depends on the availability of sufficient inspection data for carriers. For example, small carriers are less likely to receive enough roadside inspections to be scored and ranked in SMS. FMCSA has begun but not finished performance measures for CSA and has not yet collected the data needed to use them, so the extent that it can show CSA improves safety is unclear.

What GAO Recommends

GAO recommends that FMCSA (1) develop a plan to implement driver fitness ratings in a reasonable timeframe and (2) regularly report to Congress on problems and delays in implementing CSA and plans to mitigate risks. FMCSA provided technical comments and agreed to consider the recommendations.