Federal Motor Carrier Safety Administration: Education and Outreach Programs Target Safety and Consumer Issues, but Gaps in Planning and Evaluation Remain
Published: 2005-12-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Federal Motor Carrier Safety Administration (FMCSA) is responsible for improving commercial vehicle safety and uses education and outreach as part of its efforts. The House report accompanying the fiscal year 2005 Department of Transportation (DOT) appropriations bill asked GAO to report on FMCSA's education and outreach programs to the House and Senate Committees on Appropriations. GAO (1) describes FMCSA's education and outreach programs and how they relate to FMCSA's goals (2) identifies the extent to which FMCSA has evaluated its education and outreach programs and (3) describes the extent to which FMCSA's education and outreach programs are effective.
section_title: What GAO Found
paragraphs: FMCSA's education and outreach programs--New Entrant, Non-Entrant, Motor Coach, Safety Belt, and Household Goods' target different audiences, including the motor carrier industry, commercial vehicle drivers, and the public.Total funding for these programs in fiscal year 2005 was $36.3 million; the largest share (about $33 million) went to the New Entrant program, which is designed to inform newly registered motor carriers (new entrants) about motor carrier safety standards and regulations to help them gain compliance with FMCSA requirements. FMCSA uses many approaches, such as direct contact with carriers, media campaigns, distributing printed materials, and establishing Web sites to provide information to target audiences. FMCSA has not described how its education and outreach program activities link expected changes in attitudes and behavior to broader goals, such as DOT's strategic objective of reducing transportation-related fatalities. FMCSA officials state that the education and outreach activities and programs link to agency goals at a high level, but this was not evident from our review, with the exception of the Safety Belt program. FMCSA has used a logic model as a tool in other programs to show the relationship between program activities and broader goals. FMCSA has begun some evaluations of its education and outreach programs, and plans other evaluations of these programs. However, although FMCSA's New Entrant program has existed for over 2 years, FMCSA has no plans to evaluate its New Entrant program until 2008. Thus FMCSA has no information on whether information on its safety requirements, provided through the Education and Technical Assistance package or during New Entrant safety audits--targeted toward truckers newly entering the industry--effectively communicate information to new entrants. This lack of evaluation makes it difficult to determine the impact the education portion of the New Entrant program has on commercial motor vehicle safety. Since FMCSA currently has little information on how its programs have affected attitudes and behavior, it is difficult to determine the effectiveness of FMCSA's effort. However, the designs of two programs appear to follow theories and research regarding media campaigns, which are intended to influence decision making about safety. Research and behavior theory suggest that for some types of programs--such as DOT's Click It or Ticket program, which is designed to increase safety-belt use by passenger car drivers--enforcement linked to education can improve results, and FMCSA has indicated it is linking some education and outreach programs to enforcement efforts, where appropriate. Finally, motor carrier association officials whom we spoke with stated that, in their view, FMCSA is doing some positive things in its education and outreach activities. A public safety group stated that FMCSA followed reasonable approaches in starting its education and outreach efforts; however, they would like to see more information on program effectiveness to help FMCSA refine the programs.

section_title: Letter
paragraphs: In 2003, large trucks represented 3 percent of registered vehicles in the  country, but nearly 12 percent of the people killed in motor vehicle  accidents died in crashes involving large trucks. The Federal Motor Carrier  Safety Administration (FMCSA), within the Department of Transportation  (DOT), is responsible for improving the safety of commercial vehicle  operations—which includes interstate truck and motor coach (bus)  companies—and has set a safety goal to reduce fatalities from an estimated  2.81 per 100 million truck vehicle miles traveled in 1996 to no more than  1.65 per 100 million truck vehicle miles traveled by the end of 2008. Among  the ways that FMCSA attempts to achieve this and other goals is through  education and outreach activities. For example, the agency provides  information to consumers about motor coach safety records and what to  consider when hiring a moving company. It also provides new motor  carriers with information on safety requirements and conducts safety  audits to review motor carrier compliance with these requirements. In  some cases, education and outreach efforts regarding safety audits may be  reinforced when followed-up with enforcement.
In recent years, we have raised concerns about FMCSA education and  outreach efforts. In 2003, for example, we reported that FMCSA’s “Share  the Road Safely” program lacked a clear program strategy and included  activities that were only tenuously linked to program goals, and we  reported that FMCSA had not recently evaluated the program’s  effectiveness. In addition, our 2001 report on oversight of the household  goods moving industry noted that DOT and FMCSA had made limited  efforts to provide consumer education that would enable the public to be  more informed about hiring a mover. Furthermore, the Congress has  expressed concern about how FMCSA’s education and outreach programs  are meeting broader goals. In the conference report accompanying the  DOT appropriations bill for fiscal year 2005, FMCSA was asked to report by  April 2005 to the House and Senate Committees on Appropriations on  strategies linking outreach and education program initiatives to each goal.
The House report accompanying the DOT appropriations bill for fiscal year  2005 asked us to monitor and evaluate FMCSA’s education and outreach  programs and to report on the status of these programs to the House and  Senate Committees on Appropriations. We (1) describe the scope and  nature of FMCSA’s education and outreach programs and how they relate  to FMCSA’s goals, (2) identify the extent to which FMCSA has evaluated its  education and outreach programs, and (3) describe the extent to which  FMCSA’s education and outreach programs are effective. As discussed with  your staff, we focused on five specific education and outreach programs:  (1) the New Entrant; (2) the Non-Entrant; (3) Commercial Safety-Belt Use;  (4) Motor Coach Selection Outreach; and (5) Household Goods Outreach  programs and did not review the “Share the Road Safely” program, which  Congress transferred from FMCSA to the National Highway Traffic Safety  Administration (NHTSA) in fiscal year 2004, with FMCSA retaining a  supporting role. The recent highway reauthorization legislation authorizes  funding for this program to both FMCSA and NHTSA and asks us to review  the “Share the Road Safely” program by June 2006.
To describe the scope and nature of FMCSA’s education and outreach  programs and FMCSA’s intended results for these programs, we discussed  these programs with FMCSA officials at agency headquarters and two field  locations, and we reviewed program materials and documentation as well  as observed a safety audit for the new entrant program. We also analyzed  both the extent to which FMCSA describes how its programs lead to  agency goals and incorporates its education and outreach programs into its  strategic and program planning and performance budgeting. To identify  approaches to describing how agency programs contribute to agency goals,  we reviewed our previous work on government education programs that  showed how a program logic model—a model that links activities to goals  by analyzing program inputs, outputs, and outcomes—illustrates how  education and outreach activities can influence attitudes and behavior, and  ultimately contribute to agency goals. To identify what the agency was  doing to evaluate these programs, we discussed evaluations and evaluation  plans with FMCSA officials at agency headquarters and two field locations.  We also reviewed evaluation materials, including proposals and statements  of work for planned contracts to carry out programs and program  evaluations. Finally, to identify what is known about the effectiveness of  the programs, we conducted a literature review of media based campaigns  and reviewed the structure and evaluation of these campaigns related to  improving safety as well as behavioral theories used in designing the  campaigns. We selected two behavioral theories that were directly  applicable to the programs under this review, which are called the theory of  reasoned action and the general deterrence theory. We also discussed  what is currently known about the effectiveness of these programs with  FMCSA officials and with representatives of associations that serve the  trucking, motor coach, and household goods moving industries, as well as a  law enforcement association and public interest groups that are involved in  motor carrier safety. Our work was conducted in accordance with generally  accepted government auditing standards. Appendix I provides the details  of our scope and methodology.
subsections: section_title: Background
paragraphs: FMCSA, established as a separate administration under DOT by the Motor  Carrier Safety Improvement Act of 1999 is responsible for improving the  safety of commercial vehicle operations on the nation’s highways. FMCSA  is engaged in several programs and activities to carry out its mission,  including developing and enforcing Federal Motor Carrier Safety  Regulations, administering Motor Carrier Safety Assistance Program grants  to states, regulating interstate household goods movers, and performing  education and outreach. FMCSA operates through its headquarters in  Washington, D.C.; four regional service centers; and division offices located  in all 50 states, the District of Columbia, and Puerto Rico.
FMCSA undertakes education or outreach while carrying out many of its  functions—for example, in posting regulations pertaining to commercial  drivers’ licensing or transporting hazardous materials on its Web site, or in  providing on-line access to motor carrier crash statistics. FMCSA is also  allocated funding for specific education and outreach programs. In fiscal  year 2005, FMCSA used education and outreach funding for activities  supporting its Safety Belt program targeted to commercial vehicle drivers,  outreach to the public on hiring motor coach services, and outreach to  consumers using household goods movers. Congress also allocated fiscal  year 2005 funding for FMCSA’s program directed at recently registered new  entrant motor carriers and funding to establish an initiative directed to  non-entrants—which are carriers in interstate operations that have not  registered with DOT.
Under the Government Performance and Results Act of 1993 (GPRA),  federal programs should be designed with measurable goals that support  the agency’s overall strategic goals. Congress enacted GPRA to shift  agencies’ focus from simply monitoring activities undertaken to measuring  the results of these activities. Under GPRA, agencies develop multiyear  strategic plans, which are the starting point for their performance  measurement. Each strategic plan is to include a mission statement, a set of  outcome-related strategic goals, and a description of how the agency  intends to achieve these goals. To measure progress toward the strategic  goals, we have previously reported that the agency should also have a plan  for collecting data to measure and evaluate program performance.  Without measurable goals and evaluation, it is difficult to determine  whether the program is accomplishing its intended purpose and whether  the resources dedicated to the program efforts should be increased, used in  other ways, or applied elsewhere.
Research conducted on the effectiveness of media campaigns indicates  that the development of an education and outreach program should include  identifying how the program is expected to change the target audience’s  attitudes and behaviors. There are several theories and models on affecting  behavior, and two in particular are the most relevant to the education and  outreach programs under this review. This is because the theories are  consistent with the structure and goals of the FMCSA programs we  reviewed. One behavioral model that researchers have developed—the  reasoned action theory—assumes that people will take a desired action if  they believe the action is beneficial, and they have the means to undertake  it. Another model that experts have developed—the general deterrence  theory—indicates that in some circumstances people will take a desired  action when they recognize that they may be punished for failing to act.  The relative roles of education and enforcement depend upon the  program’s target audience, the desired change to be achieved, and other  factors.
Since media campaigns should identify how the program is expected to  change target audiences’ attitudes and behaviors, program managers can  use a tool or framework to show how outreach activities are intended to  influence attitudes and behaviors, and thus achieve broad program goals.  Academic literature on program development and evaluation provides a  number of models. One tool we have previously used to review education and outreach programs is called a logic model. A logic model links  program inputs and outputs to program outcomes (see fig. 3). Agencies  have used such a model to plan, evaluate, and adjust program activities. A  logic model can facilitate planning and evaluation by (1) demonstrating  accountability through focusing on measurable outcomes; (2) linking  activities to results to prevent mismatches between program activities and  outcomes; and (3) integrating planning, implementation, evaluation, and  reporting. In addition, a program logic model can help describe a program’s  components and desired results and explain the strategy—or logic—by  which the program is expected to achieve its goals. The logic model can  help clarify the links between program components, focusing on outcomes  that are measured and under some degree of control. In some cases, it may  be difficult to show a direct link—a specific cause and effect—between  program activities and outcomes due to the influence of external factors.  For example, the Safety Belt program, although targeted toward  commercial motor vehicles operators, may have outcomes influenced by  the NHTSA’s “Click It or Ticket” program targeted to passenger car drivers.  Commercial motor vehicle operators could be affected by this program, or  other safety-belt messages, and decide to wear safety belts when driving  commercially. Thus, the Safety Belt program’s initiatives may not have been  the primary factor in determining a particular driver’s safety-belt use.
Finally, once program activities are established and linked to program  goals, organizations can use program evaluation to determine whether or  not a program is meeting its goals. Program evaluations are conducted  periodically to provide an overall assessment of how well a program is  achieving its expected results. We have reported on the value of program  evaluations to assess the impact of a particular program. This work  showed that results of evaluations can support decisions on resource  allocation and ways to improve program effectiveness.
section_title: FMCSA Has Established Several Education and Outreach Efforts but Needs Specific Links to Goals
paragraphs: FMCSA has established several education and outreach programs, with  different purposes targeted to different audiences. Total funding for these  programs in fiscal year 2005 was $36.3 million, with the largest  share—$33.1 million—allotted to the New Entrant program. FMCSA’s  education and outreach efforts are intended to address recognized  problems by communicating information to motor carriers, commercial  drivers, and the public to encourage safer practices and better decision  making. FMCSA activities under these programs range from distributing  brochures and posting information on Web sites, to site visits to newly  registered motor carriers under its New Entrant program. Although FMCSA  officials consider these programs to be linked to agency goals at a high  level, this linkage to broader agency goals is not explicit in the agency’s  planning and budgeting documents. FMCSA officials state that they have  used a logic model in their performance budgets to illustrate the link  between education and outreach and agency goals at a high level. Although  the performance budget implies a link between the education and outreach  program and agency goals, it does not describe how the activities are  intended to support these goals. As a result, it is difficult for stakeholders  to see how education and outreach activities that seek changes in attitudes  and behavior will ultimately contribute to agency goals. Also, without a  clear link, program managers can not easily determine if program activities  are appropriately targeted or if they need to refine their programs to meet  agency goals. In contrast, we found, however, that FMCSA uses a logic  model in the strategic plan for its Research and Technology group to  demonstrate how its programs and activities support agency goals. As  FMCSA continues its efforts to develop the links between its education and  outreach activities and broader goals, it might consider using such a logic  model to make those relationships clear.
subsections: section_title: Education and Outreach Programs Address a Range of Safety and Consumer Protection Concerns and Are Targeted to Industry and the Public
paragraphs: We reviewed five education and outreach programs that FMCSA has  initiated in response to either congressional direction or concerns about  motor carrier operations. FMCSA’s education and outreach programs target  a variety of audiences, including the motor carrier industry, commercial  vehicle drivers, and the public; and they involve a variety of approaches,  such as direct contact with carriers, media campaigns, distributing printed  materials, and establishing Web sites. FMCSA also works with industry and  law enforcement associations, other agencies, and safety-oriented  organizations in carrying out some of these education and outreach efforts.  Four of these programs are about motor carrier, driver, or passenger safety,  while one has a consumer information focus. Table 1 provides a brief  overview of the target audience, general purpose of the program, and  program activities.
The five programs in this review are summarized below, and discussed in  greater detail in appendix II.
subsections: section_title: New Entrant Program
paragraphs: FMCSA intends the New Entrant program to inform newly registered motor  carriers (new entrants) about motor carrier safety standards and  regulations to help them gain compliance with FMCSA requirements and  improve truck safety and thus reduce crash rates. Studies have shown  that new carriers have higher safety violation and crash rates than more  experienced carriers. FMCSA uses its Education and Technical Assistance  Package to communicate safety requirements and other information to new  entrants when they register with DOT. This information package is also available at FMCSA’s Web site, along with carrier registration forms. The  primary activity of the New Entrant program is safety audits conducted  through site visits to the new carriers. Safety audits are conducted by  FMCSA, state law enforcement, or contractor personnel, who review the  carrier’s compliance with FMCSA requirements, including driver  qualifications, driver records of duty status, vehicle maintenance records,  and participation in a controlled substance and alcohol use testing  program. As of April 2005, FMCSA had conducted 52,000 safety audits since  the New Entrant program began in 2003.
section_title: Non-Entrant Program
paragraphs: FMCSA is developing a Non-Entrant program intended to inform start-up  motor carriers who have not registered with the agency—called  non-entrants—of the requirement to register with FMCSA. FMCSA officials  are concerned that carriers that have not registered with FMCSA may  represent an increased safety risk. FMCSA expects to target motor carriers  who have not registered as interstate carriers with FMCSA, by working  through truck sales and leasing firms to inform start-up motor carriers  about registration requirements and, ultimately, to ensure that the carriers  register as new entrants. FMCSA officials expect to award a contract to  develop the Non-Entrant program before the end of 2005.
section_title: Commercial Motor Vehicle Safety Belt Program
paragraphs: The Commercial Motor Vehicle Safety Belt program is intended to increase  safety-belt use by commercial truck drivers. FMCSA began this effort after  a study showed that only 48 percent of commercial truck drivers used  safety belts, compared with nearly 80 percent safety-belt use by passenger  car drivers. Working through the Commercial Motor Vehicle Safety Belt  Partnership—an organization of government, industry, law enforcement,  and safety associations—FMCSA facilitates and coordinates the  development and distribution of outreach materials, including posters,  bumper stickers, and brochures promoting safety-belt use. (See fig. 1 for an  example of a brochure describing the safety-belt partnership.) Working  through the Partnership allows FMCSA to leverage its resources and tap  into organizations that have their own communication links to the trucking  industry. FMCSA staff distribute materials through trucking industry shows  and events, and state law enforcement personnel distribute materials  during safety enforcement campaigns. FMCSA has prepared over 200,000  brochures and 50,000 bumper stickers for the 2005 “Be Ready, Be Buckled”  campaign. FMCSA also provides safety belt information through its Web  site. to help them select and hire motor coach services. The program began  following an increase in motor coach accident fatalities. Currently,  FMCSA’s primary effort is maintaining a Web site that allows individuals to  access advice on selecting a motor coach company and safety information  on specific motor coach companies. For the “Moving Kids Safely”  campaign in 2002 that was part of the Motor Coach Outreach program,  FMCSA created 49,000 brochures; 1,200 posters; and 1,200 “toolkits” to  provide advice to school districts on hiring school bus services. FMCSA  also reaches out to the industry to inform motor coach companies of  regulations and provide safety advice through printed materials and the  agency’s motor coach specialists.
section_title: Household Goods Outreach Program
paragraphs: The Household Goods Outreach program is intended to help consumers  make more informed choices in selecting an interstate moving company to  avoid unscrupulous movers. This program was begun in response to  congressional concern over increasing consumer complaints about  fraudulent moving companies. FMCSA distributes printed material through  moving companies and other means, such as its Web site and the General  Services Administration (GSA) consumer information catalog, to inform  consumers of their rights when dealing with moving companies. The  material that FMCSA has developed includes a new outreach theme:  “Protect Your Memories, Your Money, Your Move” and it includes  brochures and presentations that can be delivered by FMCSA staff. For  example, as of October 2005, FMCSA officials reported that they had  printed a total of 50,000 copies of the two brochures the agency distributes.  FMCSA also maintains a hotline to receive complaints from individuals  about problems with movers and enters them into a database. Although  FMCSA does not get involved in resolving complaints, the agency uses the  database to target firms for investigation. Also, we previously reported that  FMCSA should make the complaint information available to consumers.  According to FMCSA officials the agency has not yet done this because of  privacy and other legal issues.
section_title: Except for the New Entrant Program, Education and Outreach Funding Represents a Small Portion of FMCSA’s Budget
paragraphs: The five education and outreach programs represent a small portion of  FMCSA’s total funding. Fiscal year 2005 funding for the five programs  totaled $36.3 million, out of approximately $443.3 million to fund FMCSA  operations and grant programs. The New Entrant program accounted for  $33.1 million of the funding for the five programs, while funding for the  other four programs combined was $3.2 million—less than 1 percent of the  agency’s total fiscal year 2005 funding.
The first chart in figure 2 below depicts the New Entrant program, and the  combined funding for the other four programs, as their relative share of  total FMCSA funding in fiscal year 2005. The second chart depicts the  relative size of the four smaller education and outreach programs.
Other four programs ($3.2M)
Total New Entrant ($33.1M)
Motor coach ($150,000)
Safety Belt ($500,000)
Remaining FMCSA ($407M)
Non-Entrant ($1.05M)
Household Goods ($1.5M)
The funding available for education and outreach supports a diverse range  of activities, from the costs of state personnel conducting safety audits, to  the costs of printing and distributing brochures and other materials, and  the costs of contracts being awarded to develop and evaluate education  and outreach activities. The principal activities that FMCSA is undertaking  through the five programs are shown in table 2.
section_title: FMCSA Has Not Clearly Articulated How Its Education and Outreach Initiatives Will Support Broader Goals
paragraphs: Congress has expressed concern about how FMCSA’s education and  outreach activities are meeting broader goals, such as individual program  goals, or DOT’s strategic goals. The conference report accompanying the  DOT appropriations bill for fiscal year 2005, for example, asked that  FMCSA report by April 2005 to the House and Senate Committees on  Appropriations on strategies linking outreach and education program  initiatives to each goal. FMCSA sent its report on October 11, 2005. The  report indicates that FMCSA believes education and outreach programs  support the overall DOT safety strategic objective of “enhancing public  health and safety by elimination of transportation deaths and injuries.”  FMCSA also indicates that every education activity undertaken “is vital to  FMCSA’s goal of reducing fatalities, injuries, and crashes, and attaining a  large truck crash rate of no more than 1.65 fatalities per 100 million miles of  truck travel by the end of 2008.”
Although the report was intended to address congressional concerns about  linking education and outreach to program goals, it is unclear specifically  how this will be accomplished, since the report does not describe how  these programs are intended to influence knowledge, attitudes, and  behaviors and thus support FMCSA’s broader goals. Although the report  describes in detail the activities FMCSA is undertaking for each of these  programs, it does not describe how these activities are intended to change  attitudes and behaviors that will ultimately meet FMCSA’s goals.
In addition, although FMCSA’s fiscal year 2006 performance budget has  some discussion of how activities support FMCSA and DOT safety and  productivity goals, except for the Commercial Motor Vehicle Safety Belt  program, none of the budget or program documentation we reviewed for  FMCSA’s education and outreach programs provides detailed descriptions  about how activities link to goals. A greater level of detail in the description  of how education and outreach program activities link to goals would help  the public and other interested parties to see how program activities that  seek changes in attitudes and behavior will ultimately contribute to agency  goals. Furthermore, without this link, program managers cannot easily  determine if program activities are appropriately targeted or if they need to  refine their programs to meet agency goals.
FMCSA officials stated that information about education and outreach  programs is reported in DOT’s strategic plan and in FMCSA’s performance  budget documents. However, DOT’s strategic plan does not specifically  address FMCSA’s education and outreach programs and refers readers to  the performance budgets for more detailed, program-specific information.  FMCSA’s performance budget documents identify strategic objectives, such  as saving lives and reducing injuries by preventing bus and truck crashes,  that link and contribute to objectives in DOT’s strategic plan, such as  reducing transportation-related fatalities. It also identifies underlying  performance goals and indicators for each objective. For example,  FMCSA’s strategic objective for safety is illustrated in table 3.
The performance budget documents address education and outreach  primarily under this strategic objective for safety, by stating that “educating  carriers about the benefits of operating safely, and in compliance with  safety regulations, is advantageous to both the carrier and enforcement  community.” The budget also states that the education programs are  designed to change “the knowledge, attitudes, and behaviors of commercial  motor carriers, commercial motor vehicle drivers, and passenger vehicle  drivers, driving in the vicinity of large trucks.”
However, the performance budget does not provide specific information  for each program on how these expected attitude and behavior changes are  linked to broader goals. Of the five programs included in our review, the  description of the safety belt initiative provides the clearest information  articulating how the program is intended to affect the knowledge, attitudes,  and behaviors of its targeted audience, and consequently to improve safety.  FMCSA describes how education will highlight the risks of not wearing a  safety belt, which in turn is intended to improve drivers’ attitudes toward  wearing safety belts, and subsequently meet the program goal of increasing  safety-belt usage 10 percent by 2009 and further, meet the strategic  objective of reducing fatalities. The performance budget also indicates  that education will be combined with traffic enforcement of safety-belt  requirements to achieve this outcome. In contrast, the performance budget  discusses the New Entrant program in terms of staffing, funding, and the  number of safety audits to be done in fiscal year 2006, without defining a  goal for what the New Entrant program will accomplish and how that  accomplishment would contribute to the strategic objective of reducing  fatalities.
In addition, the links between the Motor Coach and Household Goods  program activities and broader strategic goals are difficult to discern.  Specifically, the performance budget document explains that the Motor  Coach program seeks to increase consumers’ knowledge about the safety  records of carriers, with a potential side benefit of encouraging motor  carriers to maintain good safety records. FMCSA does not, however,  describe how affecting consumers’ choice of a particular motor coach  company would support the overall strategic objective of reducing  transportation-related fatalities. FMCSA’s performance budget relates its  Household Goods program to DOT’s mobility and FMCSA’s productivity  objectives. The mobility objective is “to advance accessible, efficient, and intermodal transportation for the movement of people and goods.” The  productivity strategic objective is “to promote efficient and economical  motor carrier operations to sustain mobility and economic growth.” The  performance budget document asserts links between these objectives and  the education and outreach program, but it does not explain how specific  activities—such as establishing its Web site on selecting moving companies  and creating brochures on consumer rights and responsibilities—support  either of these objectives.
On the basis of our discussions with FMCSA officials, it appears the agency  is attempting to more clearly describe how education and outreach  activities link to the agency’s goals. FMCSA acknowledged that there may  have been some disconnection between the strategic planning and  budgeting processes in the past, but managers have recently received  training on defining a program’s desired outcome and then linking  inputs—such as funding—and outputs. Officials told us that as a result the  agency’s proposed fiscal year 2007 performance budget will better link  programs and goals to budget requests. FMCSA’s eventual goal is to link  program outputs to outcomes, as well as identifying measurable and  verifiable goals to the extent possible.
section_title: A Logic Model Provides a Framework for Linking Program Activities to Desired Results and Goals
paragraphs: Although the relationships among program activities, changes in  knowledge and behaviors, and broader goals can sometimes seem intuitive,  laying a program out through a logic model can be useful for program  managers and stakeholders to demonstrate and understand these  relationships. FMCSA officials state that they use a logic model in its  performance budget to link education and outreach to agency objectives.  However, the performance budget implies a link between the education  and outreach program budget and agency goals without describing how  FMCSA intends the program to change behaviors, such as following federal  safety regulations, to support agency goals, as would be shown in a logic  model. FMCSA has used a logic model in another program area that  provides a description of how a program is intended to support agency  goals. Specifically, FMCSA’s Research and Technology group used a logic  model along with other planning tools to demonstrate how its activities  support FMCSA’s goals in its 5-Year Strategic Plan. The Research and  Technology logic model includes inputs such as staff, outputs such as  research studies, outcomes such as improved understanding of root causes  of crashes, and impacts such as fewer commercial motor vehicle crashes.  FMCSA described the Research and Technology Logic Model as a  “mechanism to  help define metrics for performance, quality, and relevance  of the program.”
As FMCSA endeavors to describe how its education and outreach program  activities affect the knowledge, attitudes, and behaviors of its target  audiences, and link these results to broader goals, it may wish to consider a  tool such as a logic model—as the agency used in its Research and  Technology strategic plan—to clearly outline those links. In previous work  on educational and outreach, we used a logic model to describe the  program components—inputs and outputs—and how these support the  desired results, thus explaining the strategy—or logic—by which programs  are expected to achieve their goals. Figure 3 graphically depicts this model  and its components.
The logic model illustrates how program activities such as distributing  informational pamphlets contribute to the results of a program, such as  increasing safety-belt usage, and to an ultimate goal, such as reducing  fatalities. In addition, program managers should consider the effect of  external factors when defining outcomes and recognize that the correlation  between activities and desired outcomes may not be as direct as one would  like. Without a logic model or similar tool to clearly describe the links  between education and outreach programs and broader goals, FMCSA may  have difficulty designing and evaluating effective programs.
section_title: Evaluations of Specific Education and Outreach Programs Impacts Are Still in Preliminary Stages, but the Lack of a Plan to Evaluate How New Entrant Safety Audits Improve Knowledge and Attitudes Raises Concern
paragraphs: FMCSA has begun some evaluations of its education and outreach  activities, including the New Entrant program, and plans to use contractors  to evaluate some programs. Surveys and studies are the primary means by  which FMCSA or its contractor will evaluate the programs. However, some  gaps in evaluation plans remain, especially for the largest of the programs  we reviewed. Specifically, FMCSA has not evaluated whether the  Education and Technical Assistance Package provided to new entrants and  the safety audits conducted under the New Entrant program effectively  communicated information to new entrants, making it difficult to  determine the impact of that program. A list of FMCSA’s evaluations and  planned evaluations are provided in figure 4.
For each program, FMCSA’s evaluation efforts are described as follows.
subsections: section_title: New Entrant Program
paragraphs: FMCSA evaluates the performance of the New Entrant program primarily  by tracking the number of safety audits conducted of new motor carriers.  The agency has a goal of conducting a certain number of safety audits per  year, safety audits of 80 percent of new entrants within the first 6 months  they are in business, and 100 percent of all new entrants within their first 18  months of operation. FMCSA sets employee performance goals for division  office managers for conducting a particular number of safety audits of new  entrants. According to FMCSA officials, agency personnel conducted over  9,400 safety audits in fiscal year 2005, exceeding their goal of 8,000 FMCSA  completed safety audits. In addition, by April 2005, 52,000 safety audits had  been conducted since the program began, and the number conducted each  quarter has been generally increasing. However, FMCSA officials  acknowledge that there is a safety audit backlog in some states, and the  goals for auditing new entrants in the first 6 months, and the first 18  months, are not being met. The number of safety audits reflects the number  of new registrants that were personally informed about the carrier  requirements. Using registration data on the number of new entrants, the  agency can estimate the number of new entrants needing a safety audit and  agency progress toward the program goal. There are about 40,000 new  entrants registered per year, but officials told us that about 40 percent drop  out or discontinue business within the first year of operations.
By tracking the number of safety audits conducted, FMCSA seeks to  determine the extent to which it has provided information to its targeted  audience of new drivers. FMCSA has no process to measure whether new  entrants’ awareness of safe trucking practices was raised, which ideally  would be a consequence of an effective education and outreach effort.  Specifically, although FMCSA plans to evaluate the New Entrant program  in 2008, it has no current plan to determine how FMCSA’s safety  requirement information, such as its Education and Technical Assistance  Package and its safety audits, affect new entrants’ (1) awareness of FMCSA  requirements, such as drug and alcohol testing requirements or (2) their  motivation to change behaviors that might lead to safer operations. The  New Entrant program manager indicated that since the program is new and  still evolving, the outreach campaign does not have much historical data to  evaluate.
Although the program has not been in existence very long, FMCSA officials  stated that the agency has conducted one study of the New Entrant  program and has plans to conduct another. One preliminary analysis by  FMCSA compared the crash rates of new motor carriers registering with  FMCSA during 2002 to the crash rates of over 1,000 new motor carriers that  registered with FMCSA and had a safety audit during the first 6 months of  2003. This analysis showed that there was little difference between the  crash rates of these two groups. FMCSA officials said that they are  planning a study to be conducted after planned changes to the safety audit,  discussed later in this report, have been in place for a period of time.  Officials told us that the study planned to begin in 2008 will examine  carriers’ safety records over time, although they have not yet developed an  evaluation plan.
We have previously reported on the difficulties of assessing education and  outreach programs such as the New Entrant program. We have also  reported, however, that federal agencies are expected to demonstrate how  their activities contribute to achieving agency or governmentwide goals.  Additionally, the guidance that FMCSA uses when developing evaluation  efforts for education and outreach programs—a booklet entitled The Art of  Appropriate Evaluation —published by NHTSA—emphasizes that  evaluations should include measurements of changes in awareness,  knowledge, and behavior. If FMCSA’s planned evaluation of the New  Entrant program does not consider whether the program is actually  succeeding in effectively conveying the educational material on safe  trucking practices, FMCSA will be unable to clearly determine the  program’s impact. Or, if safety audits are having little impact, as suggested  by FMCSA’s preliminary analysis, the agency will not be able to determine  whether it is due to (1) the educational material provided, (2) how the  information is conveyed during the safety audits, (3) the lack of follow-up  on the safety audit results, or (4) other factors. Additionally, it will be  difficult to demonstrate to Congress that this program is having its desired  effect.
section_title: Non-Entrant Program
paragraphs: This program has not yet begun. FMCSA officials plan to award a contract  by the end of 2005 to develop, implement, and evaluate the Non-Entrant  program. The contractor is to test the effectiveness of the outreach  materials while they are being developed to determine if they will reach the  intended audience and effectively changed knowledge and attitudes.  Development of the materials will include surveying the targeted audience  and conducting outreach through truck leasing and sales firms. FMCSA’s  program plan indicated that the contractor will be expected to assess  changes in knowledge, attitudes, and behaviors of its target audience.
section_title: Commercial Motor Vehicle Safety Belt Program
paragraphs: FMCSA officials stated that they plan to evaluate this initiative in several  ways once they have finalized its evaluation arrangements. According to  FMCSA officials, during program design, FMCSA tested its marketing  materials with target audiences, including truck drivers, and found that the  marketing material was appropriately targeted. Also, FMCSA is installing  software to track the number of Web-site hits on certain of its Web pages  that contain important educational elements of the Safety Belt campaign.  This system will enable FMCSA to determine how many times its  information has been viewed, but it will not enable FMCSA officials to  know whether that information was effectively conveyed to its target  audience or changed their attitudes about wearing safety belts.  Nonetheless, FMCSA officials believe that this software will enable  program managers to monitor and determine which Web-site pages are  viewed, and therefore are more effective, based on the number of hits and  the length of time at each site. Furthermore, FMCSA is considering focus  groups and surveys with carriers on how safety belt marketing material  affected their awareness of the importance of using safety belts.
In addition, FMCSA has an agreement with NHTSA for two efforts that  could measure progress toward the expected outcome of this initiative—a  10-percent increase in use of safety belts from 2006 through 2009. The first  effort is a study, expected to be completed in December 2005, which  updates a 2003 commercial motor vehicle study that reported that 48  percent of commercial drivers wear safety belts. The new study will update  information on the percentage of commercial drivers that wear safety belts  by replicating the methodology used in the 2003 study, according to FMCSA  officials. The second effort will investigate the feasibility of annually  collecting data on safety-belt use by commercial drivers through the  National Occupant Protection Usage Study (NOPUS) sponsored by  NHTSA. Although these efforts will measure safety-belt use, they will not  evaluate whether the education and outreach program in particular  contributed to changes in the number of individuals wearing safety belts.
section_title: Motor Coach Outreach Program
paragraphs: FMCSA has a Web-based, pop-up survey for its Web site which, although  focusing on consumer satisfaction in general, gives FMCSA some ideas  about Web-site usage and exposure to the information contained on the  site. The survey randomly selects visitors to the Motor Coach Web site and  asks them to answer a series of questions concerning the accuracy, quality,  convenience, and ability of finding needed information. The survey also  asks visitors about their overall satisfaction with the Web site. Although  these questions may be generally helpful to rate the Web site experience,  more specific questions on the Motor Coach program information have not  been incorporated. The pop-up survey has flexibility for use in other  FMCSA program Web pages, and FMCSA has plans to use Web-based usage  tracking as an additional tool in the future. However, with only a small  budget—$150,000 in fiscal year 2005—for the Motor Coach program, it is  reasonable to expect that FMCSA would balance the need for this type of  evaluation information with its other education and outreach program  priorities.
As of October 2005, FMCSA officials also had plans to award a contract for  contractor support to conduct surveys, focus groups, and interviews to  evaluate its education and outreach programs, including obtaining Office of  Management and Budget (OMB) clearance for these activities. FMCSA  officials told us that the motor coach program would be evaluated under  this contract.
section_title: Household Goods Outreach Program
paragraphs: FMCSA has identified at least two approaches to evaluate the effectiveness  of the outreach efforts for the household goods outreach program. One  effort is to survey recipients who have received the Ready to Move - Tips  for a Successful Interstate Move booklet from GSA. As of May 2005, 703  surveys had been distributed to determine whether the recipients believed  that the booklet would change their intentions about which household  goods carrier they might select. The survey asks questions regarding the  effectiveness of the booklet and its information content. Also, FMCSA  officials stated that the Household Goods program would be evaluated  under the survey support contract discussed in the preceding section on  the Motor Coach program.
An FMCSA official whom we spoke with indicated that the elderly may be  particularly targeted by unscrupulous movers. Recent data released by the  Census Bureau indicate that only about 40 percent of people age 55 years  or older have computers with Internet access in their homes. Although  FMCSA has an available hard-copy brochure, the Web-based outreach  efforts may not be fully reaching this population. FMCSA’s planned  evaluations of the Household Goods program do not address the  effectiveness of FMCSA’s Web-based outreach in reaching this population.
section_title: The Effectiveness of Education and Outreach Programs Is Unclear, but Programs Seem to be Reasonably Designed According to Theories of Behavior Change and Industry Groups
paragraphs: Most of FMCSA’s evaluations of its education and outreach programs have  yet to be completed, and therefore little is currently known about the  effectiveness of the programs. FMCSA expects the contracts for the  evaluation programs to be finalized within months and will receive  information from the evaluations once the data are collected and analyzed.  Until that time, what is known about program effectiveness is largely based  on anecdotal information from stakeholders such as industry associations,  as well as a comparison of the education and outreach programs to the  design of other successful programs that have been implemented.  However, FMCSA’s education and outreach programs and marketing  materials appear to follow theories and research on behavior change.  Finally, a public-safety group suggests FMCSA followed reasonable  approaches in starting its education and outreach efforts, however, it  would like to see more quantitative evaluation in the future to help FMCSA  refine its programs.
subsections: section_title: Behavior Theory Indicates Education and Outreach Can Be Effective in Increasing Awareness and Changing Behavior for Programs that Do Not Have Regulations Governing the Behavior of Their Target Audiences
paragraphs: Studies by us, other agencies, and academics show that education and  outreach programs, such as those undertaken by FMCSA, can be effective  in creating awareness of a problem or issue and influencing some change in  behavior. One theory—the theory of reasoned action—explains the  relationship between attitudes, beliefs, intentions, and behavior. In short,  people are more likely to change their behavior if they believe the change is  valued—or that not changing their behavior has negative  consequences—and if they have the necessary opportunity to change the  behavior. For example, one antismoking advertising campaign emphasized  the negative effects of smoking on a smoker’s health. Smokers became  aware of the negative consequences, and 26 percent of respondents  indicated that they were more likely to quit smoking based on the  awareness of the negative consequences of smoking. Another study on the  effects of sun exposure showed that countering the mistaken belief that  sun tanning had a positive effect reduced the percentage of individuals who  habitually lie in the sun. These studies demonstrate that emphasizing the  consequences of a negative behavior to individuals can result in changes to  attitudes and behavior, without applying any specific sanctions.
Based on the theory of reasoned action, education and outreach can have a  positive impact through FMCSA’s two programs that have a goal of  influencing consumers to make appropriate choices of motor carriers.  FMCSA’s Motor Coach and Household Goods Outreach programs are  targeted toward consumers and do not have regulations governing the  consumers’ behavior, thus, according to the theory of reasoned action,  gaining knowledge about the positive consequences of choosing a safe  carrier will affect the individuals’ attitude toward this behavior. The change  in attitude will directly affect intention and the subsequent behavior. Thus,  education and outreach campaigns can provide an individual under these  circumstances with the information that is needed to make a more  educated choice. For example, the Motor Coach program follows this  approach by targeting consumers and conveying the negative  consequences of choosing an unsafe carrier. In the case of the Household  Goods program, the campaign conveys the positive consequences of  choosing a reputable carrier in the Households Goods program, although  the elderly, a group potentially susceptible to unscrupulous carriers, may  require a different delivery method other than FMCSA’s Web site to receive  this message.
section_title: A Link to Enforcement Can Strengthen Education and Outreach Programs’ Ability to Change Behavior
paragraphs: Independent studies as well as our prior work indicate that the level of  enforcement in safety related programs is positively correlated with  safety-belt use, especially when coupled with public-awareness efforts.  According to the general deterrence theory, individuals must be exposed to  law enforcement or receive information about legal ramifications of their  actions before they can be deterred. The perception or expectation of  certain sanctions results in attempts to avoid committing the offense.  Studies show this theory is an effective model for strengthening the change  in behavior resulting from safety programs. Thus, this link to enforcement  would apply to FMCSA’s three education and outreach programs—Safety  Belt, Non-Entrant and New Entrant—that have existing regulations that  address their target audience’s behavior.
One study, conducted by the National Safety Council, on the state of South  Carolina’s safety-belt program, showed that while media campaigns  immediately increased awareness of the importance of safety belts, the  largest increase in safety belt-use occurred after officers enforced the  requirement to wear safety belts. Additionally, a 2002 study by NHTSA  included data from the state of Washington, which showed that the  baseline percentage of individuals wearing safety belts (81 percent)  increased slightly with advertising over a 4-week period to 82 percent.  Ultimately with the addition of enforcement efforts, the percentage of  those who wear safety belts increased to 90 percent within 2 months. The  same study showed similar results in Texas, where the effects of  advertising alone raised the safety belt percentage slightly, but the  combination of advertising with enforcement caused the number to  increase another 6 percent, from the 80-percent baseline.  These studies  suggest that although publicity may increase the awareness of the need to  wear safety belts, some people are unlikely to act on that awareness unless  the enforcement component of the campaign was closely integrated with  the publicity campaign. The results of these studies suggest that education  and outreach alone can increase awareness and encourage the intended  behaviors for FMCSA’s education and outreach programs, but that  enforcement can strengthen the result of these efforts.
section_title: FMCSA Is Increasing Enforcement in the New Entrant Program and Acknowledges the Importance of Enforcement to the Non-Entrant and Safety Belt Programs
paragraphs: Three of FMCSA’s programs we examined, the New Entrant, Non-Entrant  and Commercial Motor Vehicle Safety Belt programs have regulations, such  as a requirement for motor carrier drivers to wear safety belts, affecting the  target audiences’ (motor carrier industry) behavior. This provides the  opportunity to use both enforcement and education in programs with the  goal of improving the safety behavior of the motor carrier industry. The  following sections describe how FMCSA has indicated it is using or will be  using enforcement along with education and outreach in these programs.
subsections: section_title: New Entrant Program
paragraphs: Enforcement has been only a minor component of the New Entrant  program, but FMCSA is currently strengthening the criteria for passing  New Entrant safety audits because of concerns of its effectiveness, based  on analysis and anecdotal information indicating that new entrants that  have been notified of the safety audit requirements may not be operating  more safely. In addition, even if a new entrant passes the safety audit, some  carriers are not yet following important safety requirements, such as  having a drug and alcohol testing program, at the time the safety audit was  conducted. This change should establish a better link to enforcement, in  keeping with behavioral research, in that it will be more difficult for new  entrants to retain their operating status unless they are following the safety  requirements at the time of the audit.
The safety audit has been considered an educational unit, with a strong  emphasis on training new entrants about the requirements to operate in  interstate commerce. For example, the federal register notice that  established the new entrant safety assurance process indicated that the  new entrant process is intended to “improve the safety performance of new  entrants by providing educational and technical assistance” and required  “minimum requirements to new entrant motor carriers to ensure that they  are knowledgeable about applicable federal motor carrier safety  standards.” There are few sanctions for carriers who fail critical areas of  the safety audit, although a new entrant might be subject to a more  stringent and detailed review—a compliance review—under certain  circumstances. However, the compliance review process occurs less  often with new motor carriers than experienced carriers.
Although FMCSA’s requirements indicate that the objectives of the safety  audit are to educate the carrier and determine areas where the carrier’s  compliance might be deficient, in practical application almost all new  entrants have passed the safety audit, and little is done to follow-up to  ensure that new motor carriers correct deficiencies identified in the safety  audits. A new entrant can fail two of six sections and still pass the audit.   Safety audit sections include areas such as driver qualifications, business  operations, and maintenance records. Over a 2-year period ending in April  2005, there were 51,681 carriers that passed the safety audit, and only 372  failed, producing a 99.3 percent passing rate for new entrant motor  carriers. Those who fail the audit must take corrective action within 45  days if they are a passenger or hazardous material carrier, or 60 days for all  other carriers, or they will lose operating status. Furthermore, for those  carriers who passed the audit despite failing multiple sections of it, FMCSA  currently has no mechanism to ensure the failed sections of the audit have  been corrected before the carrier achieves permanent status. Although  FMCSA sends letters to new entrants about the sections in which they are  not in compliance with motor carrier regulations; beyond the letter and  agency follow-up to obtain proof of insurance, the agency or its  representatives do not determine if the new entrants have in fact corrected  deficiencies identified during the safety audits.
Our analysis of critical factors within the safety audit questions revealed  that over the same 2-year period, about 40 percent of the carriers failed the  “driver” section of the audit, despite passing the audit overall. This section  includes critical questions, such as whether a carrier’s driver qualifications  have been reviewed and whether the carrier's drivers are covered by a drug  and alcohol testing program. Specifically, we found that about 60 percent of  the new entrant carriers that passed the safety audit had failed the  requirement to check qualifications of drivers prior to hiring them.  Checking qualifications means that before hiring a driver a motor carrier  company obtains and reviews information on an applicant’s driving history,  including accident and ticket information. In addition, we found that for the  most part, between 27 and 29 percent of carriers who passed the safety  audit had not maintained maintenance files for requested vehicles. The lack  of improvement over the 2-year period illustrates the importance of  determining if the Education and Technical Assistance Package, which the  carriers received prior to the audit, should be changed to improve these  results.
FMCSA officials agreed that the safety audit may not be improving safety  behavior of new entrants, and the agency is already tightening these  requirements. Currently, there is no one question or section of the audit  that can cause a new entrant to fail. FMCSA officials informed us that the  new federal register notice laying out stricter pass requirements—expected  to be issued shortly—identify eight critical questions that individually  could lead to failing the safety audit, thus providing more incentive for  carriers to comply with new entrant requirements. For example, carriers  would automatically fail if they do not employ a drug and alcohol program,  have insurance, or have the proper maintenance requirements. Carriers  who fail the safety audit and do not correct the deficiencies could lose their  operating status and would be required to correct deficiencies before their  operating status is reinstated.
section_title: Safety Belt Program
paragraphs: Similarly to the New Entrant program, there has been limited enforcement  associated with the Safety Belt program to date. The finding that 48 percent  of commercial drivers wear safety belts, compared with 80 percent of  passenger vehicle drivers, suggests that commercial motor vehicle  operators have thus far been less receptive to messages in the market place  regarding the importance of wearing safety belts while on duty. In light of  the studies we reviewed, an enforcement element might help strengthen  education and outreach programs’ attempts to increase awareness of the  risk to drivers of not wearing safety belts and encourage commercial  drivers to wear them. It is reasonable to assume that commercial motor  vehicle operators, similar to private operators, would be educated and  informed from the materials presented in the campaign but that some  would not act on the information unless there was some risk of sanction.  The approved plan for FMCSA’s safety belt initiatives includes a third phase  of implementation that calls for increased enforcement. Although the plan  does not specify how FMCSA will enforce federal regulations regarding  wearing safety belts and how FMCSA will work with state agencies to carry  out enforcement—as this program develops, strengthening the  enforcement link may increase the effectiveness of the safety belt  education and outreach efforts. FMCSA officials indicated that this  enforcement effort had begun and included fiscal year 2005 grants to state  agencies that can be used for safety belt education and enforcement.
section_title: Non-Entrant Program
paragraphs: FMCSA’s program plan for the Non-Entrant program indicates it will have  some enforcement component as it is developed; however, specific  information on enforcement has not yet been developed. Literature on  safety programs and experience with similar programs indicates the  Non-Entrant program might benefit from enforcement after non-entrants  are exposed to the message about the importance of registering for a DOT  license when they buy or rent a vehicle for operation.
section_title: Industry Associations Offer A Mostly Positive View of FMCSA’s Education and Outreach Efforts, but Public Advocacy Groups Expressed Some Concerns
paragraphs: Overall, the industry associations’ view is that FMCSA is doing several  positive things in its education and outreach activities. Officials of the  seven industry associations whom we spoke with had regular contact with  FMCSA officials and were familiar with the education and outreach  programs under this review. Many association officials cited direct contact  with FMCSA on industry issues on a regular basis and said that their  partnerships with FMCSA on education and outreach activities were  productive. Specifically, an official of one moving industry association told  us that it shared data with FMCSA regarding complaints against carriers  and often refers complainants to FMCSA for further information. Officials  representing a consortium of primarily state agencies involved in motor  carrier safety, to which FMCSA belongs, told us that the support FMCSA  was able to provide was important to the implementation of the Safety Belt  program. Lastly, some association officials said that FMCSA was doing a  good job with the resources it has, but others said FMCSA could do more  with its education and outreach efforts if the agency had more resources.
However, one of the two public safety groups we contacted believed that  although the education and outreach programs were based on good ideas,  FMCSA needs to do more quantitative evaluations of these programs. Both  public safety groups also felt that FMCSA should have stronger  enforcement. For example, one group indicated that research has shown  enforcement to increase the effectiveness of education programs and thus,  FMCSA should focus more on compliance and enforcement. Finally,  although industry associations emphasized the positive professional  relationships they have with FMCSA officials, a public safety group  indicated that it is wary about FMCSA’s tendency to manage programs in  alliance with industry partners, since the group believes it might affect the  agency’s objectivity.
section_title: Conclusions
paragraphs: Uncertainty about how FMCSA’s education and outreach activities link to  broader program and agency goals and about FMCSA’s New Entrant  program have emerged from our study. FMCSA officials have stated that  the agency’s education and outreach programs support department  objectives and that they have used a logic model to show this in the  agency’s performance budgets. However, in our review of agency  documents, we found that except for the Safety Belt program, FMCSA has  not established clear links between the education and outreach programs  and agency and departmental goals. For example, the conference report  accompanying the DOT appropriations bill for fiscal year 2005 requested  FMCSA report to the House and Senate Committees on Appropriations “a  goal, message, and coherent and explicit program strategy that clearly and  directly link FMCSA’s outreach and education program initiatives to each  program’s goals” because of its concerns over how FMCSA’s education and  outreach activities support agency goals. In the October 2005 response,  FMCSA stated that its education and outreach programs support DOT  strategic objectives. The report, however, did not clearly set out the links  between changing attitudes and behaviors that would describe how these  programs contribute to objectives. Such a step would be welcome, for  without an explanation of that link, FMCSA cannot transparently  demonstrate to stakeholders, including congressional oversight authorities  that the programs do, in fact, contribute to these broader goals.  Furthermore, without a clear statement of how these programs are  intended to support these goals, it is difficult to design evaluations to  determine if these programs are performing as intended or if changes are  needed in program design. FMCSA has used a type of a logic model for its  Research and Technology program, making it clear to all interested parties  how its research activities help the agency achieve its goals. While FMCSA  officials say they have used a logic model for its education and outreach  program, this is not evident in agency budget and program documents. The  use of a logic model, similar to the one used by the agency’s Research and  Technology group, in planning, budgeting or program for education and  outreach programs may be beneficial.
Second, the absence of a plan to evaluate how well the information  provided to motor carriers as part of the New Entrant program was  understood and if the information raised awareness and helped promote  safer behavior also emerges as a concern. An FMCSA official stated that  the New Entrant program is still new and has limited data to evaluate.  Further, changes to the program are planned to raise the threshold for  passing a safety audit. Thus, FMCSA is planning on starting an evaluation of  the New Entrant program in 2008. However, without an evaluation of the  effect of education on new entrants’ knowledge about safety requirement,  FMCSA is losing an opportunity to obtain information about the  effectiveness of the educational information conveyed through the  program. In addition, FMCSA cannot know whether its safety audits, which  composed almost 7.5 percent of FMCSA’s 2005 funding, are having the  desired effect; whether the Education and Technical Assistance Package  and the information conveyed during the safety audits might require some  changes to improve their effectiveness; and whether the safety audits  themselves might be achieving their purpose of improving motor carrier  safety. FMCSA has two opportunities upcoming to obtain information on  the effectiveness of the educational materials and safety audits. First, it  could use its planned contract for evaluation of education and outreach  activities to develop and conduct such an evaluation. Second, FMCSA  could include this evaluation in its plans to evaluate the New Entrant  program beginning in 2008.
section_title: Recommendations for Executive Action
paragraphs: To better demonstrate how FMCSA education and outreach programs  contribute to achieving agency goals, we recommend that the Secretary of  Transportation direct the Administrator of FMCSA to take the following  two actions:   Ensure that the agency describes and documents how education and  outreach program activities link to and support broader program and  agency goals in a planning, program, or budget document that is  available to the public, and  Evaluate the effectiveness of the education and outreach of the New  Entrant program, assessing the extent to which the Education and  Technical Assistance Package and safety audits are helping new carriers  learn and understand FMCSA requirements.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Transportation for  its review and comment. FMCSA officials, through the DOT liaison,  provided oral comments on a draft of this report. The officials did not agree  with GAO’s finding that FMCSA has not linked its education efforts with its  overall goals. FMCSA officials state they have linked their education and  outreach programs to agency goals at a high level in the agency’s  performance budgets. In addition, the officials state that they used a logic  model to make this link. We included FMCSA’s view that it has linked its  education and outreach programs to agency goals in our report. However,  we did not find this link evident for most of the education and outreach  programs. For example, Congress requested FMCSA to report on the link  between its education and outreach program initiatives to each program’s  goal. FMCSA’s report to Congress described the program activities;  however, it did not describe the link between the activities and FMCSA’s  goals. In addition, although FMCSA officials stated that they have used a  logic model in their performance budget to link education and outreach  programs to agency goals, the discussion implied a link, but does not      describe how FMCSA’s education and outreach activities are intended to  change target audience’s attitudes and behaviors that would contribute to  meeting agency goals. However, in FMCSA’s fiscal year 2006 performance  budget, we did find and report on a relatively clear discussion of how the  Commercial Motor Vehicle Safety Belt program supported safety objectives  such as reduced fatalities.
In addition, DOT and FMCSA officials were concerned that our  recommendation on documenting the link between education programs  and agency goals was too prescriptive. We recommended that FMCSA  expedite the process of describing and documenting how its education and  outreach programs link to and support overall agency goals in strategic  planning and budget documents. DOT officials said that including this  specific information on relatively small dollar projects—such as the Motor  Coach program—was not appropriate in the DOT strategic plan. In  addition, DOT officials told us that it is not appropriate to include this level  of performance information in budget requests. Since the strategic plan and  the performance budget are not the only opportunities to detail program  linkage to broad agency goals, we clarified our recommendation to provide  FMCSA with more flexibility in identifying an appropriate planning,  program, or budgeting document in which to describe and document the  link between program activities and goals, so that external stakeholders,  including congressional oversight authorities, can understand how these  programs are supporting agency goals.
Second, agency officials suggested that it is too early to evaluate the New  Entrant program, as we recommended; and they have not developed  evaluation plans because they do not plan on beginning an evaluation until  2008. However, after over 2 years of implementation and 52,000 safety  audits, FMCSA is losing an opportunity to conduct an evaluation of the  educational component of the New Entrant program, including the  Educational and Technical Assistance Package for new entrants and the  information conveyed during safety audits to determine if new motor  carriers increase their knowledge of motor carrier requirements and  change their behavior to better comply with these requirements. The lack  of such an evaluation is a missed opportunity for FMCSA to demonstrate  the effectiveness of its educational information and safety audits, which  represented almost 7.5 percent of its funding in fiscal year 2005. In  addition, we continue to believe this is an important component of any  evaluation FMCSA develops to determine the effectiveness of the New  Entrant program. Therefore, we retained our recommendation to evaluate   the New Entrant program and conclude that there are two opportunities for  doing so—either through contracts for evaluations of education and  outreach programs or through the evaluation FMCSA is planning to start in  2008.
Finally, agency officials provided technical comments, which we  incorporated as appropriate.
We are sending copies of this report to the Honorable Norman Mineta,  Secretary of Transportation. We will also make copies available to others  upon request. In addition, the report will be available at no charge on the  GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at siggerudk@gao.gov or (202) 512-2834. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Major contributors to this report are listed in appendix III.
section_title: Scope and Methodology
paragraphs: To address our first objective and describe the scope and nature of the  Federal Motor Carrier Safety Administration’s (FMCSA) education and  outreach programs, we reviewed program documentation and interviewed  FMCSA officials at agency headquarters, its Eastern Service Center in  Maryland, and its New Jersey division office. We selected this division  office and service center because they were significantly involved in all  four of the currently active education and outreach programs we reviewed,  which are the New Entrant, Commercial Safety Belt, Motor Coach  Outreach, and Household Goods Outreach programs. We obtained and  reviewed information on FMCSA’s plans to initiate an additional program  targeted at non-entrant carriers.
Further, to determine how these programs relate to FMCSA’s goals, we  reviewed its strategic planning and performance budgeting documents and  how these education and outreach programs were portrayed in the current  fiscal year 2006 documents. We also discussed the performance budgeting  process with an FMCSA senior budget analyst to identify the agency’s plans  for the fiscal year 2007 and 2008 performance budgets.  We also obtained  and reviewed the Department of Transportation’s (DOT) strategic plan,  which FMCSA refers to in its performance budget. In addition, we obtained  the strategic plan for FMCSA’s Research and Technology group. We  reviewed this plan to identify what one group within FMCSA uses for its  strategic and program planning. To identify other tools used for describing  how program activities relate to agency goals, we reviewed our previous  work and identified a program logic model, originally developed by the  University of Wisconsin Cooperative Extension, as a model that FMCSA  could use to clearly describe its education and outreach programs.
To address our second objective—identifying the extent to which FMCSA  has evaluated its education and outreach programs, we discussed the  status of these evaluation efforts with FMCSA officials and reviewed  proposals and statements of work for current and planned contracts to  carry out program evaluations. We also assessed the usefulness of FMCSA’s  ongoing or planned program evaluations to determine if the ealuations  would address how program activities affected target audiences’ attitudes,  knowledge, and behavior. We did not thoroughly assess the evaluation  design.
We took several approaches to assess what is currently known about the  effectiveness of programs like FMCSA’s education and outreach programs.  Finally, to identify what is known about the effectiveness of the programs,  we conducted a literature review of media based campaigns and reviewed  the structure and evaluation of these campaigns as well as behavioral  theories used in designing the campaigns. We selected two behavioral  theories that were directly applicable to the programs under this review— the theory of reasoned action and the general deterrence theory. We then  directly compared the characteristics of the two theories with the  characteristics of FMCSA’s programs. Additionally, through our literature  search, we also identified additional research that discusses the  importance of the relationship between education and enforcement in  changing behavior. We used this research to help assess the potential for  using enforcement to augment FMCSA’s education and outreach efforts. We  obtained aggregated data from safety audit results from FMCSA’s Motor  Carrier Management Information System (MCMIS) to examine results for  individual elements within a safety audit. We used these data to identify the  failure rates for key elements of the safety audit. We conducted a reliability  assessment of MCMIS data by (1) reviewing documentation related to  system development, (2) interviewing knowledgeable agency officials, and  (3) reviewing FMCSA’s internal controls, and we determined the data was  sufficiently reliable for our use. Finally, we interviewed officials from  several motor carrier, law enforcement, and safety organizations, including  the American Trucking Associations; the National Private Truck Council;  the American Bus Association; the United Motorcoach Association; the  American Moving and Storage Association; the Commercial Vehicle Safety  Alliance; Advocates for Highway and Auto Safety; and Public Citizen for  their perspective of the effectiveness of FMCSA’s education and outreach  efforts.
In accordance with congressional direction, we did not extend our review  to include “Share the Road Safely” which is targeted to passenger car  drivers. In fiscal year 2004, Congress transferred funding for the program  from FMCSA the National Highway Traffic Safety Administration (NHTSA).  Funding for this program is being returned to FMCSA in fiscal year 2006. In  fiscal years 2004 and 2005, Congress provided FMCSA with funding for  another outreach effort entitled “Safety Is Good Business.” The SAFETEA- LU requires GAO to review this program by June of 2006.
Our work was conducted in accordance with generally accepted  government auditing standards, from December 2004 through October  2005.
section_title: FMCSA Education and Outreach Programs
paragraphs: The following provides a more detailed discussion of the five education and  outreach programs we reviewed.
subsections: section_title: New Entrant Program
paragraphs: The New Entrant program is intended to ensure that new motor carriers  are knowledgeable about federal motor carrier safety standards and  applicable hazardous material regulations. The program is focused on the  safety audits conducted through site visits to the new carriers’ offices by  personnel from the state law enforcement agency, the FMCSA division  office, or a FMCSA contractor.
In 1998, a FMCSA commissioned study found that new entrants to the  motor carrier industry have a substantially higher level of safety regulation  violations than more experienced carriers. Other, earlier research had also  indicated that new entrant carriers had crash rates higher than more  experienced carriers. The Motor Carrier Safety Improvement Act, which  established FMCSA, also directed the Secretary of Transportation to  establish regulations specifying minimum requirements for applicant motor  carriers seeking federal interstate operating authority and to require that  new entrants undergo a safety audit within the first 18 months of  operations. Effective January 1, 2003, FMCSA established its New Entrant  program to improve the safety of new entrant motor carriers by providing  them with educational and technical assistance as they begin their  businesses. Under FMCSA’s New Entrant program, any new motor carrier  that registers with DOT is designated a “new entrant” and will be subject to  a new entrant safety audit and increased roadside performance monitoring  for an 18-month period.
The safety audit’s purpose is to provide educational and technical  assistance to the new entrant by reviewing the carrier’s compliance with  FMCSA requirements, including driver qualifications, driver records of duty  status, vehicle maintenance records, accident registers, and controlled  substances and alcohol use and testing requirements. Education and  technical assistance information is also available to new entrants through  FMCSA’s Web site. A carrier that passes the safety audit is informed of any  deficiencies and what it must do to be in compliance. A carrier that fails the  safety audit has 60 days to take necessary corrective action, and ultimately  if the deficiencies are not corrected, its registration can be revoked. At the  end of the 18-month period, if the carrier has passed its safety audit, does  not have an unsatisfactory safety rating, and is not subject to an “out of  service” order or any notice to remedy safety management controls, the  “new entrant” designation will be removed from its registration, and it will  be monitored like any other carrier.
In most states, new entrant safety audits have been done by both state law  enforcement agency and FMCSA division office personnel. Contractors  conduct safety audits in four states (Florida, Maine, Oregon, and Wyoming)  that are presently doing few or none of their own. When the New Entrant  program began, FMCSA staff performed most of the safety audits, but  states now carry out the majority of these audits, as indicated by figure 5.
section_title: Non-Entrant Program
paragraphs: FMCSA’s Non-Entrant program will target motor carriers that operate  interstate but have not registered with DOT. The purpose of the program  will be to raise awareness of registration requirements, and to increase the  number of registrations among these non-entrant motor carriers. This  program has not yet begun, but FMCSA officials told us it will include  information disseminated through truck sales and leasing firms to non- entrants.
The number of non-entrants is unknown, but a FMCSA official estimates it  may be in the tens of thousands. FMCSA officials whom we spoke with  believe that non-entrants may represent an even higher level of risk than  new entrants, because they do not get exposure to FMCSA safety  information through registration. The officials believe that, in general,  motor carriers that do not register with DOT are unaware of the  requirement to do so, rather than intentionally avoiding registering. FMCSA  will be awarding a contract for the development of the Non-Entrant  program; the contract is expected to include a requirement to identify the  target population of non-entrants, develop informational material, with  follow-up monitoring of non-entrants that are identified.
section_title: Commercial Safety Belt Program
paragraphs: FMCSA’s Commercial Safety Belt Program is intended to improve upon the  low rate of safety-belt use by commercial truck drivers. In 2003, a study  commissioned by FMCSA reported that only 48 percent of commercial  truck drivers wear safety belts, compared with 80 percent of passenger car  drivers wearing safety belts. In December 2003, FMCSA established the  Commercial Motor Vehicle Safety Belt Partnership with 16 other  organizations, including trucking industry and law enforcement  associations, commercial drivers associations, and safety organizations,  with the goal of increasing commercial safety-belt use by commercial  motor vehicle drivers. With its safety belt partners, FMCSA produces  materials including posters, bumper stickers, and brochures for  distribution at trucking industry shows and events, and through law  enforcement campaigns. In 2004, as part of the Partnership, FMCSA helped  distribute informational brochures developed by the Commercial Vehicle  Safety Alliance (CVSA). In 2005, FMCSA introduced and distributed new  materials with the theme “Be Ready. Be Buckled.” for distribution at  national and regional trucking industry shows and events, and through the  CVSA-sponsored Road Check 2005. This was a 3-day truck and motor  coach inspection effort in June 2005 that involved over 9,900 inspectors,  conducting more than 60,000 vehicle inspections in the United States,  Canada, and Mexico. Through the Commercial Motor Vehicle Safety Belt  Partnership, FMCSA also communicates its message through information  displayed at truck stops and satellite radio programming oriented to truck  drivers.
section_title: Motor Coach Outreach Program
paragraphs: FMCSA’s objective for the motor coach industry is to reduce injuries and  save lives through increased compliance with safety regulations and  promotion of safe operations by motor coach owners and operators.  FMCSA’s approach is two-fold—(1) reaching out to the industry to inform  motor coach companies of regulations and provide safety advice and (2)  providing consumers with information to help them in selecting and hiring  motor coach services. FMCSA’s motor coach outreach activities include  communication with the carriers through printed material and  participation in trade shows, and Web-based information for the public. For  the purposes of our review, we focused on FMCSA’s outreach to members  of the public that hire motor coach services.
Motor coaches have been one of the safest forms of commercial  transportation, but an increase in motor coach occupant fatalities since  2001 led FMCSA to increase its focus on motor coach companies. In 2002,  FMCSA developed “Moving Kids Safely” a toolkit of reference materials  targeted to school districts and others who contract for bus services. In  2004, FMCSA introduced a motor coach safety Web site within the FMCSA  Web site, which provides guidance to consumers on selecting a motor  coach company and allows consumers to access safety data on specific  interstate motor coach companies.
FMCSA’s most direct outreach to motor coach companies is through the  agency’s Technical Assistance Group (TAG), consisting of about 12 FMCSA  staff with motor coach expertise, primarily in FMCSA’s field offices. TAG  personnel staff information booths and give presentations at industry  conventions and events, and serve as agency contact points for carriers.  Also, a FMCSA contractor developed a series of brochures aimed at motor  coach companies that describe guidelines and regulations for motor coach  companies (for instance licensing and insurance, hours of service,  inspection and maintenance, and specific requirements for operators of  school buses). In 2004 and 2005, FMCSA also conducted mass mailings to  motor coach companies, advising them of their reporting requirements  under the Americans with Disabilities Act.
section_title: Household Goods Outreach Program
paragraphs: Most households need to hire moving services infrequently, and therefore  consumers may be unaware of what their rights and protections are in  hiring a commercial mover. The overriding objective of FMCSA’s  Household Goods program is to help consumers make better informed  choices in selecting and negotiating with a moving company, primarily  through printed material distributed through moving companies and other  sources, as well as providing information through its Web site. FMCSA also  maintains a complaint hotline to receive complaints about movers from  individuals.
The ICC Termination Act of 1995 transferred federal responsibilities for  protecting consumers who use commercial moving companies for  interstate moves to DOT; and the Motor Carrier Safety Improvement Act  granted FMCSA continuing oversight responsibility over household goods  movers. However, the act did not provide FMCSA with the authority to  intercede on behalf of individuals seeking reimbursement of moving  company charges or recovery of their goods. FMCSA officials whom we  spoke with acknowledge that when the agency was established in January  2000, its mandate was commercial vehicle accidents and fatalities and, as a  result, the household goods industry received little attention. Also, in 2001,  we reported that complaints about movers were on the rise and that DOT’s  lack of action had created a vacuum that allowed unscrupulous carriers to  take advantage of consumers. At that time, our report noted that FMCSA  had just begun planning to increase both consumer education and  enforcement effort of household goods carriers.
FMCSA distributes three consumer-oriented publications. Interstate  movers are required by law to provide a copy of one FMCSA publication—  “Your Rights and Responsibilities When You Move” to prospective  customers. A second brochure, “Ready to Move? Tips for a Successful  Interstate Move” is available through the General Services Administration’s  Pueblo, Colorado Information Center. A third brochure “Protect your  Memories…Protect Yourself from Moving Fraud” and the “Your Rights and  Responsibilities” publications are available on FMCSA’s Web site.
In March 2005, FMCSA awarded a contract for the development of a  household goods education and outreach program. FMCSA and the  contractor have developed a new outreach theme: “Protect Your Memories,  Your Money, Your Move”; and the contractor will be developing brochures,  presentations that can be delivered by FMCSA staff, and other materials. In  June 2005, FMCSA launched its redesigned household goods Web page,  designed around the “Protect Your Memories…” theme, which incorporates  new content, including a checklist for moving and information on  protecting against moving fraud. In November 2005, FMCSA officials  informed us that they were in discussions with the U.S. Postal Service to  have the Postal Service Web site provide a link to FMCSA’s Web site when  registering a change of address with the Postal Service.
FMCSA officials told us they do not arbitrate complaints that the agency  receives, but they maintain a national consumer complaint database that is  used to identify problem movers for investigation by FMCSA field staff. Our  2001 report on DOT and FMCSA oversight of the household goods industry  recommended that FMCSA make information on the number and general  nature of complaints against carriers be made available to the public.  FMCSA officials whom we spoke with said the agency intends to do this,  but Privacy Act and other issues need to be resolved beforehand.
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Cathy Colwell, Assistant Director;  Colin Fallon; Eric Fielding; Donald Kittler; Sara Ann Moessbauer; and  Elaine Vaurio also made major contributions to this report.