Information Technology: Federal Motor Carrier Safety Administration Needs to Strengthen Its Strategic Planning and Oversight to Modernize Legacy Systems
Released: 2017-07-13T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: FMCSA, established within the Department of Transportation in January 2000, is charged with reducing crashes involving commercial motor carriers (i.e., large trucks and buses) and saving lives. IT systems and infrastructure serve as a key enabler for FMCSA to achieve its mission. The agency reported spending about $46 million for its IT investments in fiscal year 2016.

In December 2015, the Fixing America's Surface Transportation Act was enacted and required GAO to review the agency's IT, data collection, and management systems. GAO's objectives were to (1) assess the extent to which the agency has plans to modernize its existing systems, (2) assess the extent to which FMCSA has implemented an IT governance structure, and (3) determine the extent to which FMCSA has ensured selected IT systems are effective. To do so, GAO analyzed FMCSA's strategic plan and modernization plans; compared governance documentation to best practices; selected four investments based on operations and maintenance spending for fiscal year 2016, among other factors, and compared assessments for the investments against OMB criteria; and interviewed officials.

section_title: What GAO Found
paragraphs: The Federal Motor Carrier Safety Administration (FMCSA) initiated a modernization effort in 2011 and developed an information technology (IT) strategic plan that describes the technical strategy, vision, mission, direction, and goals and objectives to support the agency's mission; however, the plan lacks timelines to guide FMCSA's goals and strategies. In addition, the agency has not completed a modernization plan for its existing IT systems that includes scope, an implementation strategy, schedule, results-oriented goals, and measures, although it has recently awarded a contract to develop such a plan. The Acting Chief Information Officer (CIO) said that updating FMCSA's IT strategic plan had not been a priority for the agency. However, without a complete IT strategic plan, FMCSA will be less likely to move toward its ultimate goal of modernizing its aging legacy systems.

FMCSA has begun to address leading practices of IT governance, but its investment governance framework does not adequately establish an investment board, select and reselect investments, and provide investment oversight. Specifically, regarding the practice of establishing an IT investment review board, FMCSA has not yet clearly defined roles and responsibilities for key working groups and individuals, including the Office of the CIO. Regarding selecting and reselecting IT investments, FMCSA requires participation and collaboration during the select phases for all IT investments; however, it lacks procedures for selecting new investments and reselecting investments that are already operational for continued funding. According to the Acting CIO, the agency is currently drafting these procedures and intends to finalize them by the end of May 2017. Regarding the practice of IT investment oversight, the agency has policies and procedures to ensure that corrective actions and related efforts are executed and tracked, but they have not yet been fully implemented by the three boards. These weaknesses are due to the agency not adhering to its IT orders that establish its governance structure. As a result, FMCSA lacks adequate visibility into and oversight of IT investment decisions and activities, which could ultimately hinder its modernization efforts.

FMCSA had not fully ensured that the four systems GAO selected to review are effectively meeting the needs of the agency because none of the program offices completed operational analyses as required by the Office of Management and Budget (OMB). However, as part of its capital planning and investment control process, FMCSA assessed the four key factors of an operational analysis—costs, schedules, investment performance, and customer and business needs. One of the selected programs had partially implemented all four of these factors; two programs had partially implemented one factor, and one program had not addressed any of these factors. This was due to FMCSA not having guidance for conducting operational analyses for investments in operations and maintenance. Until FMCSA fully reviews its operational investments, the agency will lack assurance that these systems meet mission needs.

section_title: What GAO Recommends
paragraphs: GAO is making five recommendations to FMCSA to improve its IT strategic planning, oversight, and operational analyses. The Department of Transportation concurred with all of the recommendations.

section_title: Letter
paragraphs: The mission of the Department of Transportation’s (DOT) Federal Motor  Carrier Safety Administration (FMCSA) is to reduce crashes, injuries, and  fatalities involving the more than 500,000 commercial motor carriers (i.e.,  large trucks and buses) operating in the United States. Information  technology (IT) systems and infrastructure serve as a key enabler for  FMCSA to achieve its mission of preventing crashes and saving lives.  The agency reported spending about $46 million for its IT investments in  fiscal year 2016.

The Fixing America’s Surface Transportation Act, which was enacted in  December 2015, included a provision calling for us to review FMCSA’s IT,  data collection, and management systems. Our specific objectives were  to (1) assess the extent to which the agency has plans to modernize its  existing systems, (2) assess the extent to which FMCSA has  implemented an IT governance structure, and (3) determine the extent to  which FMCSA has ensured selected IT systems are effective.

To address the first objective, we obtained and evaluated FMCSA  documentation on modernizing its systems, including its IT strategic plan  and IT modernization plans. We analyzed whether these plans complied  with best practices for IT strategic planning that we have previously  identified. These practices include developing a strategic plan that  defines the agency’s vision and provides a road map to help align  information resources with business strategies and investment decisions.  We also interviewed agency officials in the Office of Information  Technology; and the Enforcement and Compliance, Information Security,  and Privacy divisions, among others, to discuss the agency’s plans to  modernize existing systems.

To address the second objective, we evaluated agency documentation,  including executive board meeting minutes and briefings, charters, and  governance orders, against critical processes found in GAO’s IT  investment management framework that provides a method for evaluating  and assessing how well an agency is selecting and managing its IT  resources. We specifically focused on key processes identified in the  framework for instituting an investment board to manage its investments,  selecting and reselecting investments that meet business needs, and  providing investment oversight. We also interviewed agency officials in  the Office of Information Technology; and the Enforcement and  Compliance, Information Security, and Privacy divisions to better  understand FMCSA’s governance structure.

To address the third objective, we selected existing IT systems to  determine the extent that FMCSA has ensured they are effectively  meeting the needs of the agency. In selecting these systems, we  identified the systems in FMCSA’s fiscal year 2016 IT portfolio summary  submitted to the Office of Management and Budget (OMB) that had  planned operations and maintenance (O&M) spending funds for fiscal  year 2017. Our criteria focused on selecting at least one major system,  as defined by OMB; and at least one mission critical system, as defined  by FMCSA. Further, we aimed to select systems that were not included in  a recent GAO or inspector general review that examined program  effectiveness. Based on these criteria, we selected four investments for  our review:

Aspen is a non-major investment that is a desktop application which  collects commercial driver/vehicle inspection details and creates and  prints a vehicle inspection report. In fiscal year 2016, the agency  reported that O&M costs for Aspen was $138,000.

Motor Carrier Management Information System (MCMIS) is a non- major information system that captures data from field offices and is  the authoritative source for inspection, crash, compliance review,  safety audit, and registration data. In fiscal year 2016, the agency  reported that the O&M cost for MCMIS was $221,000.

Safety Enforcement Tracking and Investigation System (Sentri) is a  non-major investment that is currently used to facilitate safety audits  conducted by FMCSA and state users. FMCSA program officials told  us that Sentri involves two primary components: the New Entrant  Program component that has been operational since May 2010 and  the Compliance, Safety, and Accountability component that is  currently under development (also known as Sentri 2.0 and Sentri 2.1,  respectively). In fiscal year 2016, FMCSA reported spending about  $3.1 million on Sentri, of which about $138,000 was spent on O&M for  the existing component.

Unified Registration System (URS) is a major system that, when fully  deployed, is intended to replace existing registration systems with a  single comprehensive, online system and provide FMCSA-regulated  entities a more efficient means of submission and manipulation of  data pertaining to registration applications. The system is expected to  be delivered in three phases: the first phase was delivered in March  2014, the second phase was delivered in December 2015, and the  delivery of the final phase is to be determined. In fiscal year 2016,  FMCSA reported spending about $3.8 million on URS, of which about  $332,000 was for O&M of the capabilities already delivered.

We compared FMCSA’s IT documentation on the performance of the  selected systems (i.e., business cases and performance management  reviews) to OMB criteria on operational analysis that provides guidance to  help agencies ensure their existing investments are effective.

We also conducted interviews with selected users to obtain their insight  into whether the identified systems were meeting their needs and whether  they had faced any challenges in using these systems. Specifically,  based on recommendations from agency officials in the Office of  Information Technology; the Enforcement and Compliance, Information  Security, and Privacy divisions; and industry stakeholder  representatives, we selected 22 system users from the following groups:  FMCSA users, state agencies, law enforcement officials, and private  sector individuals involved in the motor carrier industry. Our selection  included at least one representative from each of these user groups.  Additional details on our objectives, scope, and methodology are  contained in appendix I.

We conducted this performance audit from April 2016 to July 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: FMCSA was established within DOT in January 2000 and was tasked  with promoting safe commercial motor vehicle operations and preventing  large truck and bus crashes, injuries, and fatalities. The commercial motor  carrier industry is a vital part of the U.S. economy and, as of December  2015, FMCSA estimated that there were 551,150 active carriers and  approximately 6 million commercial drivers operating in the United  States. The domestic commercial motor carrier industry covers a range  of businesses, including private and for-hire freight transportation,  passenger carriers, and specialized transporters of hazardous  materials. These carriers also range from small carriers with only one  vehicle that is owned and operated by a single individual, to large  corporations that own thousands of vehicles.

In carrying out its mission, FMCSA is responsible for four key safety  service areas.

Registration Services: Motor carriers are required to register with  FMCSA; have insurance; and attest that they are fit, willing, and  able to follow safety standards. Vehicles must be properly registered  and insured with the state of domicile and are subject to random and  scheduled inspections by both state and FMCSA agents. Drivers must  have a valid commercial driver’s license issued by their state of  residence and pass a physical examination as evidenced by a current  valid medical card every 2 years. In calendar year 2015, there were  57,358 active interstate new entrant carriers that registered with  FMCSA.

Inspection Services: Conducting roadside inspections is central to  FMCSA’s mission. States and, to a lesser extent, FMCSA staff,  perform roadside inspections of vehicles to check for driver and  maintenance violations and then provide the data from those  inspections to the agency for analysis and determinations about a  carrier’s safety performance. FMCSA also obtains data from the  reports filed by state and local law enforcement officers when  investigating commercial motor vehicle accidents or regulatory  violations. The agency provides grants to states that may be used to  offset the costs of conducting roadside inspections and improve the  quality of the crash data the states report to it. In addition, the field  offices in each state, known as divisions, have investigators who  conduct compliance reviews of carriers identified by state inspection  and other data as unsafe or at risk of being unsafe. FMCSA and its  state partners conduct about 3.4 million inspections a year.

Compliance Services: FMCSA monitors and ensures compliance  with regulations governing both safety and commerce. The  compliance review process is performed by safety auditors and  investigators who collect safety compliance data by visiting a motor  carrier’s location to review safety and personnel records. In the  instances of new carriers entering the commercial market, FMCSA  audits these carriers within 12 months of service. In 2015, FMCSA  conducted 14,656 investigations and 30,000 new entrant safety  audits, and sent about 21,000 warning letters. FMCSA uses data  collected from motor carriers, federal and state agencies, and other  sources to monitor motor carrier compliance with the Federal Motor  Carrier Safety Regulations and Hazardous Materials Regulations.  These data are also used to evaluate the safety performance of motor  carriers, drivers, and vehicle fleets. The agency uses the data to  characterize and evaluate the safety experience of motor carrier  operations to help federal safety investigators focus their enforcement  resources by identifying the highest-risk carriers, drivers, and  vehicles.

Enforcement Services: FMCSA is responsible for bringing legal  action against companies that are not in compliance with motor carrier  safety policies. In fiscal year 2015, FMCSA closed 4,766 enforcement  cases.

FMCSA’s estimated budget for fiscal year 2017 is approximately $794.2  million. The agency employs more than 1,000 staff members who are  located in its Washington, D.C., headquarters, 4 regional service centers,  and 52 division offices.
subsections: section_title: Overview of FMCSA’s IT Environment
paragraphs: FMCSA’s Chief Information Officer (CIO) oversees the development,  implementation, and maintenance of the IT systems and infrastructure  that serve as the key enabler in executing FMCSA’s mission. The CIO  reports directly to the Chief Safety Officer within FMCSA’s Office of  Information Technology. This office supports a highly mobile workforce by  operating the agency’s field IT network of regional and state service  centers, and ensuring that inspectors have the tools and mobile  infrastructure necessary to perform their roadside duties. In addition, the  office supports FMCSA headquarters, regional, and state service centers,  which depend on the agency’s IT infrastructure including servers, laptops,  desktops, printers, and mobile devices. Currently, the Office of  Information Technology is undergoing a reorganization to establish an  Office of the CIO. While a revised structure has been proposed, it has not  yet been approved.

Of its total budget, in fiscal year 2017, FMCSA’s expected IT budget is  $58 million, of which approximately 60 percent ($34.4 million) is to be  spent on the O&M of existing systems. In fiscal year 2013, the Office of  Information Technology led an effort to establish a new IT portfolio that  was intended to provide FMCSA with the ability to look across the  investments in these portfolios and identify the linkages of business  processes and strategic improvement opportunities to enhance mission  effectiveness. To do so, the office implemented a product development  team to integrate activities within and across the portfolio, interacting with  business and program stakeholders. Specifically, it established four key  safety process areas—registration, inspection, compliance, and  enforcement—and two operations process areas—mission support  systems and infrastructure.

The registration portfolio includes systems that process and review  applications for operating authority.

The inspection portfolio includes systems that aid inspectors in  conducting roadside inspections of large trucks and buses and ensure  inspection data are available and useable.

The compliance portfolio includes systems that help investigators to  identify and investigate carriers for safe operations and maintain high  safety standards to remain in the industry.

The enforcement portfolio includes systems to assist the agency in  ensuring that carriers and drivers are operating in compliance with  regulations.

The mission support portfolio includes systems and services that  crosscut multiple portfolios.

The infrastructure portfolio includes those systems that provide  support services, hardware, software, licenses, and tools.

As of August 2016, FMCSA had identified and categorized 40  investments in its IT portfolio, as described in table 1.

According to the Acting CIO, by creating the IT portfolio, the agency  determined that the functionality of these investments was not redundant,  but that the aging legacy systems were in need of modernization. Further,  the Acting CIO stated that the agency is planning to consolidate many of  the systems that are in O&M, which, as of fiscal year 2016, had a  combined cost of $2.9 million.

section_title: FMCSA Modernization Efforts
paragraphs: FMCSA has acknowledged the need to upgrade its aging systems to  improve data processing and data quality, and reduce system  maintenance costs. Accordingly, in 2013, it began a modernization effort  that includes both developing new systems and retiring legacy systems  for each of its four key safety process areas—registration, inspection,  compliance, and enforcement.

To modernize its registration systems, in 2013, the agency began  developing the URS system to streamline and strengthen the  registration process. When fully implemented, URS is intended to  replace the current registration systems with a single, online federal  system. Program officials stated that the Licensing and Insurance  system, Operations Authority Management system, and the  registration function in MCMIS are to be retired upon URS’s  deployment. The Acting CIO stated that the agency has not  determined when URS will be fully deployed.

To modernize its inspection systems, FMCSA began planning efforts  in 2014 to develop Integrated Inspection Management System (IIMS),  which is intended to provide inspectors with a single system to  perform checks. As of May 2017, the agency was still in the planning  stage of this effort, as it was assessing the current state of its  inspection processes and data management systems, and planning to  issue a report detailing actions the agency needs to take. According to  officials from the Office of Information Technology, subsequent to this  report, a detailed analysis will be conducted, including development of  acquisition and development plans. According to agency officials, its  six operational inspection systems—Query Central, Safety and  Fitness Electronic Records, SAFETYNET, Aspen, Inspection  Selection System, and Commercial Driver’s License Information  System Access—are intended to be retired upon deployment of IIMS.

To modernize its compliance systems, FMCSA began developing  Sentri 2.1. According to the Acting CIO, the agency’s three legacy  compliance systems—ProVu, National Registry of Certified Medical  Examiners, and Compliance Analysis and Performance Review  Information—are to be retired upon deployment of Sentri 2.1. As of  May 2017, agency officials from the Office of Information Technology  stated they have stopped the development of Sentri 2.1.

To modernize its enforcement systems, FMCSA intends to migrate  the functionality of its current enforcement systems into an existing  mission support system. Specifically, the functionality of FMCSA’s  three operational enforcement systems—CaseRite, Electronic  Management Information System, and Uniform Fine Assessment—is  to be migrated into its Portal system, which is a website that provides  users a single sign-on to access applications. The agency did not  provide a date for when this effort is expected to be completed.

section_title: FMCSA’s Plans to Modernize Existing Systems Are Not Complete
paragraphs: A federal agency’s ability to effectively and efficiently maintain and  modernize its existing IT environment depends, in large part, on how well  it employs certain IT management controls, including strategic planning.  Strategic planning is essential for an agency to define what it seeks to  accomplish, identify strategies to efficiently achieve the desired results,  and effectively guide modernization efforts. Key elements of IT strategic  planning include establishing a plan with well-defined goals, strategies,  measures, and timelines to guide these efforts. Our prior work stressed  that an IT strategic plan should define the agency’s vision and provide a  road map to help align information resources with business strategies and  investment decisions.

Additionally, as we have previously reported, effective modernization  planning is essential. Such planning includes defining the scope of the  modernization effort, an implementation strategy, and a schedule, as well  as establishing results-oriented goals and measures.

However, FMCSA lacks complete plans to guide its systems  modernization efforts. Specifically, the agency’s IT strategic plan lacks  key elements. While the agency has an IT strategic plan that describes  the technical strategy, vision, mission, and direction for managing its IT  modernization programs, and defines the strategic goals and objectives to  support its mission, the plan lacks timelines to guide its goals and  strategies related to integrated project planning and execution, IT  security, and innovative IT business solutions, among others. For  example, there were no identified milestones for achieving efficient,  consolidated, and reliable IT solutions for IT modernization that meet the  changing business needs of users and improve safety.

The Acting CIO acknowledged that the strategic plan is not complete and  that a date by which a revised plan will be completed has not been  established. The official further acknowledged that updating the current  strategic plan has not been a priority. However, until the agency  establishes a complete strategic plan, it is likely to face challenges in  aligning its information resources with its business strategies and  investment decisions.

In addition, FMCSA has not yet developed an effective modernization  plan that defines the overall scope, implementation strategy, and  schedule for its efforts. According to the Acting CIO, the agency has  recognized the need for such a plan and has recently awarded a contract  to develop one by June 2017. If FMSCA develops an effective  modernization plan and uses it to guide its efforts, it should be better  positioned to successfully modernize its aging legacy systems.

section_title: FMCSA Has Not Fully Implemented Key Elements of a Sound IT Governance and Oversight Process
paragraphs: GAO’s IT investment management framework is comprised of five  progressive stages of maturity that mark an agency’s level of  sophistication with regard to its IT investment management capabilities.  Such capabilities are essential to the governance of an agency’s IT  investments. At the Stage 2 level of maturity, an agency lays the  foundation for sound IT investment management to help it attain  successful, predictable, and repeatable investment governance  processes at the project level. These processes focus on the agency’s  ability to select, oversee, and review IT projects by defining and  developing its IT governance board(s) and documented processes for  directing the governance boards operations. According to the framework,  Stage 2 includes the following three processes:  Instituting the investment board: As part of this process, an agency  is to establish an investment review board comprised of senior  executives, including the agency’s head or a designee, the CIO or  other senior executive representing the CIO’s interests, and heads of  business units that are responsible for defining and implementing the  department’s IT investment governance process. The agency’s IT  investment process guidance should lay out the roles of investment  review boards, working groups, and individuals involved in the  agency’s IT investment processes.

Selecting investments that meet business needs: As part of the  process for selecting and reselecting investments, an agency is to  establish and implement policies and procedures made by senior  executives that meet the agency’s needs. This includes selecting  projects by identifying and analyzing projects’ risks and returns before  committing any significant funds to them and selecting those that will  best support the agency’s mission needs.

Providing investment oversight: This process includes establishing  and implementing policies and procedures for overseeing IT projects  by reviewing the performance of projects against expectations and  taking corrective action when these expectations are not being met.

FMCSA has partially addressed the three processes associated with  having a sound governance structure to manage its modernization efforts.  Table 2 provides a summary of the extent to which the agency’s IT  investment management structure implemented the key processes.

With regard to establishing an IT investment review board, FMCSA  recently restructured its governance boards. Specifically, in January  2017, FMCSA finalized its IT governance order to have three major  governance boards that are to serve as the decision-making structure for  how IT investment decisions are made and escalated—the Executive  Management Team, the Technical Review Board, and the Change  Control Board.

At the highest level, the Executive Management Team is to provide  strategic direction and decision making for major IT investments. The  team, which is to meet at least quarterly, is chaired by the FMCSA Deputy  Administrator. Below this team, the Technical Review Board is to provide  oversight for all IT investments and is chaired by the Director of the Office  of Information Technology Policy, Plans, and Oversight. According to the  governance order, this team is to meet monthly. Further, underneath the  Technical Review Board is the Change Control Board that has  responsibility for reviewing and approving system change requests  associated with a new system, a major release or modification to an  existing system, a change in contract funding, or a change in contract  scope. This board, which also is to meet monthly, is chaired by the  Enterprise Architect of the Office of Information Technology Policy, Plans,  and Oversight. Figure 1 depicts the agency’s governance structure.

Nevertheless, FMCSA has not yet clearly defined roles and  responsibilities of all working groups and individuals involved in the  agency’s IT governance process. For example, FMCSA’s governance  order calls for the Office of Information Technology Policy, Plans, and  Oversight to adopt specific IT performance measures, but does not define  the manner in which these measures should be tracked.

Moreover, in August 2016, the agency finalized an order that established  10 integrated functional areas of IT management and the development of  an Office of the CIO. However, FMCSA has not yet finalized a new  structure for the Office of the CIO or clearly defined how this office and  the CIO will manage, direct, and oversee the implementation of these  areas as it relates to the agency’s IT governance process. Further,  FMCSA officials have not identified time frames for doing so. Without  clearly defined roles and responsibilities for the agency’s working groups  and individuals involved in the governance process, FMCSA has less  assurance that its modernization investments will be reviewed by those  with the appropriate authority and aligned with agency goals.

With regard to selecting and reselecting IT investments, FMCSA’s  January 2017 governance order requires participation and collaboration  of the IT system owner, business owner, IT planning staff, and  governance boards during the select phases for all investments.  However, the agency lacks procedures for selecting new modernization  investments and for reselecting investments that are already operational  (which makes up the majority of the agency’s IT portfolio) for continued  funding. For example, the order calls for the Executive Management  Team, comprised of senior executives, to make decisions regarding the  funding of the IT portfolio, among other things, and for the Technical  Review Board to provide recommendations to the team on the  prioritization of IT investments including the allocation of funds. However,  the order does not specify the procedures for approving the movement of  funds within the IT and capital planning and investment control portfolio.

According to the Acting CIO, FMCSA is currently drafting procedures for  selecting new investments and reselecting investments that are already  operational and intends to finalize the procedures by the end of May  2017. Upon establishing and implementing such procedures, FMCSA’s  decision makers should have a common understanding of the process  and the cost, benefit, schedule, and risk criteria that will be used to  reselect IT projects.

With regard to IT investment oversight, the agency’s order established  policies and procedures to ensure that governance bodies review  investments and track corrective actions to closure. However, the policies  and procedures for reviewing and tracking actions have not yet been fully  implemented by the three governance bodies. For example,

The boards have not met regularly to review the performance of IT  investments, including those investments that are part of its  modernization efforts, against expectations. In particular, in calendar  year 2016, the Executive Management Team met once and the  Technical Review Board met four times. The Change Control Board  was not formally approved until January 2017 and, thus, has held no  meetings.

Also, while the Technical Review Board met four times in calendar  year 2016, none of the meetings discussed the cost, schedule,  performance, and risks for FMCSA’s major IT modernization  investment, systems in development, or existing systems. For  example, in February 2016, the IT Director presented to the board  members an overview of the statutory provisions commonly referred  to as the Federal Information Technology Acquisition Reform Act  and their implications for FMCSA. In April 2016, the board members  were provided with an overview of OMB’s regulatory guidance for the  budget process. In addition, in August 2016, the Technical Review  Board met to discuss the planned fiscal year 2017 budget for its IT  investments and, in November 2016, the Director of the Office of  Information Technology discussed with board members the status of  the planning efforts for the IIMS project. The Acting CIO did not attend  any of the four meetings.

Further, neither the Executive Management Team nor the Technical  Review Board discussed with its members the transition of FMCSA’s  investments into the cloud environment, to include identifying any key  risks. For example, in November 2016, over 70 issues regarding the  migration effort were identified by the contractor and a FMCSA official,  but none were discussed at the Technical Review Board or Executive  Management Team board meetings. As a result, program officials  stated that there were delays to program’s transition to the cloud  environment because additional time was needed to securely migrate  data from multiple legacy platforms into a new central database and  conduct further testing.

Action items have been noted in meeting minutes, but have not been  fully addressed or updated to closure. For example, in August 2016,  the Capital Planning and Investment Control Coordinator, within the  Office of Information Technology, provided an overview of the fiscal  year 2017 budget to the Technical Review board members. As part of  this discussion, the Director of the Office of Information Technology  stated that, during the next board meeting, additional details would be  provided on the planned budget for fiscal year 2018. However, the  meeting minutes from November 2016 did not include any evidence  that this subject was discussed at the next meeting.

These weaknesses were due, in part, to the agency not adhering to its IT  orders and governance board charters, which establish FMCSA’s  governance structure, as described above. As a result, the agency lacks  adequate visibility into and oversight of IT investment decisions and  activities, and cannot ensure that its investments are meeting cost and  schedule expectations and that appropriate actions are taken if these  expectations are not being met.

section_title: FMCSA Did Not Fully Ensure That Selected IT Systems Are Effectively Meeting Agency Needs
paragraphs: According to OMB guidance, the O&M phase is often the longest phase  of an investment and can consume more than 80 percent of the total  lifecycle costs. Thus, it is essential that agencies effectively manage this  phase to ensure that the investments continue to meet agency needs. As  such, OMB and DOT direct agencies to monitor all O&M investments  through operational analyses, which should be performed annually.  These analyses should include assessments of four key factors: costs,  schedules, investment performance (i.e., structured assessments of  performance goals), and customer and business needs (i.e., whether the  investment is still meeting customer and business needs, and identifies  any areas for innovation in the area of customer satisfaction).

FMCSA had not fully ensured that the selected systems—Aspen, MCMIS,  Sentri 2.0, and URS—were effectively meeting the needs of the agency.  Specifically, none of the program offices conducted the required  operational analyses for the four systems. The program offices stated  that, in lieu of conducting these analyses, they assessed the key factors  of costs, schedules, investment performance, and customer and business  needs as part of the capital planning and investment control process.  Nonetheless, only one program office (URS) partially met the four key  factors. Table 3 provides a summary of the extent to which the four  selected systems implemented the key operational analysis factors.

Aspen: The Aspen program office had partially implemented one of  the required operational analysis factors and had not implemented the  three other factors. Specifically, as part of its plans to modernize this  system, FMCSA had taken steps to assess customer and business  needs. For example, it reached out to users and found that 33 states  use Aspen and the remaining states use their own in-house  developed programs or third-party vendor-based systems. However,  while the agency collected feedback from users via phone calls and  meetings, it had not yet assessed this feedback, including identifying  any opportunities for innovation in the areas of customer satisfaction,  strategic and business results, and financial performance. In addition,  the program office did not assess current costs against life-cycle  costs, perform a structured schedule assessment, or compare current  performance against cost baseline and estimates developed when the  investment was being planned.

MCMIS: The MCMIS program office had not implemented any of the  required operational analysis factors. Specifically, program officials  did not assess current costs against life-cycle costs, perform  structured assessments of schedule and performance goals, or  identify whether the investment supports business and customer  needs and is delivering the services it was designed to, including  identifying whether the system overlaps with other systems. This is  particularly concerning given that all seven users we interviewed  stated that the system does not interact well with other systems and  users have to access other systems to gather information that they  cannot obtain in MCMIS.

Sentri 2.0: Sentri’s program office partially implemented one of the  required operational analysis factors and did not implement the three  other factors for the component that has been operational since May  2010, also known as Sentri 2.0. Specifically, the program had partially  implemented assessments of customer and business needs by  reviewing Sentri 2.0 user needs as it develops the business and user  requirements for development of Sentri 2.1. However, while all five  users we interviewed stated that their feedback regarding Sentri was  provided to FMCSA, they were not sure whether the feedback was  being implemented. Moreover, the program office had not identified  whether the investment supports customer processes, as designed,  and is delivering the goods and services it was intended to deliver. In  addition, the program did not assess current costs against life-cycle  costs or perform structured schedule and performance goal  assessments.

URS: The URS program office partially implemented four of the  required operational analysis factors for functionality of the system  that was delivered in December 2015. Specifically, the program office  developed a business case that outlines costs, schedules, investment  performance goals, and customer and business needs. Additionally,  the program office communicated with stakeholders through  meetings, conferences, webinars, and call centers. For example, it  has hosted over 30 webinars to better understand how the system is  working for the users. Nevertheless, the program office had not yet  conducted an analysis to assess current costs against life-cycle costs,  performed a structured assessment of the schedule or performance  goals, or ensured the functionality delivered is operating as intended  and is meeting user needs. The need for conducting an analysis is  particularly pressing for this program since all four system users we  interviewed stated that URS is difficult to use and does not work as  intended: they stated that they are unable to complete filings, carrier  registration, and request changes to DOT numbers.

With regard to the deficiencies we identified, the Acting CIO stated that  the agency does not yet have FMCSA-specific guidance to assist  programs to conduct operational analyses on an annual basis. The Acting  CIO stated that FMCSA has drafted guidance, including templates, to  assist programs in conducting these analyses and officials in the Office of  Information Technology stated that the agency planned to have the  guidance finalized by end of June 2017. While finalizing this guidance is a  positive step to assist programs in conducting operational analyses,  FMCSA does not adequately ensure its systems are effective at meeting  user needs. Until FMCSA fully reviews its O&M investments as part of its  annual operational analyses, the agency will lack assurance that these  systems meet mission needs, and the associated spending could be  wasteful.

section_title: Conclusions
paragraphs: While FMCSA has recognized the need to develop an effective  modernization plan and has awarded a contract to do so, it has not  completed an IT strategic plan needed for modernizing its existing legacy  systems. In addition, while the agency has established governance  boards for overseeing IT systems, these boards do not exhibit key  processes of a sound governance approach, such as ensuring corrective  actions are executed and tracked to closure. Further, FMCSA does not  have the processes in place for ensuring that systems currently in use are  meeting agency needs or for overseeing its IT portfolio. The four systems  we reviewed did not have completed operational analyses that show if a  system is, among other things, effective at meeting users’ needs. Until the  agency addresses shortcomings in strategic planning, IT governance, and  oversight, its progress in modernizing its systems will likely be limited and  the agency will be unable to ensure that the systems are working  effectively.

section_title: Recommendations for Executive Action
paragraphs: To help improve the modernization of FMCSA’s IT systems, we are  recommending that the Secretary of Transportation direct the FMCSA  Administrator to take the following five actions:

Update FMCSA’s IT strategic plan to include well-defined goals,  strategies, measures, and timelines for modernizing its systems.

Ensure that the IT investment process guidance lays out the roles and  responsibilities of all working groups and individuals involved in the  agency’s governance process.

Finalize the restructure of the Office of Information Technology,  including fully defining the roles and responsibilities of the CIO.

Ensure that appropriate governance bodies review all IT investments  and track corrective actions to closure.

Ensure that required operational analyses are performed for Aspen,  MCMIS, Sentri 2.0, and URS on an annual basis.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Transportation for  review and comment. In its written comments, reproduced in appendix II,  the department concurred with our five recommendations.

The department also described actions that FMCSA has completed or is  finalizing to improve its IT strategic planning and investment governance  processes. These actions include updating the FMCSA IT strategic plan  and finalizing investment review board charters to better define all  stakeholders roles and responsibilities. Effective implementation of these  actions should help FMCSA improve the modernization of its IT systems.  In addition to the written comments, the department provided technical  comments on the draft report, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Transportation, the Administrator of  FMCSA, and other interested parties. This report also is available at no  charge on the GAO website at http://www.gao.gov.

Should you or your staff have any questions on information discussed in  this report, please contact me at (202) 512-4456 or Harriscc@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The Fixing America’s Surface Transportation Act included a provision for  us to conduct a comprehensive analysis of the information technology (IT)  and data collection management systems of the Federal Motor Carrier  Safety Administration (FMCSA) by June 4, 2017. Our objectives were to  (1) assess the extent to which the agency has plans to modernize its  existing systems, (2) assess the extent to which FMCSA has  implemented an IT governance structure, and (3) determine the extent to  which FMCSA has ensured selected IT systems are effective.

To address the first objective, we obtained and evaluated FMCSA IT  systems modernization documentation that discuss future changes to  ensure user needs are met, including its IT strategic plan for fiscal years  2014 to 2016 and systems modernization plans. We analyzed whether  these plans complied with best practices that we have previously  identified. These practices call for developing a strategic plan that  includes defining the agency’s vision and providing a road map to help  align information resources with business strategies and investment  decisions. We also interviewed agency officials including those from the  Office of Information Technology; Enforcement and Compliance,  Information Security, and Privacy divisions to discuss the agency’s plans  to modernize existing systems, including any actions the agency is taking  to identify redundancies among the systems and explore the feasibility of  consolidating data collection and processing systems.

To corroborate this information, we reviewed the FMCSA’s budgetary  data (i.e., its fiscal year 2016 IT portfolio summary) submitted to the  Office of Management and Budget (OMB) that identifies all of the  agency’s IT investments to identify whether it included any potentially  redundant systems. Specifically, we reviewed the name and narrative  description of each investment’s purpose to identify any similarities  among related investments and discussed any potential redundancies  with the Acting Chief Information Officer (CIO).

For the second objective, we compared agency documentation, including  executive board meeting minutes and briefings from fiscal years 2015 and  2016, FMCSA IT governance orders, and charters, against critical  processes associated with Stage 2 of GAO’s IT investment management  framework. In particular, Stage 2 of the framework includes the following  key processes for effective governance:  instituting the investment board;  selecting and reselecting investments that meet business needs; and providing investment oversight.

We also interviewed agency officials to better understand FMCSA’s  governance structure, which included identifying whether the agency is  taking appropriate steps with respect to IT governance.

To address the third objective, we selected four existing IT systems to  review. In selecting these investments, we analyzed FMCSA’s fiscal year  2016 IT portfolio summary submitted to OMB which included the agency’s  existing IT, data collection, processing systems, data correction  procedures, and data management systems and programs. To assess  the reliability of the OMB budget data, we reviewed related  documentation, such as OMB guidance on budget preparation and capital  planning. In addition, we corroborated with FMCSA that the data was  accurate and reflected the data it had reported to OMB. We determined  that the budget data was reliable for our purposes of selecting these  systems. Specifically, we used the following criteria to select four systems  to review:

At least one investment must have been identified as a major IT  investment, as defined by OMB. FMCSA had only identified one  major IT investment in fiscal year 2016.

The remaining non-major systems must have had planned operations  and maintenance (O&M) spending in fiscal year 2017.

The system is mission critical.

The program must not have been included in a recent GAO or  inspector general review that examined the program’s effectiveness.

Using the above criteria, we selected the following four systems:  1.  Aspen: A non-major desktop application that collects commercial  driver/vehicle inspection details, performs some immediate data  analysis, creates and prints a vehicle inspection report, and transfers  inspection data into the FMCSA information systems.  2.  Motor Carrier Management Information System (MCMIS): A non- major information system that captures FMCSA inspection, crash,  compliance review, safety audit, and registration data. It is FMCSA’s  authoritative source for the safety performance records for all  commercial motor carriers and hazardous materials shippers.  3.  Safety Enforcement Tracking and Investigation System (Sentri): A  non-major application used to facilitate safety audits and interventions  by FMCSA and state users. It is intended to combine roadside  inspection, investigative, and enforcement functions into a single  interface.  4.  Unified Registration System (URS): A major system that is intended to  replace the existing registration systems with a single comprehensive,  online system and provide FMCSA-regulated entities a more efficient  means of submission and management of data pertaining to  registration applications.

We then assessed the agency’s efforts to determine the effectiveness of  these systems in meeting the needs of the agency by reviewing  documentation from the four selected systems and compared it to key  factors identified in OMB’s guidance on conducting annual operational  analysis, which are a key method for examining the performance of  investments with O&M funding. More specifically, we assessed whether  FMCSA had conducted an operational analysis on each of the systems.  For those systems that did not have an analysis performed, we reviewed FMCSA’s IT documentation on the performance of these systems (i.e.,  business cases and performance management reviews) to determine  whether key factors of an operational analysis were conducted. For  example, we assessed whether the agency assessed cost, schedule, and  investment performance, including its interaction with other systems; and  customer and business needs, including adaptability of the system in  order to make necessary future changes to ensure user needs are met  and areas for innovation in the areas of customer satisfaction.

We also conducted interviews with 22 selected system users to obtain  insight into whether the identified systems are meeting their needs and  any challenges users face in using these systems, including whether the  systems are adaptable to future needs and methods to improve user  interface. We selected these users based on recommendations from  FMCSA program officials and industry stakeholder representatives.  Based on these recommendations, we then selected users based on the  type of users, including FMCSA users, state agencies, law enforcement  officials, and private sector individuals involved in the motor carrier  industry. While these user interviews are illustrative, they cannot be used  to make generalizable statements about users’ experience as a whole.

Based on our work to determine selected programs’ effectiveness, we  made recommendations regarding deficiencies identified in the report. We  did not make recommendations regarding methods to improve user  interfaces since two of the selected systems (Aspen and MCMIS) are  planned to be modernized and the remaining two systems (Sentri and  URS) have components still under development, as discussed in our  report.

We conducted this performance audit from April 2016 to July 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

section_title: Appendix II: Comments from the Department of Transportation

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, the following staff also made key  contributions to this report: Eric Winter (Assistant Director), Niti Tandon  (Analyst in Charge), Rebecca Eyler, Lisa Maine, and Tyler Mountjoy.


Question-summary annotations: questions: question: What is the nature of FMCSA's modernization plan?
answer: The Federal Motor Carrier Safety Administration (FMCSA) initiated a modernization effort in 2011 and developed an information technology (IT) strategic plan that describes the technical strategy, vision, mission, direction, and goals and objectives to support the agency's mission; however, the plan lacks timelines to guide FMCSA's goals and strategies.
child_questions: question: What else is the existing plan missing?
answer: In addition, the agency has not completed a modernization plan for its existing IT systems that includes scope, an implementation strategy, schedule, results-oriented goals, and measures, although it has recently awarded a contract to develop such a plan.

question: Why hasn't a comprehensive plan been created yet?
answer: The Acting Chief Information Officer (CIO) said that updating FMCSA's IT strategic plan had not been a priority for the agency.

question: In what way is the lack of a comprehensive plan holding back FMCSA?
answer: However, without a complete IT strategic plan, FMCSA will be less likely to move toward its ultimate goal of modernizing its aging legacy systems.

questions: question: What characterizes FMCA's IT governance strategy?
answer: FMCSA has begun to address leading practices of IT governance, but its investment governance framework does not adequately establish an investment board, select and reselect investments, and provide investment oversight.
child_questions: question: How is the investment board lacking?
answer: Specifically, regarding the practice of establishing an IT investment review board, FMCSA has not yet clearly defined roles and responsibilities for key working groups and individuals, including the Office of the CIO.

question: What is the current status of investment procedures?
answer: Regarding selecting and reselecting IT investments, FMCSA requires participation and collaboration during the select phases for all IT investments; however, it lacks procedures for selecting new investments and reselecting investments that are already operational for continued funding. According to the Acting CIO, the agency is currently drafting these procedures and intends to finalize them by the end of May 2017.

question: How is FMCSA faring in the area of investment oversight?
answer: Regarding the practice of IT investment oversight, the agency has policies and procedures to ensure that corrective actions and related efforts are executed and tracked, but they have not yet been fully implemented by the three boards.

question: What is the root of these weaknesses?
answer: These weaknesses are due to the agency not adhering to its IT orders that establish its governance structure.

questions: question: Why hasn't FMCSA ensured the reviewed programs were meeting their needs?
answer: FMCSA had not fully ensured that the four systems GAO selected to review are effectively meeting the needs of the agency because none of the program offices completed operational analyses as required by the Office of Management and Budget (OMB).
child_questions: question: What factors did FMCSA decide were crucial for operational analysis?
answer: However, as part of its capital planning and investment control process, FMCSA assessed the four key factors of an operational analysis—costs, schedules, investment performance, and customer and business needs.
child_questions: question: How did the FMCSA programs perform against these standards?
answer: One of the selected programs had partially implemented all four of these factors; two programs had partially implemented one factor, and one program had not addressed any of these factors.
child_questions: question: Why didn't they meet all of the standards?
answer: This was due to FMCSA not having guidance for conducting operational analyses for investments in operations and maintenance.

question: What is the risk of the status quo?
answer: Until FMCSA fully reviews its operational investments, the agency will lack assurance that these systems meet mission needs.

questions: question: What was GAO's role in the Fixing America's Surface Transportation Act?
answer: In December 2015, the Fixing America's Surface Transportation Act was enacted and required GAO to review the agency's IT, data collection, and management systems.
child_questions: question: What was GAO specifically reviewing?
answer: GAO's objectives were to (1) assess the extent to which the agency has plans to modernize its existing systems, (2) assess the extent to which FMCSA has implemented an IT governance structure, and (3) determine the extent to which FMCSA has ensured selected IT systems are effective.
child_questions: question: How was GAO to do this?
answer: To do so, GAO analyzed FMCSA's strategic plan and modernization plans; compared governance documentation to best practices; selected four investments based on operations and maintenance spending for fiscal year 2016, among other factors, and compared assessments for the investments against OMB criteria; and interviewed officials.