Information Technology: Federal Motor Carrier Safety Administration Needs to Strengthen Its Strategic Planning and Oversight to Modernize Legacy Systems
Released: 2017-07-13T00:00:00

Letter

Background

FMCSA was established within DOT in January 2000 and was tasked with promoting safe commercial motor vehicle operations and preventing large truck and bus crashes, injuries, and fatalities. The commercial motor carrier industry is a vital part of the U.S. economy and, as of December 2015, FMCSA estimated that there were 551,150 active carriers and approximately 6 million commercial drivers operating in the United States. The domestic commercial motor carrier industry covers a range of businesses, including private and for-hire freight transportation, passenger carriers, and specialized transporters of hazardous materials. These carriers also range from small carriers with only one vehicle that is owned and operated by a single individual, to large corporations that own thousands of vehicles.

In carrying out its mission, FMCSA is responsible for four key safety service areas.

Registration Services: Motor carriers are required to register with FMCSA; have insurance; and attest that they are fit, willing, and able to follow safety standards. Vehicles must be properly registered and insured with the state of domicile and are subject to random and scheduled inspections by both state and FMCSA agents. Drivers must have a valid commercial driver’s license issued by their state of residence and pass a physical examination as evidenced by a current valid medical card every 2 years. In calendar year 2015, there were 57,358 active interstate new entrant carriers that registered with FMCSA.

Inspection Services: Conducting roadside inspections is central to FMCSA’s mission. States and, to a lesser extent, FMCSA staff, perform roadside inspections of vehicles to check for driver and maintenance violations and then provide the data from those inspections to the agency for analysis and determinations about a carrier’s safety performance. FMCSA also obtains data from the reports filed by state and local law enforcement officers when investigating commercial motor vehicle accidents or regulatory violations. The agency provides grants to states that may be used to offset the costs of conducting roadside inspections and improve the quality of the crash data the states report to it. In addition, the field offices in each state, known as divisions, have investigators who conduct compliance reviews of carriers identified by state inspection and other data as unsafe or at risk of being unsafe. FMCSA and its state partners conduct about 3.4 million inspections a year.

Compliance Services: FMCSA monitors and ensures compliance with regulations governing both safety and commerce. The compliance review process is performed by safety auditors and investigators who collect safety compliance data by visiting a motor carrier’s location to review safety and personnel records. In the instances of new carriers entering the commercial market, FMCSA audits these carriers within 12 months of service. In 2015, FMCSA conducted 14,656 investigations and 30,000 new entrant safety audits, and sent about 21,000 warning letters. FMCSA uses data collected from motor carriers, federal and state agencies, and other sources to monitor motor carrier compliance with the Federal Motor Carrier Safety Regulations and Hazardous Materials Regulations. These data are also used to evaluate the safety performance of motor carriers, drivers, and vehicle fleets. The agency uses the data to characterize and evaluate the safety experience of motor carrier operations to help federal safety investigators focus their enforcement resources by identifying the highest-risk carriers, drivers, and vehicles.

Enforcement Services: FMCSA is responsible for bringing legal action against companies that are not in compliance with motor carrier safety policies. In fiscal year 2015, FMCSA closed 4,766 enforcement cases.

FMCSA’s estimated budget for fiscal year 2017 is approximately $794.2 million. The agency employs more than 1,000 staff members who are located in its Washington, D.C., headquarters, 4 regional service centers, and 52 division offices.

Overview of FMCSA’s IT Environment

FMCSA’s Chief Information Officer (CIO) oversees the development, implementation, and maintenance of the IT systems and infrastructure that serve as the key enabler in executing FMCSA’s mission. The CIO reports directly to the Chief Safety Officer within FMCSA’s Office of Information Technology. This office supports a highly mobile workforce by operating the agency’s field IT network of regional and state service centers, and ensuring that inspectors have the tools and mobile infrastructure necessary to perform their roadside duties. In addition, the office supports FMCSA headquarters, regional, and state service centers, which depend on the agency’s IT infrastructure including servers, laptops, desktops, printers, and mobile devices. Currently, the Office of Information Technology is undergoing a reorganization to establish an Office of the CIO. While a revised structure has been proposed, it has not yet been approved.

Of its total budget, in fiscal year 2017, FMCSA’s expected IT budget is $58 million, of which approximately 60 percent ($34.4 million) is to be spent on the O&M of existing systems. In fiscal year 2013, the Office of Information Technology led an effort to establish a new IT portfolio that was intended to provide FMCSA with the ability to look across the investments in these portfolios and identify the linkages of business processes and strategic improvement opportunities to enhance mission effectiveness. To do so, the office implemented a product development team to integrate activities within and across the portfolio, interacting with business and program stakeholders. Specifically, it established four key safety process areas—registration, inspection, compliance, and enforcement—and two operations process areas—mission support systems and infrastructure.

The registration portfolio includes systems that process and review applications for operating authority.

The inspection portfolio includes systems that aid inspectors in conducting roadside inspections of large trucks and buses and ensure inspection data are available and useable.

The compliance portfolio includes systems that help investigators to identify and investigate carriers for safe operations and maintain high safety standards to remain in the industry.

The enforcement portfolio includes systems to assist the agency in ensuring that carriers and drivers are operating in compliance with regulations.

The mission support portfolio includes systems and services that crosscut multiple portfolios.

The infrastructure portfolio includes those systems that provide support services, hardware, software, licenses, and tools.

As of August 2016, FMCSA had identified and categorized 40 investments in its IT portfolio, as described in table 1.

According to the Acting CIO, by creating the IT portfolio, the agency determined that the functionality of these investments was not redundant, but that the aging legacy systems were in need of modernization. Further, the Acting CIO stated that the agency is planning to consolidate many of the systems that are in O&M, which, as of fiscal year 2016, had a combined cost of $2.9 million.

FMCSA Modernization Efforts

FMCSA has acknowledged the need to upgrade its aging systems to improve data processing and data quality, and reduce system maintenance costs. Accordingly, in 2013, it began a modernization effort that includes both developing new systems and retiring legacy systems for each of its four key safety process areas—registration, inspection, compliance, and enforcement.

To modernize its registration systems, in 2013, the agency began developing the URS system to streamline and strengthen the registration process. When fully implemented, URS is intended to replace the current registration systems with a single, online federal system. Program officials stated that the Licensing and Insurance system, Operations Authority Management system, and the registration function in MCMIS are to be retired upon URS’s deployment. The Acting CIO stated that the agency has not determined when URS will be fully deployed.

To modernize its inspection systems, FMCSA began planning efforts in 2014 to develop Integrated Inspection Management System (IIMS), which is intended to provide inspectors with a single system to perform checks. As of May 2017, the agency was still in the planning stage of this effort, as it was assessing the current state of its inspection processes and data management systems, and planning to issue a report detailing actions the agency needs to take. According to officials from the Office of Information Technology, subsequent to this report, a detailed analysis will be conducted, including development of acquisition and development plans. According to agency officials, its six operational inspection systems—Query Central, Safety and Fitness Electronic Records, SAFETYNET, Aspen, Inspection Selection System, and Commercial Driver’s License Information System Access—are intended to be retired upon deployment of IIMS.

To modernize its compliance systems, FMCSA began developing Sentri 2.1. According to the Acting CIO, the agency’s three legacy compliance systems—ProVu, National Registry of Certified Medical Examiners, and Compliance Analysis and Performance Review Information—are to be retired upon deployment of Sentri 2.1. As of May 2017, agency officials from the Office of Information Technology stated they have stopped the development of Sentri 2.1.

To modernize its enforcement systems, FMCSA intends to migrate the functionality of its current enforcement systems into an existing mission support system. Specifically, the functionality of FMCSA’s three operational enforcement systems—CaseRite, Electronic Management Information System, and Uniform Fine Assessment—is to be migrated into its Portal system, which is a website that provides users a single sign-on to access applications. The agency did not provide a date for when this effort is expected to be completed.

FMCSA’s Plans to Modernize Existing Systems Are Not Complete

A federal agency’s ability to effectively and efficiently maintain and modernize its existing IT environment depends, in large part, on how well it employs certain IT management controls, including strategic planning. Strategic planning is essential for an agency to define what it seeks to accomplish, identify strategies to efficiently achieve the desired results, and effectively guide modernization efforts. Key elements of IT strategic planning include establishing a plan with well-defined goals, strategies, measures, and timelines to guide these efforts. Our prior work stressed that an IT strategic plan should define the agency’s vision and provide a road map to help align information resources with business strategies and investment decisions.

Additionally, as we have previously reported, effective modernization planning is essential. Such planning includes defining the scope of the modernization effort, an implementation strategy, and a schedule, as well as establishing results-oriented goals and measures.

However, FMCSA lacks complete plans to guide its systems modernization efforts. Specifically, the agency’s IT strategic plan lacks key elements. While the agency has an IT strategic plan that describes the technical strategy, vision, mission, and direction for managing its IT modernization programs, and defines the strategic goals and objectives to support its mission, the plan lacks timelines to guide its goals and strategies related to integrated project planning and execution, IT security, and innovative IT business solutions, among others. For example, there were no identified milestones for achieving efficient, consolidated, and reliable IT solutions for IT modernization that meet the changing business needs of users and improve safety.

The Acting CIO acknowledged that the strategic plan is not complete and that a date by which a revised plan will be completed has not been established. The official further acknowledged that updating the current strategic plan has not been a priority. However, until the agency establishes a complete strategic plan, it is likely to face challenges in aligning its information resources with its business strategies and investment decisions.

In addition, FMCSA has not yet developed an effective modernization plan that defines the overall scope, implementation strategy, and schedule for its efforts. According to the Acting CIO, the agency has recognized the need for such a plan and has recently awarded a contract to develop one by June 2017. If FMSCA develops an effective modernization plan and uses it to guide its efforts, it should be better positioned to successfully modernize its aging legacy systems.

FMCSA Has Not Fully Implemented Key Elements of a Sound IT Governance and Oversight Process

GAO’s IT investment management framework is comprised of five progressive stages of maturity that mark an agency’s level of sophistication with regard to its IT investment management capabilities. Such capabilities are essential to the governance of an agency’s IT investments. At the Stage 2 level of maturity, an agency lays the foundation for sound IT investment management to help it attain successful, predictable, and repeatable investment governance processes at the project level. These processes focus on the agency’s ability to select, oversee, and review IT projects by defining and developing its IT governance board(s) and documented processes for directing the governance boards operations. According to the framework, Stage 2 includes the following three processes: Instituting the investment board: As part of this process, an agency is to establish an investment review board comprised of senior executives, including the agency’s head or a designee, the CIO or other senior executive representing the CIO’s interests, and heads of business units that are responsible for defining and implementing the department’s IT investment governance process. The agency’s IT investment process guidance should lay out the roles of investment review boards, working groups, and individuals involved in the agency’s IT investment processes.

Selecting investments that meet business needs: As part of the process for selecting and reselecting investments, an agency is to establish and implement policies and procedures made by senior executives that meet the agency’s needs. This includes selecting projects by identifying and analyzing projects’ risks and returns before committing any significant funds to them and selecting those that will best support the agency’s mission needs.

Providing investment oversight: This process includes establishing and implementing policies and procedures for overseeing IT projects by reviewing the performance of projects against expectations and taking corrective action when these expectations are not being met.

FMCSA has partially addressed the three processes associated with having a sound governance structure to manage its modernization efforts. Table 2 provides a summary of the extent to which the agency’s IT investment management structure implemented the key processes.

With regard to establishing an IT investment review board, FMCSA recently restructured its governance boards. Specifically, in January 2017, FMCSA finalized its IT governance order to have three major governance boards that are to serve as the decision-making structure for how IT investment decisions are made and escalated—the Executive Management Team, the Technical Review Board, and the Change Control Board.

At the highest level, the Executive Management Team is to provide strategic direction and decision making for major IT investments. The team, which is to meet at least quarterly, is chaired by the FMCSA Deputy Administrator. Below this team, the Technical Review Board is to provide oversight for all IT investments and is chaired by the Director of the Office of Information Technology Policy, Plans, and Oversight. According to the governance order, this team is to meet monthly. Further, underneath the Technical Review Board is the Change Control Board that has responsibility for reviewing and approving system change requests associated with a new system, a major release or modification to an existing system, a change in contract funding, or a change in contract scope. This board, which also is to meet monthly, is chaired by the Enterprise Architect of the Office of Information Technology Policy, Plans, and Oversight. Figure 1 depicts the agency’s governance structure.

Nevertheless, FMCSA has not yet clearly defined roles and responsibilities of all working groups and individuals involved in the agency’s IT governance process. For example, FMCSA’s governance order calls for the Office of Information Technology Policy, Plans, and Oversight to adopt specific IT performance measures, but does not define the manner in which these measures should be tracked.

Moreover, in August 2016, the agency finalized an order that established 10 integrated functional areas of IT management and the development of an Office of the CIO. However, FMCSA has not yet finalized a new structure for the Office of the CIO or clearly defined how this office and the CIO will manage, direct, and oversee the implementation of these areas as it relates to the agency’s IT governance process. Further, FMCSA officials have not identified time frames for doing so. Without clearly defined roles and responsibilities for the agency’s working groups and individuals involved in the governance process, FMCSA has less assurance that its modernization investments will be reviewed by those with the appropriate authority and aligned with agency goals.

With regard to selecting and reselecting IT investments, FMCSA’s January 2017 governance order requires participation and collaboration of the IT system owner, business owner, IT planning staff, and governance boards during the select phases for all investments. However, the agency lacks procedures for selecting new modernization investments and for reselecting investments that are already operational (which makes up the majority of the agency’s IT portfolio) for continued funding. For example, the order calls for the Executive Management Team, comprised of senior executives, to make decisions regarding the funding of the IT portfolio, among other things, and for the Technical Review Board to provide recommendations to the team on the prioritization of IT investments including the allocation of funds. However, the order does not specify the procedures for approving the movement of funds within the IT and capital planning and investment control portfolio.

According to the Acting CIO, FMCSA is currently drafting procedures for selecting new investments and reselecting investments that are already operational and intends to finalize the procedures by the end of May 2017. Upon establishing and implementing such procedures, FMCSA’s decision makers should have a common understanding of the process and the cost, benefit, schedule, and risk criteria that will be used to reselect IT projects.

With regard to IT investment oversight, the agency’s order established policies and procedures to ensure that governance bodies review investments and track corrective actions to closure. However, the policies and procedures for reviewing and tracking actions have not yet been fully implemented by the three governance bodies. For example,

The boards have not met regularly to review the performance of IT investments, including those investments that are part of its modernization efforts, against expectations. In particular, in calendar year 2016, the Executive Management Team met once and the Technical Review Board met four times. The Change Control Board was not formally approved until January 2017 and, thus, has held no meetings.

Also, while the Technical Review Board met four times in calendar year 2016, none of the meetings discussed the cost, schedule, performance, and risks for FMCSA’s major IT modernization investment, systems in development, or existing systems. For example, in February 2016, the IT Director presented to the board members an overview of the statutory provisions commonly referred to as the Federal Information Technology Acquisition Reform Act and their implications for FMCSA. In April 2016, the board members were provided with an overview of OMB’s regulatory guidance for the budget process. In addition, in August 2016, the Technical Review Board met to discuss the planned fiscal year 2017 budget for its IT investments and, in November 2016, the Director of the Office of Information Technology discussed with board members the status of the planning efforts for the IIMS project. The Acting CIO did not attend any of the four meetings.

Further, neither the Executive Management Team nor the Technical Review Board discussed with its members the transition of FMCSA’s investments into the cloud environment, to include identifying any key risks. For example, in November 2016, over 70 issues regarding the migration effort were identified by the contractor and a FMCSA official, but none were discussed at the Technical Review Board or Executive Management Team board meetings. As a result, program officials stated that there were delays to program’s transition to the cloud environment because additional time was needed to securely migrate data from multiple legacy platforms into a new central database and conduct further testing.

Action items have been noted in meeting minutes, but have not been fully addressed or updated to closure. For example, in August 2016, the Capital Planning and Investment Control Coordinator, within the Office of Information Technology, provided an overview of the fiscal year 2017 budget to the Technical Review board members. As part of this discussion, the Director of the Office of Information Technology stated that, during the next board meeting, additional details would be provided on the planned budget for fiscal year 2018. However, the meeting minutes from November 2016 did not include any evidence that this subject was discussed at the next meeting.

These weaknesses were due, in part, to the agency not adhering to its IT orders and governance board charters, which establish FMCSA’s governance structure, as described above. As a result, the agency lacks adequate visibility into and oversight of IT investment decisions and activities, and cannot ensure that its investments are meeting cost and schedule expectations and that appropriate actions are taken if these expectations are not being met.

FMCSA Did Not Fully Ensure That Selected IT Systems Are Effectively Meeting Agency Needs

According to OMB guidance, the O&M phase is often the longest phase of an investment and can consume more than 80 percent of the total lifecycle costs. Thus, it is essential that agencies effectively manage this phase to ensure that the investments continue to meet agency needs. As such, OMB and DOT direct agencies to monitor all O&M investments through operational analyses, which should be performed annually. These analyses should include assessments of four key factors: costs, schedules, investment performance (i.e., structured assessments of performance goals), and customer and business needs (i.e., whether the investment is still meeting customer and business needs, and identifies any areas for innovation in the area of customer satisfaction).

FMCSA had not fully ensured that the selected systems—Aspen, MCMIS, Sentri 2.0, and URS—were effectively meeting the needs of the agency. Specifically, none of the program offices conducted the required operational analyses for the four systems. The program offices stated that, in lieu of conducting these analyses, they assessed the key factors of costs, schedules, investment performance, and customer and business needs as part of the capital planning and investment control process. Nonetheless, only one program office (URS) partially met the four key factors. Table 3 provides a summary of the extent to which the four selected systems implemented the key operational analysis factors.

Aspen: The Aspen program office had partially implemented one of the required operational analysis factors and had not implemented the three other factors. Specifically, as part of its plans to modernize this system, FMCSA had taken steps to assess customer and business needs. For example, it reached out to users and found that 33 states use Aspen and the remaining states use their own in-house developed programs or third-party vendor-based systems. However, while the agency collected feedback from users via phone calls and meetings, it had not yet assessed this feedback, including identifying any opportunities for innovation in the areas of customer satisfaction, strategic and business results, and financial performance. In addition, the program office did not assess current costs against life-cycle costs, perform a structured schedule assessment, or compare current performance against cost baseline and estimates developed when the investment was being planned.

MCMIS: The MCMIS program office had not implemented any of the required operational analysis factors. Specifically, program officials did not assess current costs against life-cycle costs, perform structured assessments of schedule and performance goals, or identify whether the investment supports business and customer needs and is delivering the services it was designed to, including identifying whether the system overlaps with other systems. This is particularly concerning given that all seven users we interviewed stated that the system does not interact well with other systems and users have to access other systems to gather information that they cannot obtain in MCMIS.

Sentri 2.0: Sentri’s program office partially implemented one of the required operational analysis factors and did not implement the three other factors for the component that has been operational since May 2010, also known as Sentri 2.0. Specifically, the program had partially implemented assessments of customer and business needs by reviewing Sentri 2.0 user needs as it develops the business and user requirements for development of Sentri 2.1. However, while all five users we interviewed stated that their feedback regarding Sentri was provided to FMCSA, they were not sure whether the feedback was being implemented. Moreover, the program office had not identified whether the investment supports customer processes, as designed, and is delivering the goods and services it was intended to deliver. In addition, the program did not assess current costs against life-cycle costs or perform structured schedule and performance goal assessments.

URS: The URS program office partially implemented four of the required operational analysis factors for functionality of the system that was delivered in December 2015. Specifically, the program office developed a business case that outlines costs, schedules, investment performance goals, and customer and business needs. Additionally, the program office communicated with stakeholders through meetings, conferences, webinars, and call centers. For example, it has hosted over 30 webinars to better understand how the system is working for the users. Nevertheless, the program office had not yet conducted an analysis to assess current costs against life-cycle costs, performed a structured assessment of the schedule or performance goals, or ensured the functionality delivered is operating as intended and is meeting user needs. The need for conducting an analysis is particularly pressing for this program since all four system users we interviewed stated that URS is difficult to use and does not work as intended: they stated that they are unable to complete filings, carrier registration, and request changes to DOT numbers.

With regard to the deficiencies we identified, the Acting CIO stated that the agency does not yet have FMCSA-specific guidance to assist programs to conduct operational analyses on an annual basis. The Acting CIO stated that FMCSA has drafted guidance, including templates, to assist programs in conducting these analyses and officials in the Office of Information Technology stated that the agency planned to have the guidance finalized by end of June 2017. While finalizing this guidance is a positive step to assist programs in conducting operational analyses, FMCSA does not adequately ensure its systems are effective at meeting user needs. Until FMCSA fully reviews its O&M investments as part of its annual operational analyses, the agency will lack assurance that these systems meet mission needs, and the associated spending could be wasteful.

Conclusions

While FMCSA has recognized the need to develop an effective modernization plan and has awarded a contract to do so, it has not completed an IT strategic plan needed for modernizing its existing legacy systems. In addition, while the agency has established governance boards for overseeing IT systems, these boards do not exhibit key processes of a sound governance approach, such as ensuring corrective actions are executed and tracked to closure. Further, FMCSA does not have the processes in place for ensuring that systems currently in use are meeting agency needs or for overseeing its IT portfolio. The four systems we reviewed did not have completed operational analyses that show if a system is, among other things, effective at meeting users’ needs. Until the agency addresses shortcomings in strategic planning, IT governance, and oversight, its progress in modernizing its systems will likely be limited and the agency will be unable to ensure that the systems are working effectively.

Recommendations for Executive Action

To help improve the modernization of FMCSA’s IT systems, we are recommending that the Secretary of Transportation direct the FMCSA Administrator to take the following five actions:

Update FMCSA’s IT strategic plan to include well-defined goals, strategies, measures, and timelines for modernizing its systems.

Ensure that the IT investment process guidance lays out the roles and responsibilities of all working groups and individuals involved in the agency’s governance process.

Finalize the restructure of the Office of Information Technology, including fully defining the roles and responsibilities of the CIO.

Ensure that appropriate governance bodies review all IT investments and track corrective actions to closure.

Ensure that required operational analyses are performed for Aspen, MCMIS, Sentri 2.0, and URS on an annual basis.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Transportation for review and comment. In its written comments, reproduced in appendix II, the department concurred with our five recommendations.

The department also described actions that FMCSA has completed or is finalizing to improve its IT strategic planning and investment governance processes. These actions include updating the FMCSA IT strategic plan and finalizing investment review board charters to better define all stakeholders roles and responsibilities. Effective implementation of these actions should help FMCSA improve the modernization of its IT systems. In addition to the written comments, the department provided technical comments on the draft report, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Transportation, the Administrator of FMCSA, and other interested parties. This report also is available at no charge on the GAO website at http://www.gao.gov.

Should you or your staff have any questions on information discussed in this report, please contact me at (202) 512-4456 or Harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The Fixing America’s Surface Transportation Act included a provision for us to conduct a comprehensive analysis of the information technology (IT) and data collection management systems of the Federal Motor Carrier Safety Administration (FMCSA) by June 4, 2017. Our objectives were to (1) assess the extent to which the agency has plans to modernize its existing systems, (2) assess the extent to which FMCSA has implemented an IT governance structure, and (3) determine the extent to which FMCSA has ensured selected IT systems are effective.

To address the first objective, we obtained and evaluated FMCSA IT systems modernization documentation that discuss future changes to ensure user needs are met, including its IT strategic plan for fiscal years 2014 to 2016 and systems modernization plans. We analyzed whether these plans complied with best practices that we have previously identified. These practices call for developing a strategic plan that includes defining the agency’s vision and providing a road map to help align information resources with business strategies and investment decisions. We also interviewed agency officials including those from the Office of Information Technology; Enforcement and Compliance, Information Security, and Privacy divisions to discuss the agency’s plans to modernize existing systems, including any actions the agency is taking to identify redundancies among the systems and explore the feasibility of consolidating data collection and processing systems.

To corroborate this information, we reviewed the FMCSA’s budgetary data (i.e., its fiscal year 2016 IT portfolio summary) submitted to the Office of Management and Budget (OMB) that identifies all of the agency’s IT investments to identify whether it included any potentially redundant systems. Specifically, we reviewed the name and narrative description of each investment’s purpose to identify any similarities among related investments and discussed any potential redundancies with the Acting Chief Information Officer (CIO).

For the second objective, we compared agency documentation, including executive board meeting minutes and briefings from fiscal years 2015 and 2016, FMCSA IT governance orders, and charters, against critical processes associated with Stage 2 of GAO’s IT investment management framework. In particular, Stage 2 of the framework includes the following key processes for effective governance: instituting the investment board; selecting and reselecting investments that meet business needs; and providing investment oversight.

We also interviewed agency officials to better understand FMCSA’s governance structure, which included identifying whether the agency is taking appropriate steps with respect to IT governance.

To address the third objective, we selected four existing IT systems to review. In selecting these investments, we analyzed FMCSA’s fiscal year 2016 IT portfolio summary submitted to OMB which included the agency’s existing IT, data collection, processing systems, data correction procedures, and data management systems and programs. To assess the reliability of the OMB budget data, we reviewed related documentation, such as OMB guidance on budget preparation and capital planning. In addition, we corroborated with FMCSA that the data was accurate and reflected the data it had reported to OMB. We determined that the budget data was reliable for our purposes of selecting these systems. Specifically, we used the following criteria to select four systems to review:

At least one investment must have been identified as a major IT investment, as defined by OMB. FMCSA had only identified one major IT investment in fiscal year 2016.

The remaining non-major systems must have had planned operations and maintenance (O&M) spending in fiscal year 2017.

The system is mission critical.

The program must not have been included in a recent GAO or inspector general review that examined the program’s effectiveness.

Using the above criteria, we selected the following four systems: 1. Aspen: A non-major desktop application that collects commercial driver/vehicle inspection details, performs some immediate data analysis, creates and prints a vehicle inspection report, and transfers inspection data into the FMCSA information systems. 2. Motor Carrier Management Information System (MCMIS): A non- major information system that captures FMCSA inspection, crash, compliance review, safety audit, and registration data. It is FMCSA’s authoritative source for the safety performance records for all commercial motor carriers and hazardous materials shippers. 3. Safety Enforcement Tracking and Investigation System (Sentri): A non-major application used to facilitate safety audits and interventions by FMCSA and state users. It is intended to combine roadside inspection, investigative, and enforcement functions into a single interface. 4. Unified Registration System (URS): A major system that is intended to replace the existing registration systems with a single comprehensive, online system and provide FMCSA-regulated entities a more efficient means of submission and management of data pertaining to registration applications.

We then assessed the agency’s efforts to determine the effectiveness of these systems in meeting the needs of the agency by reviewing documentation from the four selected systems and compared it to key factors identified in OMB’s guidance on conducting annual operational analysis, which are a key method for examining the performance of investments with O&M funding. More specifically, we assessed whether FMCSA had conducted an operational analysis on each of the systems. For those systems that did not have an analysis performed, we reviewed FMCSA’s IT documentation on the performance of these systems (i.e., business cases and performance management reviews) to determine whether key factors of an operational analysis were conducted. For example, we assessed whether the agency assessed cost, schedule, and investment performance, including its interaction with other systems; and customer and business needs, including adaptability of the system in order to make necessary future changes to ensure user needs are met and areas for innovation in the areas of customer satisfaction.

We also conducted interviews with 22 selected system users to obtain insight into whether the identified systems are meeting their needs and any challenges users face in using these systems, including whether the systems are adaptable to future needs and methods to improve user interface. We selected these users based on recommendations from FMCSA program officials and industry stakeholder representatives. Based on these recommendations, we then selected users based on the type of users, including FMCSA users, state agencies, law enforcement officials, and private sector individuals involved in the motor carrier industry. While these user interviews are illustrative, they cannot be used to make generalizable statements about users’ experience as a whole.

Based on our work to determine selected programs’ effectiveness, we made recommendations regarding deficiencies identified in the report. We did not make recommendations regarding methods to improve user interfaces since two of the selected systems (Aspen and MCMIS) are planned to be modernized and the remaining two systems (Sentri and URS) have components still under development, as discussed in our report.

We conducted this performance audit from April 2016 to July 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Transportation

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, the following staff also made key contributions to this report: Eric Winter (Assistant Director), Niti Tandon (Analyst in Charge), Rebecca Eyler, Lisa Maine, and Tyler Mountjoy.