National Transportation Safety Board: Management and Operational Improvements Found, but Strategy Needed to Utilize Cost Accounting System
Published: 2013-07-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The NTSB plays a vital role in transportation safety. It is charged with investigating all civil aviation accidents in the United States and selected accidents in other transportation modes, determining the probable cause of these accidents, and making appropriate recommendations, as well as performing safety studies.
In 2006, NTSB's reauthorization legislation mandated GAO to annually evaluate its programs. From 2006 to 2008, GAO made 21 recommendations to NTSB aimed at improving management and operations across several areas. Since that time, NTSB has taken action to address all 21 recommendations. Some of these were completed by requiring only a single action, whereas others required continuing effort to achieve operational improvement. For this review, GAO examined the extent to which desired outcomes are being achieved in five areas where continuing effort was necessary. GAO analyzed workforce, financial, and program data, and interviewed agency officials about actions NTSB has taken.
section_title: What GAO Found
paragraphs: GAOs analysis found varying degrees of improvement associated with the National Transportation Safety Boards (NTSB) actions in areas selected for review.
 Training Center utilization . NTSB increased utilization of its Training Centerboth non-classroom and classroom spacesince 2006. NTSB has also set and achieved its cost recovery goal at the Training Center in the last 2 fiscal years, allowing NTSB to recover half of its operating costs.
 Recommendation close-out process . By automating the recommendation follow-up process, NTSB has reduced by about 3 months the amount of time it takes to respond to agencies on whether planned actions to implement NTSB recommendations are acceptable; this allows agencies to move forward with approved actions sooner than under NTSBs former paper-driven process.
 Communication . NTSB employees responses on federal employee surveys from 2004 to 2012 indicated an increase from 49 to 57 percent in employees positive responses regarding managers communication about agency goals, and from 44 to 49 percent regarding the amount of information received. We compared NTSB employees responses to those of employees from a group of small agencies and found that NTSB employees satisfaction level was about the same or more positive depending on the question. NTSB officials continue to monitor employees views about communication to address any remaining concerns.
 Diversity management . NTSB employees positive responses to the federal employee survey questions about managers commitment to diversity and NTSBs diversity policies and programs increased from about 54 percent to over 70 percent from 2004 to 2012. However, employees positive responses to the question about managers ability to work well with employees with different backgrounds declined 6 percentage points over the same period. In addition, the proportion of minority and women employees in NTSBs workforce, including in its investigator staff, showed little appreciable change over the period 2008 to 2012. NTSBs workforce had a smaller proportion of some minority groups than the civilian labor force. NTSB officials are using results from their recent diversity survey to identify gaps in their diversity management efforts and to benchmark future progress. It is too soon to tell whether NTSBs actions will lead to additional changes in its workforce diversity profile.
 Financial management . To improve operational effectiveness, NTSB has implemented a cost accounting system that includes a time and attendance program to track staff hours and costs related to accident investigations. NTSB is currently focused on ensuring the quality of the time and attendance data, but has not yet developed a strategy to maximize the utility of its cost accounting system for making resource and operational decisions. Thus, NTSB has not yet fully achieved its vision of using the data to improve labor productivity and mission effectiveness.
section_title: What GAO Recommends
paragraphs: In each of the five areas NTSB needs to continue its improvement efforts. Further, GAO recommends that NTSB senior managers develop a strategy for maximizing the utility of NTSB's cost accounting system. GAO provided a draft of this report to officials at NTSB. NTSB officials concurred with the recommendation and provided technical comments, which GAO incorporated as appropriate.

section_title: Letter
paragraphs: The National Transportation Safety Board (NTSB) plays a vital role in  advancing transportation safety by investigating every civil aviation  accident in the United States and selected accidents involving other  transportation modes, determining the probable causes of these  accidents, and making subsequent recommendations. NTSB also  develops studies that raise awareness of emerging safety issues. Since  2008, NTSB has investigated more than 1,100 aviation accidents and  dozens of accidents in other transportation modes, and issued over 1,000  recommendations to different agencies for improving safety. Most  recently, in 2012 NTSB launched 279 accident investigations and made  207 recommendations to improve transportation safety. NTSB is a  relatively small agency with a staff of about 400 employees and a fiscal  year 2013 budget of $104 million. In light of NTSB’s expansive mission to  investigate accidents across all transportation modes, it is critical that  NTSB manage its resources and operations as efficiently as possible.
In 2006, NTSB’s reauthorization legislation mandated GAO to annually  evaluate its programs. GAO conducted a series of annual reviews  covering NTSB’s management and operations, including examining the  extent to which NTSB (1) followed leading practices in selected  management areas, (2) addressed challenges in completing  investigations, and (3) generated sufficient revenues to cover its Training  Center costs. GAO’s reviews resulted in 21 recommendations to NTSB  aimed at improving management and operations across 12 areas. In  2012, we reported that all 21 of these recommendations had been  implemented and closed. Since then, NTSB has undertaken numerous  actions and continues to plan and implement changes in an effort to  improve its management and operations. However, measuring the  outcomes of NTSB’s actions is necessary to determine whether those  actions have led to measurable improvements and whether NTSB is  efficiently marshalling these efforts and using resources effectively. Our  objective in this review was to assess whether there has been  management and operational improvements associated with NTSB’s  actions. We considered recommendations within management and  operational areas of key importance to NTSB’s current priorities, as  stated in its 2013-2016 Strategic Plan and areas that required continuing  effort to achieve desired outcomes. This process yielded five  recommendations for our review. We then identified outcome measures  to assess changes by comparing current conditions with the initial  conditions at the time we performed our previous work, or earlier in some  instances, to establish a baseline. We used NTSB’s financial and  program data, employee survey data from the Office of Personnel  Management (OPM), and workforce data from NTSB and the Bureau of  Labor and Statistics (BLS). To ensure the data used were of sufficient  reliability for our analysis, we examined program reporting procedures  and quality assurance controls, and discussed various data elements with  knowledgeable agency officials. We also spoke with NTSB officials who  were knowledgeable about management and operations related to the 5  selected recommendations.
We conducted this performance audit from August 2012 through July  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. See Appendix I for a more  detailed explanation of our scope and methodology.
subsections: section_title: Background
paragraphs: NTSB was initially established within the newly formed Department of  Transportation (DOT) in 1966, but was made independent from DOT in  1974. NTSB is charged by Congress with investigating every civil  aviation accident in the United States and significant accidents in other  modes of transportation—railroad, highway, marine and pipeline. NTSB  determines the probable cause of the accidents and issues safety  recommendations aimed at preventing future accidents. In addition,  NTSB carries out special studies concerning transportation safety and  coordinates the resources of the federal government and other  organizations to provide assistance to victims and their family members  impacted by major transportation disasters. Unlike regulatory  transportation agencies, such as the Federal Aviation Administration,  NTSB does not have the authority to promulgate regulations to promote  safety, but instead makes recommendations in its accident reports and  safety studies to agencies that have such regulatory authority.
NTSB is comprised of a five-person board—a chairman, vice chairman,  and three members—appointed by the President with the advice and  consent of the Senate. The chairman is the NTSB’s chief executive and  administrative officer. The agency is headquartered in Washington, D.C.,  and maintains 7 regional offices and a training center located in Ashburn,  Virginia. In fiscal year 2013, the board was supported by a staff of about  400, which includes nearly 140 investigators assigned to its modal  offices—aviation; highway; marine; and rail, pipeline, and hazardous  materials—as well as 73 investigation-related employees, such as  engineers and meteorologists. NTSB’s modal offices vary in size in  relation to the number of investigators, with the Aviation Safety office  being the largest. In addition, the Office of Research and Engineering  provides technical, laboratory, analytical, and engineering support for the  modal offices. Staff from this office interpret information from flight data  recorders, create accident computer simulations, and publish general  safety studies.
This review focuses on the extent to which NTSB has achieved  measurable improvements from actions the agency has taken in five  management and operational areas based on prior GAO  recommendations.
Training Center utilization. Making efficient and effective use of  resources provided by Congress is a key responsibility of federal  agencies. NTSB’s Training Center, which opened in August 2003 in  Ashburn, Virginia, consists of classrooms, offices, and laboratory  facilities used for instructional purposes and active investigations.  NTSB uses this center to train its own staff and others from the  transportation community to improve accident investigation  techniques. NTSB charges tuition for those outside NTSB to take its  courses, and generates additional revenue from space rentals to other  organizations for events such as conferences on a cost reimbursable  basis. Although there is no statutory requirement that NTSB cover the  cost of its Training Center through the revenues generated from the  facility, a 2007 review we conducted found that NTSB was not  capitalizing on its lease flexibility to generate additional revenues and  classrooms were significantly underutilized. For example, we found  that less than 10 percent of the available classroom capacity was  used in fiscal years 2005 and 2006. Furthermore, NTSB was  encouraged in a Senate report accompanying the 2006 appropriations  bill for DOT to be more aggressive in imposing and collecting fees to  cover the costs of the Training Center. Since then, NTSB leased a  large portion of the Training Center’s non-classroom space to the  Federal Air Marshall Service and provided short-term leases of  classroom space to other organizations. In addition, NTSB increased  the amount of training it delivered at the Training Center.
Recommendation close-out process. Efficiently managing the  recommendation tracking process is a key function, according to  NTSB officials. The recommendation close-out process is managed  by the Safety Recommendations and Quality Assurance Division,  which has responsibility for tracking the status of its  recommendations. When NTSB receives correspondence from an  agency about an NTSB recommendation, this division ensures it is  properly routed and reviewed and contacts the agency about whether  the response is acceptable. If NTSB is delayed in communicating with  agencies about whether NTSB considers actions to address  recommendations acceptable, that agency could delay implementing  a course of action pending approval. In fiscal year 2010, NTSB  replaced a lengthy, paper-based process with an automated system— the Correspondence, Notation, and Safety Recommendation system  (CNS)—intended to facilitate the recommendation close out process  by electronically storing and automatically routing agencies’ proposals  to the appropriate NTSB reviewers, allowing for concurrent reviews by  multiple parties within NTSB, and more accurately tracking responses.  It is important to note that an agency is not necessarily restricted from  implementing action prior to formal NTSB approval of that action.  Depending on the complexity of the issue, agencies may begin to  address issues prior to NTSB’s providing formal approval. In other  circumstances, NTSB addresses safety deficiencies immediately  before the completion of an investigation. For example, during the  course of the TWA flight 800 investigation, NTSB issued an urgent  safety recommendation once it was determined that an explosion in a  fuel tank caused the breakup of the aircraft.
Communication. Useful management practices include seeking and  monitoring employee attitudes, encouraging two-way communication  between employees and management, and incorporating employee  feedback into new policies and procedures. This type of  communication and collaboration across offices at all levels can  improve an agency’s ability to carry out its mission by providing  opportunities to share best practices and helping to ensure that any  needed input is provided in a timely manner. To this end, NTSB  managers and board members began holding periodic meetings with  staff, conducting outreach to regional offices, and surveying staff  about the effectiveness of communication techniques.
Diversity management. Implementing a diversity management  strategy and a more diverse workforce helps foster a work  environment that not only empowers and motivates people to  contribute to mission but also provides accountability and fairness for  all employees. Diversity management helps an organization create  and maintain a positive work environment where the similarities and  differences of individuals are valued, so that all can reach their  potential and maximize their contributions to the organization’s  strategic goals. NTSB has developed diversity training courses and  held events to educate staff on diversity and inclusiveness issues,  created career development and mentoring programs to create  upward mobility, targeted its recruitment program to reach a more  diverse pool of applicants, and surveyed staff to assess the  effectiveness of its efforts.
Financial management. Sound financial management is crucial for  responsible stewardship of federal resources. Traditionally,  government financial systems and government managers have  focused on tracking how agencies spend their budgets but did not  focus on assessing the costs of activities to achieve efficiencies. More  recently however, some government agencies have adopted cost  accounting systems that track the cost of providing a service—in  NTSB’s case, an accident investigation. In 2006, GAO recommended  that NTSB develop a cost accounting system to track the amount of  time employees spend on each investigation and other activities. This  approach allows management to link the cost of providing a service  directly with the budget and allocate resources based on those costs.  To determine the costs associated with conducting accident  investigations, NTSB launched a time and attendance program tied to  its cost accounting platform that allows the agency to collect and  analyze labor and certain other costs associated with individual  investigations. Investigators account for their time on investigations  through the time and attendance system using specific codes that  identify different investigations.
section_title: Results across NTSB Management Areas
paragraphs: Our analysis found varying degrees of improvement associated with  NTSB’s actions in each of the management and operational areas we  selected for review.
subsections: section_title: Training Center Utilization
paragraphs: Our analysis showed that NTSB improved the utilization of the Training  Center, which allowed it to recover a larger portion of its operating costs.  NTSB increased utilization of both classroom and non-classroom space  at the Training Center since we conducted our work in 2006. NTSB  subleased all available office space at the Training Center to the Federal  Air Marshal Service in 2007, and utilization of non-classroom spaces has  been at 95 percent since then. At the same time, NTSB increased  utilization of the classroom space, increasing its own use of classrooms,  subleasing approximately one-third of the classroom space to the  Department of Homeland Security in 2008, and providing short term  leases to other outside parties for classroom use. Subsequently, NTSB  reported classroom utilization rose from less than 10 percent in 2005 to  18 percent in fiscal year 2007. By fiscal year 2009, it had increased to  over 60 percent—the target we identified in our 2008 report as the  appropriate minimum level. Classroom utilization has remained above  60 percent through fiscal year 2012.
We also found that improved Training Center utilization generated  additional revenue over time, which allowed NTSB to recover a larger  portion of the facility’s operating costs. When the Training Center first  opened in fiscal year 2004, NTSB recovered about 4 percent of its  operating costs, resulting in a deficit of nearly $6.3 million. Portions of the  Training Center’s costs that are not covered by revenues from tuition and  other sources such as facility rentals are offset by general appropriations  to the agency; therefore, generating additional revenue makes those  appropriated funds available for other uses. In 2011, NTSB indicated that  it was committed to improving cost recovery at the Training Center. That  year the agency set a goal to recover costs of the Training Center within  10 percent of the previous fiscal year. For example, in fiscal year 2010,  NTSB recovered $2 million in operating costs, making the fiscal year  2011 goal to recover at least $1.8 million of the Training Center’s costs.  NTSB achieved its goal in fiscal years 2011 and 2012 by which time the  agency was recovering about half of the Training Center’s operating  costs, reducing the operating deficit at the Training Center to $2.1 million,  one-third of what it was in 2004. (See fig. 1 for changes in the Training  Center’s expenses and revenues.)
The automation of NTSB’s recommendation follow-up process has  reduced the amount of time it takes to formally respond to agencies about  whether planned actions to implement an NTSB recommendation are  acceptable. In fiscal year 2010, NTSB deployed the previously  described CNS to manage the Board’s correspondence, including  accident reports, safety studies, recommendation transmittals, and public  notice responses. CNS allows for the relevant modal offices and the  Research and Engineering Office to simultaneously review and assess  planned actions to address NTSB recommendations. According to NTSB  officials, prior to the implementation of CNS, the average time NTSB took  to respond to an agency’s proposals to address an NTSB  recommendation was 216 days. After CNS was implemented, that figure  dropped to 115 days—a reduction of 47 percent. At the same time, the  number of responses the agency put out each quarter also increased.  (See fig. 2.) NTSB officials have indicated that they have an internal goal  to further reduce the response time to 90 days on average.
Our analysis of improvements in NTSB’s employee and management  communication related to NTSB’s efforts indicated uneven results;  specifically we observed improvements in some but not all measures. We  reviewed NTSB employees’ responses to the three federal survey  questions we determined related to employees’ perceptions about  managers’ communication, as described below:
Managers communicate the goals and priorities of the organization.
NTSB respondents increased their positive responses to this  question, from about 49 percent in 2004 to 57 percent in 2012. (See  fig. 3.) We compared NTSB employees’ responses with employees in  a group of small federal and independent agencies and found that  NTSB employees’ satisfaction level increased while the proportion of  employees from small agencies responding positively to this question  during the same period was relatively unchanged from 57 percent in  2004 to 59 percent in 2012.
How satisfied are you with the information you receive from  management on what’s going on in your organization? Responses to  this question indicated an increase in the level of satisfaction, from 44  percent in 2004 to 49 percent in 2012. NTSB employees’ responses  were similar to the positive responses by employees from small  agencies that showed an increase from 44 percent to 50 percent.
Managers promote communication among different work units (for  example, about projects, goals, needed resources). The proportion of  respondents reporting positive responses on this survey question from  2004 to 2012 was relatively unchanged from 48 percent to 50 percent.  Similarly, there was little change in the proportion of positive  responses reported by federal employees from small agencies from  50 percent in 2004 and 49 percent in 2012.
NTSB officials stated that their internal communication surveys, which the  agency administered 2009 through 2011, provided information that  helped them identify continuing barriers to employee and management  communication. In 2012, NTSB developed an action plan in this area that  included detailed activities, target dates, and regular status reports.  Furthermore, because of lingering concerns, NTSB continues to monitor  employees’ views about employee and management communication to  address any remaining weaknesses.
section_title: Diversity Management
paragraphs: Our analysis of outcomes associated with NTSB’s efforts to improve its  diversity management program indicated uneven results, with indications  of improvements in some measures but not all. We reviewed NTSB  employees’ responses to the three federal survey questions that we  determined related to employees’ perceptions about managers’ diversity  and inclusiveness efforts, as described below:
My supervisor/team leader is committed to a workforce representative  of all segments of society. The federal survey of NTSB employees  indicated an increase in the positive responses, from 54 percent to  about 71 percent. (See fig. 4.) We compared NTSB employees’  responses to those of employees in a group of small federal and  independent agencies and found that NTSB employees were more  positive in 2012 than the employees from small agencies, whose  positive responses rose from 57 percent in 2004 to 69 percent in  2012.
Policies and programs promote diversity in the workplace. Again,  NTSB employees’ responses to this question indicated an increase  from 2004 to 2012, from 55 percent to 73 percent. NTSB employees’  satisfaction level exceeded that of employees from small agencies,  whose positive responses to this question stayed about the same,  from 56 percent in 2004 to 57 percent in 2012.
Managers/supervisors/team leaders work well with employees of  different backgrounds. The proportion of NTSB employees reporting  positive responses on this survey question from 2004 to 2012  declined from 66 percent to 60 percent. The decline in NTSB  employees’ level of satisfaction was greater than that shown by  employees from small agencies, whose positive responses also  declined from 66 percent in 2004 to 63 percent in 2012.
One of the potential outcomes of a robust diversity management program  is an increase in the diversity of the workforce. Based on our analysis of  NTSB’s workforce diversity data, we found that the proportion of white  employees in NTSB’s workforce declined from 77 percent in 2008 (289  employees) to 73 percent (293 employees) in 2012. (See table 1.)  NTSB’s total workforce increased 6 percent over the same period, from  378 in 2008 to 402 in fiscal year 2012. The proportion of women  remained roughly the same at about 40 percent, as did the proportion of  African American employees at 17 percent, and Hispanic employees at 2  percent of the total NTSB workforce. Conversely, NTSB increased the  number of employees who are Native American although these  employees represent only 2 percent of the overall workforce. We  compared these figures to those representing comparative groups in the  civilian labor force and found that NTSB’s labor force had a larger  proportion of some minority groups (e.g., African American) and smaller  proportion of other groups (e.g., Hispanic) than the civilian labor force.
Roughly half of NTSB’s workforce performs investigations and  investigation-related work—work directly related to NTSB’s core mission.  In 2008 there were 190 investigators and related staff; in 2010, there  were 198, and in 2012 there were 206. Based on our analysis, we found  that the proportion of investigator and investigation-related staff that were  white was about 90 percent over the period of 2008 to 2012 and the  proportion of women was about 19 percent. We compared these figures  to those representing comparative groups in the civilian labor force and  found that NTSB’s investigator and investigation-related workforce had a  smaller proportion of minority and other groups, including African  American, Asian, Hispanic, and women than the civilian labor force. In  addition, NTSB reported that from fiscal years 2008 to 2012, it had no  minority group members among its 15 senior executives although the  number of women increased from 3 to 4.
Despite its efforts, NTSB has not been able to appreciably change its  diversity profile for minority group members and women. However, as  mentioned previously, NTSB has taken steps to implement initiatives as a  result of its diversity management strategy, including its recently  completed diversity and inclusiveness survey, which the agency plans to  use to identify gaps in its diversity and inclusiveness efforts and to  benchmark future progress. It is too soon to tell whether initiatives, such  as its recruitment strategies, will lead to additional changes in its  workforce diversity profile.
section_title: Financial Management
paragraphs: We were unable to determine if NTSB’s cost accounting system had  improved the agency’s ability to make operational decisions because it  has not yet fully utilized the system for its intended purposes. For the  implementation of a cost accounting system to be effective, it must be  tailored to the needs of the organization, be a tool managers can use to  make everyday decisions, and be based on sound data that captures time  spent on all activities, such as investigations and training. In 2011, NTSB  implemented a cost accounting system that includes a time and  attendance program in response to a GAO recommendation. This  program allows an investigator to assign his or her time to specific  investigations through a series of codes, allowing NTSB to assess the  cost of conducting investigations rather than simply tracking and  managing a budget. NTSB officials stated that the time and attendance  system has allowed it to obtain information about the cost of  investigations more efficiently than its prior method.
In a May 2011 advisory, NTSB management envisioned that the cost  accounting system would enable NTSB to measure and compare  performance with other organizations, and the data from the system  would help the agency monitor and improve productivity and mission  effectiveness by better utilizing personnel resources. However, officials  provided no time frame for when the data might be used by management  for making resource and operational decisions. NTSB officials stated that  they are currently focused on ensuring the quality of the time and  attendance data before developing goals, targets, and management tools  or using such information to make resource or operational decisions.  While ensuring the quality of data is a necessary step in fully  implementing a cost accounting system, it has been over 2 years since  NTSB first began collecting time and attendance data to establish the  cost of conducting investigations. As a result, NTSB is not using the  system’s full capabilities. Thus, NTSB has not yet fully achieved its vision  of using the data to improve labor productivity and mission effectiveness.
section_title: Conclusion
paragraphs: NTSB has implemented a cost accounting system, but effective utilization  is required to achieve the long-term rewards of those efforts. Although  NTSB has been collecting data from this system to account for costs of  investigations, it has not yet developed a management strategy that  would allow the agency to maximize the utility of the cost accounting  system. This has prevented NTSB from using that information to make  the decisions necessary to better manage its labor resources. Without  fully utilizing the cost accounting system, NTSB will not achieve the  intended benefits of improving labor productivity and mission  effectiveness.
section_title: Recommendation
paragraphs: NTSB needs to continue its improvement efforts in each of the five areas  discussed in this report. Further, to improve financial management and  provide information to managers for operational decisions, we  recommend that the Chairman of the NTSB direct senior management to  develop a strategy for maximizing the utility of NTSB’s cost accounting  system.
section_title: Agency Comments
paragraphs: We provided a draft of this report to NTSB for its review and comment.   The agency provided written comments (see app. II). NTSB agreed with  our recommendation and provided technical clarifications that we  incorporated as appropriate.
We are sending copies of this report to the appropriate congressional  committees and to the Chairman of the National Transportation Safety  Board. In addition, the report is available at no charge on the GAO  website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-2834. Contact points for Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made key contributions to this report are listed in appendix  III.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective in this review was to assess whether there have been  management and operational improvements associated with the National  Transportation Safety Board’s (NTSB) actions in areas where GAO had  conducted previous work and made recommendations. In that previous  work, GAO made 21 recommendations within 12 management and  operational areas. Some areas have more than one recommendation.  For example, the communication area has two related recommendations;  the first called for NTSB to develop mechanisms to facilitate  communication from staff to management and the other called for NTSB  to report to Congress on the status of the GAO recommendations.
To determine whether there were any associated improvements based on  NTSB’s actions, we first identified a subset of the 12 original management  and operational areas that were included in NTSB’s 2013-2016 Strategic  Plan. Being highlighted in NTSB’s current strategic plan indicates that  these areas were relevant and important areas for the agency. We  identified 7 of the 12 original management and operational areas in  NTSB’s current strategic plan. These 7 areas contained 13 of the 21  original recommendations. Many of the 13 recommendations within these  areas required only a single action by NTSB to implement (e.g. develop a  strategic plan that follows performance based practices, articulate risk- based criteria for selecting which accidents to investigate, or obtain  authority to use appropriations to make lease payments in order to correct  its violation of the Anti-Deficiency Act) while other of those  recommendations included strategies involving continual action and  monitoring overtime in order to achieve desired improvements (e.g.,  increased utilization of the training center and improve the  recommendation close out process). It is these latter types of  recommendations that we focused on in this review, and based on these  criteria, we identified 5 for review:  increase the utilization of the Training Center,  improve the process for changing the status of recommendations  through computerization and concurrent review, develop mechanisms to facilitate communication from staff to develop strategies for diversity management as part of the human develop a full cost accounting system to track time employees spend  on each investigation in training.
We identified a sixth recommendation—maximize the delivery of core  curriculum for each mode at the Training Center—based on our selection  criteria. However, we did not include this recommendation in our review  because NTSB lacked information about employee training and we could  not identify an outcome measure without such data.
To identify the outcome measures to assess changes associated with  NTSB’s actions, we used information from prior GAO reports, information  from NTSB including strategic plans, workforce reports, and financial  reports as well as information gathered from interviews with NTSB  officials. The measures we identified for each recommendation are: (1)  Training Center utilization—utilization of classroom and non-classroom  space and operating deficit; (2) recommendation close-out process— average time to respond to agency proposals; (3) employee and  management communication—employee responses to Office of  Personnel Management’s (OPM) federal employee surveys; (4) diversity  management—employee responses to OPM’s federal employee surveys  and NTSB employment levels of women and members of racial and  ethnic groups; and (5) financial management—cost accounting reports  used to measure performance.
To measure whether there was improvement in the outcomes and results  associated with NTSB’s actions, we compared current conditions with the  initial conditions at the time we performed our previous work or earlier in  some instances in order to establish a baseline before actions occurred.  We used NTSB’s financial and program data, employee survey data from  OPM, and workforce data from NTSB and the Bureau of Labor and  Statistics (BLS). To ensure the data used were of sufficient reliability for  our analysis, we examined program reporting procedures and quality  assurance controls, and discussed various data elements with  knowledgeable agency officials. We also spoke with NTSB officials who  were knowledgeable about operations and management in these five  selected areas.
section_title: Appendix II: Comments from the National Transportation Safety Board
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, H. Brandon Haller (Assistant  Director), Christopher Jones, Gail Marnik, Josh Ormond, Amy  Rosewarne, and Jack Warner made key contributions to this report.
section_title: Related GAO Products
paragraphs: National Transportation Safety Board: Implementation of GAO  Recommendations. GAO-12-306R. Washington, D.C.: January 6, 2012  National Transportation Safety Board: Issues related to the 2010  Reauthorization. GAO-10-366T. Washington, D.C.: January 27, 2010  National Transportation Safety Board: Reauthorization Provides an  Opportunity to Focus on Implementing Leading Management Practices  and Addressing Human Capital and Training Center Issues.  GAO-10-183T. Washington, D.C.: October 29, 2009  National Transportation Safety Board: Progress Made in Management  Practices, Investigation Priorities, Training Center Use, and Information  Security, But These Areas Continue to Need Improvement.  GAO-08-652T. Washington, D.C.: April 23, 2008  National Transportation Safety Board: Observations on the Draft  Business Plan for NTSB’s Training Center. GAO-07-886R. Washington,  D.C.: June 14, 2007  National Transportation Safety Board: Progress Made, Yet Management  Practices, Investigation Priorities, and Training Center Use Should Be  Improved. GAO-07-118. Washington D.C.: November 22, 2006.