You are given two reports by the government agency.  

Report 1:
Justice Management Division (JMD) JMD provides the Federal Bureau of Prisons senior management with guidance as it relates to Department of Justice (DOJ) policy for all matters pertaining to organization, management, and administration, including the use of human capital flexibilities such as retention incentives. BOP is responsible for incarcerating all federal offenders sentenced to prison. To carry out its mission, BOP, under the oversight of DOJ’s JMD, manages the human resource operations of its institutions, including the use of retention incentives. BOP administers, monitors, and oversees retention incentives through its Central Office, regional offices, and institutions. Central Office. The Central Office serves as BOP’s headquarters and provides oversight of BOP operations and program areas. Within the Central Office is BOP’s Human Resource Management Division (HRMD) which is responsible for developing, implementing and administering human resource policies and programs, including the use of retention incentives that meet OPM and DOJ requirements. In addition, the Central Office’s Program Review Division (PRD) is responsible for assessing BOP programs, including human resources, to ensure that they are managed and operated effectively. Regional offices. BOP has six regional offices that cover the Mid- Atlantic, North Central, Northeast, South Central, Southeast, and Western regions of the United States. These offices, each led by a regional director, oversee the operations of the 122 federal institutions within their respective geographic regions of the country. According to BOP officials, regional office staff also provide local level oversight of institutions’ human capital programs, such as retention incentives, among other things. Institutions. BOP institutions are managed by a warden and other officials, including an executive assistant and associate warden who generally provide overall direction and, in part, administer the institution’s human capital policies, including policies on retention incentives. Correctional services staff represent the largest segment of each institution’s workforce and are responsible for the correctional treatment, custody, and supervision of inmates. Non-correctional services staff include, among others, those employees assigned to non-correctional services management, facility operations, and the health services unit. Workers in health services and psychology services are responsible for providing inmates with medical, dental, and mental health services and include, for example, dentists, pharmacists, physicians, nurses, psychologists, and drug treatment specialists. The Federal Employees Pay Comparability Act of 1990 first authorized OPM to allow federal agencies to give incentives, including retention incentives, to employees. The Federal Workforce Flexibility Act of 2004 provided federal agencies increased flexibilities regarding these incentives. For example, individual retention incentives that were capped at 25 percent of an employee’s basic pay rate could be increased up to 50 percent in cases of critical agency need with OPM’s approval. Generally, under OPM regulations, an agency is authorized to pay a retention incentive to employees. This happens when the agency determines that the unusually high or unique qualifications of the employee or a special need of the agency for the employee’s services makes it essential to retain the employee and that the employee would be likely to leave federal service in the absence of an incentive. In addition, OPM requires agencies to develop plans for using retention incentives outlining, in part, the required documentation for justifying the retention incentive and any criteria for determining the amount of incentive and the length of the service period. Generally, agencies must require that employees sign a written service agreement that outlines the terms of the service such as the employee’s agreement to remain a certain length of time with the agency. Additionally, according to OPM regulations, to qualify for a retention incentive, each employee must have a performance rating of at least “fully successful” or an agency’s equivalent performance rating. BOP funds the majority of its retention incentives through its Salaries and Expenses appropriation account which represented almost 93 percent of BOP’s budget in FY 2016. According to BOP officials, BOP’s Central Office allocates funding from the Salaries and Expenses account to the regional offices. These regional offices then determine how to allocate their budget among various salary and expense activities, including retention incentives. HRMD delegates retention incentive determinations to each institution. In accordance with OPM requirements and BOP’s October 2016 Program Statement on Compensation, the wardens make retention incentive requests based on documented evidence that the employee possesses unusually high or unique qualifications or meets a special need of the agency and has a performance rating of at least “successful or its equivalent.” These incentives are calculated as a percentage of the employee’s basic pay and are disbursed in installments to the employee each pay period. In addition to retention incentives, BOP has authority to provide other compensation-based human capital flexibilities to employees, in certain circumstances. The following summarizes some of the compensation- based human capital flexibilities that BOP uses in addition to retention incentives, to retain and recruit staff: Recruitment and relocation incentives. BOP pays recruitment incentives to new hires and relocation incentives to current employees who elect to move to a different geographic area, when a position is likely to be difficult to fill in the absence of an incentive. Student loan repayments. Using this authority, BOP may repay federally-insured student loans to attract job candidates or retain current employees. Special salary rates. With OPM approval, BOP may establish higher rates of pay for an occupation or group of occupations nationwide or in a local area when it finds the government’s recruitment or retention efforts are, or would likely become, significantly handicapped without those higher rates. Physicians and dental comparability allowances. Comparability allowances may be paid to certain eligible physicians or dental professionals who enter into service agreements. These allowances are paid only to categories of physicians and dentists for which the agency is experiencing recruitment and retention problems and are fixed at the minimum amounts necessary to deal with such problems. BOP retention incentive expenditures generally increased from $10.7 million in fiscal year 2012 to $14.0 million in fiscal year 2016. Additionally, as illustrated in table 1, the number of employees who received retention incentives increased each year from 2,024 employees in fiscal year 2012 to 2,460 employees in fiscal year 2016. In general, BOP employees who received retention incentives received the incentive for more than one year. For example, from fiscal year 2012 through fiscal year 2016, a total of 3,382 BOP employees received retention incentive payments. Of those, 82 percent (2,766 of 3,382) received retention incentive payments for at least 2 years and 39 percent received retention incentives all 5 years, as shown in figure 1. From fiscal years 2012 through 2016, BOP spent more than 97 percent of its total retention incentive expenditures on employees at four California institutions and for medical professionals nationwide. BOP’s total retention incentive expenditures for the four California institutions and medical professionals nationwide in fiscal year 2016 are provided in figure 2. Four California Institutions. The California institutions—United States Penitentiary (USP) Atwater, Federal Correctional Institution (FCI) Herlong, FCI Mendota, and Federal Correctional Complex (FCC) Victorville—constituted the largest portion of BOP’s total retention incentive expenditures, and the level of their expenditures remained relatively steady from fiscal year 2012 through 2016. BOP provides group retention incentives for staff at the General Schedule (GS) grades level 12 and below and those in the Federal Wage System at three institutions—USP Atwater, FCI Herlong, and FCC Victorville. BOP also provides individual retention incentives to its employees at GS grades level 12 and below and in the Federal Wage System at FCI Mendota. As shown in figure 3, our analysis of BOP data found that from fiscal years 2012 through 2016, these four California institutions had the largest percentage of retention incentive expenditures across institutions as well as the largest percentage of employees who received retention incentives. Additionally, the four California institutions’ retention incentive expenditures remained relatively steady—around $8.1 to $8.2 million during the 5-year period—even though the overall number of employees who received the incentives generally increased. BOP officials told us that these California institutions’ retention incentive expenditures remained relatively steady in spite of an overall increase in the number of employees receiving incentives, in part, because in fiscal year 2013 BOP reduced the retention incentive rate—the percentage of an employee’s basic pay that determines the employee’s retention incentive— by 3 percent at the four California institutions. BOP officials reported using retention incentives primarily at these four institutions to supplement correctional officers’ salaries and compensate for the gap between BOP’s and other institutions’ salaries. Specifically, officials told us that these four California institutions were consistently understaffed as a result of their lower salaries in comparison to salaries offered at California state and local prisons and at other BOP institutions in California metropolitan areas. The Department of Labor’s Bureau of Labor Statistics reports that the average salary for correctional officers in California in 2016 was $70,020. For the same year, the annual average salary for BOP correctional officers at these four institutions was $50,859. To bring these four California institutions’ salaries in line with those offered by state, local, and other BOP institutions in California metropolitan areas, BOP officials told us that they first use recruitment incentives to attract and hire staff and then provide retention incentives to employees with a performance rating of at least “successful.” Medical Professionals. From fiscal years 2012 through 2016, BOP retention incentive expenditures for medical professionals increased by an average of approximately 21 percent per year. Our analysis showed that most recently—for fiscal years 2015 and 2016—BOP retention incentive expenditures for medical professionals accounted for the largest portion of BOP’s total retention incentive expenditures across the various occupation groups and was primarily responsible for the overall increase in BOP’s total retention incentive expenditures from fiscal year 2012 through fiscal year 2016. For example, in fiscal year 2016, BOP spent approximately 42 percent of total retention incentives expenditures for medical professionals ($5.8 million), 27 percent on correctional officers ($3.8 million), and the remaining 31 percent on employees in other occupations. In total, BOP retention incentive expenditures for medical professionals increased from approximately $2.7 million in fiscal year 2012 to $5.8 million in fiscal year 2016, as shown in figure 4. The increase accounted for 92 percent of BOP’s total increase in retention incentive expenditures during the five-year period. In comparison, BOP’s retention incentive expenditures for correctional officers and all other occupations remained relatively steady from fiscal year 2012 through fiscal year 2016, increasing by an average of approximately 1 percent per year. According to our analysis, the increase in retention incentive expenditures for medical professionals during the five years is partially explained by the increase in the number of institutions providing retention incentives to medical professionals. Specifically, from fiscal years 2012 through 2016, the number of institutions providing retention incentives to medical professionals increased from 53 institutions with 341 employees in medical occupations receiving retention incentives to 84 institutions providing retention incentives to a total of 646 employees in medical occupations. According to BOP officials, BOP primarily uses retention incentives for medical professionals in an effort to retain these employees by supplementing BOP salaries which are generally lower than salaries offered to medical professionals in the private sector. Officials told us that BOP has designated medical professional positions as hard-to-fill and, therefore, BOP retaining these professionals in a correctional setting requires the use of a variety of incentives, including retention incentives, in order to increase pay. BOP has a number of internal controls in place to ensure that retention incentive applications meet BOP and other requirements. BOP officials told us that these controls are part of a multilayered application and review process that begins at the institution and culminates at BOP’s Central Office. Our review of a random sample of 40 application packet case files for retention incentives awarded from fiscal year 2014 through fiscal year 2016 found that they all generally incorporated the internal controls described by officials. The key controls in this process include: Application review at the institution and regional levels. According to BOP officials, the retention incentive application process begins with an institution’s human resources office, whose staff complete a retention incentive application on behalf of an employee. The institution’s human resources office verifies that the information in the application justifies a retention incentive and that funds are available to pay the incentive. Although it is not required, BOP officials said that they use a retention incentive application checklist to help institutions ensure that retention incentive applications are complete. The institution’s human resources office then submits the completed application packet, which includes supporting documentation, to the warden for review. Next, the application packet is forwarded to the respective BOP regional director who also reviews it for accuracy and completeness. The regional director then adds an approval statement and forwards the packet to the Central Office for final review and approval. Of the 40 randomly selected application packet case files that we reviewed, 36 included a retention incentive checklist used by the institutions and all contained information to justify the retention incentive as well as a statement of the regional director’s approval. Central Office’s final application approval. BOP policy requires that all retention incentive applications undergo two levels of review in BOP’s Central Office: first by the Human Resource Management Division’s (HRMD) Staffing and Employee Relations Section (SERS) and next by HRMD’s Personnel Director, for final review and approval. According to BOP officials, during the review process there is ongoing communication between the various entities to ensure that applications are complete and accurate; for example, if SERS finds an error in the application or requests additional information, SERS returns the application to the regional or institutional level for correction and re-review. All of the 40 BOP application packet case files that we reviewed included approvals by HRMD’s Personnel Director or an authorized official, as required by BOP policy. Annual review and re-certification to continue retention incentives. According to BOP policy, on an annual basis, institutions’ human resources offices are required to review employees’ retention incentives to determine whether the incentive is still warranted. Payment of a retention incentive may be recertified and continued as long as the conditions giving rise to the original determination to pay the incentive still exist and funds are available. For each retention incentive, an institution’s human resources office must determine whether to continue, adjust, or terminate the incentive within one year of the initial or most recent approval. If the human resources office decides to continue the retention incentive, the institution’s warden must again submit a retention incentive application. Applications to continue the retention incentive proceed through the same review and approval process as initial applications. Of the 40 application files that we reviewed, 29 were continuations and 8 were initial requests for a retention incentive. According to BOP officials, after the initial approval of a retention incentive, an institution’s human resources office has primary responsibility for the monitoring of retention incentive payments. According to officials, institutions use a variety of internal controls to monitor the expiration, continuation, or termination of retention incentives, for example: Monitoring expiration dates. BOP officials stated that institutions’ human resources offices monitor retention incentives in order to identify incentives that are approaching their expiration date and need to be terminated or renewed. For example, according to BOP officials from USP Atwater, FCC Butner and FCI Phoenix, staff from their institutions’ human resources offices may generate a retention incentive activity report and cross reference this report with their locally generated tracking sheets. This process helps identify retention incentives approaching their expiration dates so that the human resources offices can submit a request for continuation before the incentive expires. Using automated reminders to prompt file review. BOP officials stated that institutions use automated reminders to alert human resources staff to check the records of retention incentive recipients for human resources-related events such as promotions or relocations that could affect the continuation of a retention incentive. Following a checklist of steps for relocation processes. BOP officials told us that in April 2016 they instituted a checklist that outlines steps that an institution’s human resources staff must take when employees relocate to a different institution. Based on our review of this checklist, one step on the sheet prompts human resources staff to review the employee’s retention incentive. According to BOP policy, when an employee receiving a retention incentive transfers to another location, the human resources office where the employee was receiving the retention incentive is responsible for submitting a request to terminate the incentive. The termination must be effective the last day of the pay period that the employee occupies the position. Submitting forgiveness waivers. BOP officials told us that institutions submit forgiveness waivers if a request to continue a retention incentive is not submitted and approved prior to the retention incentive expiring. BOP officials said that a forgiveness waiver is considered an acknowledgement of an administrative error and is a late submission of a retention incentive renewal that was still warranted. The waiver is not a request to forgive an overpayment since the employee was still considered to be eligible for the retention incentive. Of the 40 retention incentive applications that we reviewed, 5 applications included forgiveness waivers to excuse the tardiness of the filing and request continuations of the retention incentive. According to BOP officials, BOP conducts periodic audits and reviews of its human capital activities and related internal controls, to ensure that retention incentives are being used appropriately. The following offices conduct various audits and reviews involving BOP’s retention incentives: BOP’s Program Review Division (PRD) audits regional and institutional human resources functions. PRD audits BOP’s regional and institutional human resources offices to ensure that they are in compliance with BOP policies and procedures. According to BOP officials, as part of the audit process, PRD audits retention incentives to ensure that they have the proper approvals and are justified. PRD audits each institution’s human resources office at least every three years. During these audits, PRD generates retention incentive activity reports (the same reports that institutions run when monitoring for expiration dates), to check the accuracy of retention incentive programs under review. Following each audit, PRD issues a final report with findings to the institution and to the staff operating the program area under audit. Institutions respond to the report with corrective actions that the institution will take to address the findings. When the institution has resolved all corrective actions from the audit, the audit is closed. Additionally, each quarter, PRD provides HRMD with a report that summarizes its quarterly audit findings. According to BOP officials, HRMD uses these reports to identify any agency-wide trends that need to be addressed. Our review of BOP data showed that between fiscal years 2012 and 2016, PRD conducted nearly 200 audits. For example, in the fourth quarter of fiscal year 2016, PRD audited five institutions’ and regional offices’ human resource management functions. During these audits, PRD identified nine deficiencies, one of which pertained to retention incentives. Specifically, it found that one audited institution did not terminate an employee’s retention incentive after the employee had relocated to another institution. To correct the deficiency, the institution cancelled the retention incentive which discontinued future disbursements. According to BOP officials, a bill was generated to recoup the overpayment from the employee. BOP institutions conduct annual operation reviews of internal functions, such as human resources. BOP officials told us that each institution conducts annual operational reviews of various internal functions, such as human resources. According to BOP’s Program Review Guidelines for Human Resource Servicing Offices, during these reviews, institutions are required to review supporting documentation for staff currently receiving an incentive to determine if the incentives are still warranted. If the initial request for the retention incentive was made over the preceding 12 months, institutions are also required to ensure that it was approved. According to BOP officials, the results of these reviews are reported to PRD through the Central Office. DOJ’s Justice Management Division (JMD) audits BOP’s human resources programs. According to BOP officials, JMD conducts audits of component-level human resources programs to determine whether BOP’s systems are compliant with DOJ policy and aligned with DOJ’s Human Capital Strategic Plan. JMD’s most recent audit of BOP’s human resources programs that included a review of BOP’s retention incentives occurred in September 2010 at BOP’s Human Resource Service Center in Grand Prairie, Texas. JMD found that in some cases BOP granted retention incentives prior to the signing of service agreements. JMD also found that BOP lacked documentation to authorize a group retention incentive for employees at its Victorville, California institution. BOP’s written response to the findings stated that JMD incorrectly applied the service agreement requirement, as service agreements were not warranted in the specific case that it identified. Additionally, BOP stated that the documents JMD identified as missing from the case files in question were kept in separate files and not required to be part of the retention incentive application. JMD agreed with BOP’s responses and in January 2013, JMD closed out the audit’s findings noting that these responses satisfied all required corrective actions. While BOP takes a number of steps to determine current workforce needs and how to fill those needs, BOP does not strategically plan for how retention incentives can be used to meet long-term human capital goals. BOP officials stated that planning for human capital needs is conducted at institutions during quarterly workforce utilization meetings or manpower salary meetings. During these meetings, executive staff at the institution discuss the current state of the institution’s workforce. According to the BOP officials, while considering attrition, hiring, and turnover rates, the executive staff decide strategies they will employ to attract and retain employees for their current needs. While officials we spoke with at four institutions have discussed retention incentives at their workforce utilization meetings, details about the content of these discussions ranged. According to these officials and our review of meeting minutes from the four institutions, discussions about retention incentives respond to each institution’s short-term staffing situation rather than address future staffing needs based on an overall strategic human capital plan. For example: USP Atwater officials told us that they review the current turnover rate, budget, projected vacancies, and use of retention incentives at annual budget development meetings. Meeting minutes reflected the following on retention incentives: “retention … still necessary to retain staff and hard-to-fill positions.” FCC Butner is a medical facility that offers retention incentives to all medical officers (all types of doctors) and nurses (practitioners, registered, etc.) at the institution. According to Butner officials, during workforce utilization meetings, Butner officials discuss recruitment and staffing trends for the institution and plans for how to address any staffing challenges. Meeting minutes we reviewed did not indicate specific discussions about the use of retention incentives. FCC Pollock executive staff discuss current institutional salary expenditures and projections and the status of vacant positions at workforce utilization meetings. While meeting minutes we reviewed indicated discussions about projected expenditures for incentive awards, the minutes did not differentiate between retention incentive awards, and other incentive awards such as recruitment or relocation incentive awards. FCI Phoenix officials stated that in their workforce utilization meetings, executive staff discuss salary projections and vacancy statuses. Meeting minutes we reviewed did not indicate specific discussions about the use of retention incentives. BOP decisions about retention incentives are currently not tied to any strategic human capital plan for how to use human capital flexibilities— such as retention incentives—to address their ongoing challenge of retaining staff in hard-to-fill positions. According to officials, retention incentives are awarded on an as-needed basis, determined by an institution’s warden, if funds are available. According to key principles for effective strategic human capital planning, such planning is an important component of an agency’s effort to develop long-term strategies for acquiring, developing, and retaining staff needed for an agency to achieve its goals. Specifically, senior leaders should be involved in developing, communicating, and implementing strategic human capital plans. Within an agency’s strategic human capital plan, the human capital policies, practices, and programs—for example, an agency’s retention incentive program—should clearly link to the human capital and program goals of the organization. By not having a strategic human capital plan that clearly establishes strategies that will be used to achieve specific human capital goals, BOP cannot ensure that its institutions are strategically managing their workforces in a manner that meets the agency’s human capital needs. In August 2017, BOP officials told us that they began drafting a strategic human capital operating plan that will include strategic objectives, action plans, performance objectives and measures, and evaluation/reporting requirements. Officials stated that the plan will also include planning regarding the use of human capital flexibilities, such as retention incentives. BOP officials told us that they anticipate that the strategic human capital operating plan will be a supplement to their workforce utilization meetings and that an agency-wide plan will provide a set of strategies for all institutions to consider. However, BOP could not provide documentation of the project beginning or whether it would include a strategic approach specific to retention incentives. Including retention incentives in BOP’s strategic human capital operating plan would create a roadmap for the agency and the institutions to use to move from being reactive in their current workforce needs—for example, awarding retention incentives on an ad hoc basis when funds are available—to being strategic in how retention incentives are used and to ensure that these and other flexibilities help the agency achieve its long-term workforce goals. From fiscal year 2012 through fiscal year 2016, BOP spent more than $59 million on retention incentives but has not established any measures to evaluate their effectiveness. According to officials, BOP has not evaluated the effectiveness of its use of retention incentives because BOP officials consider a retention incentive successful if an employee does not leave the agency. However, BOP also uses other human capital flexibilities along with retention incentives to help retain staff. For example, BOP uses physician and dental comparability allowances—additional pay to a physician or dentist who enters into an agreement for a specified period of service—to help retain these medical personnel. According to officials, it would otherwise be difficult to compete with private sector salaries without the use of all available incentives. However, BOP has not studied whether or how retention incentives have contributed to employees’ retention in relation to other incentives such as physician and dental comparability allowances. According to our work on strategic human capital management and OPM’s guidance, it is crucial for organizations to evaluate the success of their human capital strategies, such as the use of retention incentives. In measuring the performance of these strategies and their contribution to key programmatic results, agencies can make adjustments, if necessary. For example, agencies can use evaluation results to make targeted investments in certain human capital strategies—such as the use of retention incentives—creating a cycle of strategic workforce management, where evaluation informs planning, planning dictates strategies, and strategies are evaluated for effectiveness. While BOP uses retention incentives to address critical skills gaps—such as with medical professionals—evaluating the effectiveness of retention incentives would help BOP determine whether and how retention incentives, as well as other human capital flexibilities, contribute to an employee’s continued employment at BOP or if adjustments to BOP retention strategies must be made for improved results. BOP officials agreed that evaluating the effectiveness of retention incentives would help them be more strategic about their human capital needs and spending on incentives. By including and implementing such an evaluation in its upcoming strategic human capital operating plan, BOP could better determine if it is making maximum use of its funds to retain the necessary qualified personnel or if changes must be made to most effectively retain its staff. As the largest employer within DOJ with some staff working in remote locations and undesirable conditions, BOP relies on a number of available flexibilities, including retention incentives, to help retain its employees. However, BOP currently lacks a strategic approach for using and evaluating retention incentives to address human capital goals. Given BOP’s ongoing staffing challenges, for example, retaining staff in hard-to- fill medical positions, developing a plan that includes a thoughtful blueprint for using retention incentives could help BOP better anticipate and address staffing needs. Moreover, evaluating its use of retention incentives could help BOP determine whether these incentives are effective or whether adjustments are needed to better retain its employees. By using evaluation results to inform planning, and planning to inform how retention incentives are used, BOP would be better positioned to achieve its long-term human capital goals and address its critical staffing needs. We are making two recommendations to BOP: 1. The Director of BOP should include in the forthcoming strategic human capital operating plan, 1) human capital goals and 2) strategies on how human capital flexibilities—including retention incentives—will be used to meet these goals. (Recommendation 1) 2. The Director of BOP should evaluate the effectiveness of BOP’s use of retention incentives to help determine whether the incentives have helped BOP achieve its human capital goals or if adjustments in retention incentives are needed. (Recommendation 2) We requested comments on a draft of this report from DOJ. In an email received November 15, 2017, the DOJ liaison stated that DOJ concurred with our recommendations. The Department did not provide official written comments to include in our report, but did provide written technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Attorney General and the Director of BOP. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9627 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in Appendix III. This report examines (1) how BOP has used its authority to pay retention incentives; (2) what internal controls are in place for the use of retention incentives; and (3) the extent to which BOP plans for and evaluates the use of retention incentives. To determine how BOP has used its authority to pay retention incentives, we reviewed BOP’s July 2012 report on its use of recruitment, relocation, and retention (3R) incentives. We then obtained underlying retention incentive expenditure data from DOJ’s Justice Management Division because it serves as the focal point for performance and financial information for all Department of Justice components and employees, including BOP. In particular, we obtained employee-level retention incentive payroll data for fiscal years 2012 through 2016. We selected this time period because it includes the most recent five complete fiscal years for which data were available and because we believe five years is sufficient time to identify trends in BOP’s retention incentive expenditures. We analyzed and aggregated the employee-level data by institution, occupation, and employee grade level. To identify trends, we compared per fiscal year expenditures across the various categories of occupations and locations across the five years. Additionally, we categorized institutions by BOP region, institutions that use group retention incentives, and institutions that use individual retention incentives. We also categorized occupations as medical professionals, correctional officers, and all other occupations and compared aggregate retention incentive expenditures for the different groups. Using information from BOP’s website and testimonial evidence from BOP officials on its health care system, for the purposes of this report, we defined medical professionals as BOP employees in occupations that provide medical, dental, and mental health care services and who do not solely provide these services in an administrative function. For the purposes of our analyses, medical professionals are dentists, dental assistants and hygienists, diagnostic radiological technologists, health aid and technicians, medical doctors (including psychiatrists), medical technologists, nurses, pharmacists, pharmacy technicians, physician assistants, and practical nurses and psychologists. To assess the employee-level retention incentive payroll data’s reliability, we obtained and analyzed documentation on systems’ capabilities and data control, interviewed data users and managers responsible for maintaining data, conducted checks for completeness and logical consistency, and compared the employee-level data to aggregated institution-level retention incentive expenditure data from BOP’s Financial Management Information System. We found the employee-level data to be sufficiently reliable for the purpose of this report. Additionally for this objective, we reviewed documents such as the DOJ’s Financial Management Information System Sub-Object Classification Code Guide and the Office of Personnel Management (OPM) Handbook of Occupational Groups and Families to respectively identify the system codes used to track retention incentives expenditures and to identify the names for each occupational series code in the datasets. We also interviewed BOP Human Resource Management headquarters officials to obtain information on the primary purposes for BOP’s use of retention incentives and their views on identified retention incentive expenditures trends. We also interviewed U.S. Department of Health and Human Services’ (HHS) Public Health Service (PHS) officials to better understand how BOP and PHS manage costs, including retention incentive expenditures, for PHS staff assigned to BOP. BOP partners with PHS to acquire medical staff to provide medical care for BOP’s inmate population. BOP reimburses PHS for the costs of compensation and benefits—including retention incentive payments, if applicable—for PHS staff assigned to BOP. PHS has final approval authority for retention incentives paid to PHS staff assigned to BOP facilities. Furthermore, we obtained aggregated retention incentive expenditure data from PHS on the total amount of funds BOP reimbursed PHS for fiscal years 2012 through 2016. For the reliability of PHS’s data, we reviewed the system’s data fields to check that the appropriate fields were used to provide data and interviewed data users and managers to discuss how expenditures are recorded and maintained. We found the PHS data to be sufficiently reliable for the purpose of this report. To identify and describe the internal controls that BOP has in place related to retention incentives, we obtained and analyzed documentation regarding BOP requirements and guidance for the use of retention incentives. We also interviewed officials from BOP’s Central Office who are responsible for the administration, management, and oversight of BOP’s human capital management systems, including retention incentives. We focused on the management and administrative controls used by BOP to review, approve, re-certify, and monitor retention incentives. Additionally, we interviewed the warden and human capital officers at 4 of the 122 institutions to obtain illustrative examples regarding the internal controls in place at these institutions to ensure the proper disbursement of retention incentives. We interviewed BOP officials at Federal Correctional Complex Pollock in Pollock, LA; Federal Correctional Complex Butner in Butner, NC; United States Penitentiary, Atwater in Atwater, CA and Federal Correctional Institution Phoenix, in Phoenix, AZ. These institutions were selected to ensure variation in the number and types of employees receiving retention incentives, BOP region, and security-level. Although the information we obtained from the interviews with officials at these four institutions cannot be generalized to other BOP institutions, these interviews provided important insights and perspectives about the use of retention incentives at BOP institutions. We also reviewed a non-generalizable random sample of 40 retention incentive application packet case files to determine the extent to which these files contained documentation on the internal control activities in place to monitor the application, approval, and funds disbursement processes of BOP’s retention incentive program. To identify our sample, we used employee-level expenditure data to randomly select 40 application files from the universe of BOP employees who received retention incentives from fiscal years 2014 through 2016. Each application file was reviewed by two GAO analysts who each assessed the extent to which each application contained the appropriate justification, approval signatures, and other documentation such as an application checklist and whether the application was an initial or continuation application. To determine the extent to which BOP plans for and evaluates the use of retention incentives, we interviewed BOP officials regarding their experiences with retention incentives, how they use retention incentives to strategically manage their workforce needs, how the agency evaluates the effectiveness of retention incentives, and how retention incentives contribute to BOP’s broader human capital goals. We then compared these efforts to our work on strategic human capital planning, specifically in terms of planning for and evaluating the use of human capital flexibilities. Additionally, we interviewed the warden and human capital officers at four BOP institutions mentioned above to obtain illustrative examples of how workforce planning occurs at these institutions. We also reviewed the DOJ’s Office of Inspector General Report 16-02 “Review of the Federal Bureau of Prisons’ Medical Staffing Challenges” (March 2016) and our past work to better understand the challenges that BOP faces in retaining medical professionals and other staff. Table 2 provides the Bureau of Prisons’ (BOP) fiscal year 2016 retention incentive expenditures by various occupations and groups of occupations, such as medical professionals, correctional officers, and other occupations. A range of occupations are reflected in the table primarily as a result of four California institutions—United States Penitentiary (USP) Atwater, Federal Correctional Institution (FCI) Herlong, FCI Mendota, and Federal Correctional Complex Victorville—providing retention incentives to all employees at General Schedule grades level 12 and below and those in the Federal Wage System. In addition to the contact named above, Dawn Locke (Assistant Director) and Meghan Squires (Analyst-in-Charge) managed the work. Also, David Alexander, Renee Caputo, Willie Commons III, Jamarla Edwards, Robert Goldenkoff, Chelsa Gurkin, Eric Hauswirth, Janice Latimer, Lerone Reid, Rachel Stoiko, and Adam Vogt made significant contributions to this report.

 

 Report 2:
Safety defect vehicle recalls (auto recalls) are initiated when a defect in a vehicle or vehicle equipment creates an unreasonable safety risk, as determined by NHTSA or a manufacturer. After a recall is initiated, manufacturers are required to provide written notification to vehicle owners via First-Class Mail within 60 days and remedy the defect. Franchised dealers—which sell or lease an auto manufacturer’s new vehicles—perform the recall remedy. Before manufacturers send recall notification letters to affected vehicle owners, NHTSA reviews draft letters and envelopes to ensure they include required information about the safety defect. Required information includes, among other things, a clear description of the safety defect, an evaluation of the risk to vehicle safety, and a statement that the manufacturer will remedy the defect without charge. See appendix II for an example of a notification letter. The number of vehicles affected by safety defect vehicle recalls has increased dramatically since 2011 (see fig. 1). The increase reflects, in part, several large-scale recalls. For example, in 2014, General Motors initiated a recall of over 8 million vehicles with faulty ignition switches. Similarly, according to NHTSA in 2014 and 2015, some passenger vehicle manufacturers began recalling Takata air bag inflators, recalls that have grown to include approximately 34 million vehicles and 19 auto manufacturers. For the Takata recall, NHTSA issued various orders and established a Coordinated Remedy Program under which the agency oversees the supply of remedy parts and risk-based prioritization of vehicles for repair, and manages related recalls with the assistance of an Independent Monitor. The Independent Monitor assesses compliance with the applicable orders issued by NHTSA and makes recommendations aimed at enhancing the remedy program. According to NHTSA’s Strategic Plan 2016–2020, this unprecedented recall activity encouraged the agency to improve its system for identifying and addressing defective vehicles. For example, the plan states that NHTSA’s “vision is to achieve a 100-percent completion rate for every recall by improving communication at every level, at every step of the way.” Thus, according to the plan, NHTSA and the auto industry have committed to identifying and implementing effective strategies to inform consumers of safety defects and envision that their coordination will bolster recall efforts to improve completion rates. NHTSA reported that annual completion rates for passenger vehicle recalls have remained relatively flat, ranging from 63 to 67 percent between calendar year 2011 and calendar year 2014. See appendix III for completion rates by vehicle component and vehicle type. In part, to improve communication and encourage consumers to complete repairs, NHTSA and manufacturers provide auto recall information to the public on their websites. For example, certain motor vehicle manufacturers are required to allow consumers to search a vehicle’s recall remedy status on the Internet using the vehicle identification number (VIN). NHTSA also provides publicly available auto recall information on its website, including examples of recall notification letters. In December 2016, NHTSA began consolidating its websites into NHTSA.gov to provide a single access point for its auto recall content. One of these websites, safercar.gov, was once NHTSA’s primary method of communicating auto recall information to consumers; however, the agency is in the process of moving this information to NHTSA.gov. NHTSA’s Strategic Plan 2016–2020 states that the agency wants NHTSA.gov to be a comprehensive user-friendly platform that serves as the premier source of vehicle safety information by, for example, improving the website’s search capabilities. NHTSA also aims to encourage consumers to use its website’s auto recall information through its communications program. NHTSA’s Office of Communications and Consumer Information (OCCI) is the primary office responsible for implementing the agency’s public communication efforts. OCCI intends to increase public engagement with the agency’s information through its social media channels, such as Instagram, Twitter, and Facebook. The amount OCCI obligated to support the agency’s auto recall efforts has increased from nearly $.5 million in fiscal year 2011 to about $2.5 million in fiscal year 2016. According to NHTSA officials, these obligations supported various efforts, including public awareness campaigns, an auto recall hotline, advertising agencies, exhibits at auto shows, and NHTSA’s mobile application. As part of our focus group discussion sessions, consumers selected safety risk and convenience as the two most influential factors they considered when using auto recall information to decide whether to complete repairs. All factors considered: During each session, we first asked consumers to describe all the factors they considered. Across the sessions, consumers shared a wide variety of factors including availability of a loaner vehicle, time to schedule and complete the repair, safety risk, and other factors. For example, some consumers had not yet repaired their vehicles because they were “just waiting” for parts to become available. Other consumers considered their previous customer service experiences at the franchised dealership or the distance they would need to travel to complete the repair. For example, one consumer at our rural focus group location told us it would take roughly 2 hours to reach the dealership’s repair shop. Most influential factors considered: After the discussion of all factors, we then asked each consumer to select the single most influential factor they considered. Consumers in the sessions overwhelmingly selected safety risk and convenience as the two most influential factors (see table 1). More than half of consumers in our focus group discussion sessions selected safety risk as the most influential factor they considered when making repair decisions. They told us that their perception of the risk influenced whether or not they repaired their vehicle. For instance, some consumers stated that they completed repairs immediately, because the risks “sounded serious” or that they considered the defect a “safety concern.” Conversely, some consumers said they did not complete the repairs because the defect “didn’t sound very urgent.” While each recall notification letter is required to include an evaluation of the risk to vehicle safety reasonably related to the defect, consumers in our focus group sessions shared mixed opinions about the quality and clarity of safety risk information included in the notification letter they received. For example, some consumers told us the letter’s safety risk information seemed vague. For instance, one consumer told us the letter’s description of the safety defect did not clearly state the chances of an increased risk of injury and so he “had to figure out on his own.” In addition, some consumers commented that the safety risk information could be more prominent in the notification letter, that the letter could emphasize the severity of the risk, or that the letter could describe the risk in simpler language. However, other consumers stated the notification letter they received adequately described the recall’s safety risk. In June 2011, we recommended that NHTSA modify the requirements for defect notification letters to include additional information to obtain readers’ attention. In 2013, NHTSA responded to our recommendation by requiring manufacturers to include the statement “IMPORTANT SAFETY RECALL” at the top of auto recall notification letters. Focus Group Participant’s Comment “I don't want to be without a car for half the day or stay with my kids all day.” Consumers in our focus group discussion sessions selected convenience as the second most influential factor they considered in making repair decisions. While some consumers described the “hassle” of the repair and being “too busy” to schedule and fix the defect, other consumers told us they repaired their vehicles more easily because, for example, they could take advantage of previously scheduled service appointments to also repair the defect. Also, some consumers in our sessions stated that the letter or notification they received could better address the inconvenience of the recall, for example by including better estimates of how long repairs might take. In addition, some consumers recommended the letter include options for scheduling needed repairs. As we discuss later in the report, NHTSA officials told us they continue to work with auto manufacturers to identify ways to encourage consumers to complete needed repairs, while representatives from some manufacturers we met with described specific steps they have taken to address some of the inconveniences consumers may experience in completing repairs. For example, one manufacturer facilitated a pilot program for a third-party service provider in conjunction with dealers to repair vehicles at the owner’s home or place of work, while another manufacturer told us they work with individual dealers to hold events specifically for recall repairs when consumers can come in to have repairs performed after normal business hours. Industry stakeholders’ use of auto recall information varies because these stakeholders play different roles in the auto recall process. Auto manufacturers are primarily responsible for providing auto recall information to the public and others, including NHTSA and auto dealers. Franchised dealers are responsible for performing the recall remedy for manufacturers and therefore use manufacturer-provided information for that purpose. Specifically, all of the franchised dealers we interviewed told us they identify recalls on new vehicles in their inventory primarily by accessing auto recall information through internal manufacturer databases. These franchised dealers may also use information from third-party providers or publicly available auto recall information on NHTSA’s website to identify recalls affecting used vehicles. Independent dealers—which are not generally authorized by manufacturers to perform recall remedies—may use publicly available auto recall information to identify open recalls. Specifically, 2 of the 3 independent dealers we met with told us they use NHTSA’s VIN look-up tool to search for open recalls affecting vehicles in their inventory before selling them to consumers. However, these dealers told us that the current design of the tool takes too much time to use, because it requires users to search each VIN individually. For example, one dealer told us each search took about 15 seconds to perform, resulting in significant time and cost because the dealership has tens of thousands of vehicles in its inventory. These dealers told us being able to search multiple VINs in a single search (i.e., VIN-batch search) could save them time or money. Representatives from the Alliance of Automobile Manufacturers stated they—in coordination with other industry stakeholders—are working with a third-party provider to develop a search tool that would address this concern by enabling VIN-batch searches for use by government agencies, such as state departments of motor vehicles, and commercial entities. The group anticipates the tool will be available in the first half of 2018. Although the vast majority of consumers who participated in our focus group discussion sessions reported a preference to receive auto recall notification by mail, most preferred to receive notifications by at least one additional electronic means such as e-mail, phone calls, and text messages. Eighty of the 94 consumers in our sessions reported a preference for receiving notification by mail, and all but 4 reported actually receiving mailed notification (see fig. 2). However, 69 of the 94 consumers in our sessions also reported a preference for receiving recall notification by electronic means, but only 7 reported actually receiving at least one type of electronic notification. This result suggests a gap between industry recall notification practices and notification preferences for most consumers in our focus groups, especially for younger consumers who were more likely to report a preference for notification by electronic means. For complete results of the questionnaire we administered to consumers for the discussion session, see appendix IV. As we discuss later in this report, in September 2016, NHTSA issued a Notice of Proposed Rulemaking (NPRM) that proposes to require auto manufacturers to notify consumers about auto recalls by electronic means in addition to First-Class Mail. NHTSA officials told us the agency is working with the administration on NHTSA’s regulatory portfolio and priorities, including this rulemaking. Some manufacturers told us they use additional methods to reach consumers, including notifying consumers by electronic means and translating recall information into Spanish. For example, representatives from one manufacturer told us they always notify consumers by e-mail before sending out the required First-Class Mail letter notification. These representatives told us using multiple recall notification means resulted in higher recall completion rates. In addition, eight of the remaining nine manufacturers told us they use supplemental electronic means notification on a case-by-case basis—generally using additional means to improve recall completion rates—while four manufacturers stated they consider safety risk severity when deciding when or how to use additional notification means for individual recalls. Also, representatives from 3 of the 10 manufacturers we spoke with told us they translate the entire mailed notification letter into Spanish. In late 2016, NHTSA launched its redesigned NHTSA.gov website, including the auto recall areas consumers assessed during our testing sessions. According to responses to a questionnaire we administered during our testing sessions, 78 of the 94 consumers found the auto recall areas of NHTSA.gov either “somewhat” or “very easy” to use (see fig. 3). See appendix V for complete participant responses to the questionnaire we administered to each consumer. To inform the development of the redesigned website, NHTSA worked with a contractor to conduct a usability study in 2015 to evaluate users’ reactions to the agency’s websites, including NHTSA.gov. According to agency officials, NHTSA implemented several changes based on the findings from the usability study, including: the creation of a dedicated “recalls” area of NHTSA.gov, and the ability for users to access the VIN look-up tool in three different ways—on the homepage, in the “recalls” area, and through direct links either in a NHTSA e-mail for subscribers or from an external website. In addition, NHTSA officials told us that Department of Transportation (DOT) and NHTSA staff meet as needed to discuss the website and consider improvements. For example, the officials said they monitor user searches for the relevance and accuracy of results and adjust the search software to better assist users in finding auto recall information. Officials also told us the agency collects a variety of other information about how visitors use NHTSA.gov, including how visitors access the website, and makes adjustments accordingly. For instance, NHTSA incorporated responsive web design as part of the agency’s ongoing consolidation effort—meaning the site is optimized for viewing on desktop, tablet, and mobile devices. In addition to monitoring searches and how visitors access NHTSA.gov, NHTSA officials told us they collect and consider online survey data to make website improvements and use web-analytic software to monitor, for example, where visitors choose to exit the website. Officials stated that such monitoring activities have allowed NHTSA to identify and correct problems with NHTSA.gov. We did not directly evaluate the accessibility of the auto recall areas of NHTSA.gov to ensure the ability of people with physical or mental disabilities to use the website. However, NHTSA officials provided us with an overview of several steps the agency takes to ensure NHTSA.gov complies with website accessibility requirements. For example, according to officials, NHTSA subscribes to a service that provides monthly accessibility scans of the agency’s websites. While most consumers in our usability testing sessions generally found the auto recall areas of NHTSA’s website easy to use, some consumers experienced difficulties completing tasks we asked them to perform (see table 2). Specifically, during each testing session we asked participants to perform tasks using the primary means NHTSA.gov provides for consumers to access information about auto recalls affecting their vehicles: searching for auto recalls using their vehicle’s VIN; searching for auto recalls using their vehicle’s year, make, and model; and locating NHTSA’s auto recall notification e-mail subscription service. In addition, an evaluation we requested to corroborate the results of our consumer usability testing, identified similar issues. As discussed below, consumers experienced these difficulties because the auto recall areas of NHTSA.gov do not always reflect federal and industry key website usability practices, which describe standards and guidelines for making websites easy to use. Following such practices can assist agencies in creating quality websites while providing the flexibility necessary to meet organizational needs. Website usability is particularly important for agencies, such as NHTSA, that are responsible for conveying safety information to the public. Federal standards for internal control state that agencies should communicate quality information externally and select appropriate methods for communicating with the public. While most consumers in our usability testing sessions found searching for recalls by VIN somewhat or very easy, some consumers found the search results did not provide the information they were seeking. When we asked consumers to perform VIN searches, they generally found the VIN look-up tool easy to use—88 of 94 consumers found searching with a VIN either somewhat or very easy. But some consumers experienced difficulties performing this task. Specifically, some consumers who had had their vehicles repaired expected to find the completed recall on the search results page. However, they were confused because the page is designed to display only open (i.e., unrepaired) recalls, not completed (i.e., repaired) recalls—leading these consumers to question the accuracy of the results. In addition, the evaluation conducted by website usability professionals found that, when an error occurred during a VIN search, the error message was too difficult to locate on the search results page. The evaluation recommended the error message have greater weight and more prominence on the page. Federal key website usability practices state that agencies should ensure that results of user searches provide the precise information being sought, and in a format matching users’ expectations. When users are confused by search results, or do not immediately find what they are searching for, they become frustrated and may abandon the search or the website entirely. Since NHTSA launched the VIN look-up tool in August 2014, the number of VIN searches performed has increased (see fig. 4). According to NHTSA officials, major increases occurred in mid-2015— when the Takata air bag inflator recalls were announced—and in early 2017, when NHTSA made the VIN look-up tool search function available on NHTSA.gov and displayed it prominently on the website. Ensuring the usability of NHTSA’s VIN look-up tool is particularly important because it is the only way on NHTSA.gov for a consumer to determine whether their specific vehicle has an open safety recall. Recall Search Using Vehicle Year, Make, and Model Some consumers’ vehicle year, make, and model searches were hampered by the information required to conduct an accurate search, as the content on the website is not always in plain language. We asked consumers to perform a recall search using their vehicles’ year, make, and model, and 78 of 94 consumers found the task to be either somewhat or very easy. However, some consumers found that they did not know enough information about their specific vehicles to feel confident that they were searching for the correct vehicle. For example, a year, make, and model search for a 2009 Toyota Tacoma may ask the consumer to choose among vehicle options, including “2009 TOYOTA TACOMA REGULAR CAB W/SAB RWD/AWD.” Acronyms such as “W/SAB”— which stands for “with side air bags”—may be confusing to consumers. Federal key website usability practices state that federal agencies should write website content using plain language, so website visitors can easily find and use what they need. Focus Group Participant’s Comment “I think [the Recall Notification E-mail System Sign-Up is] poorly placed. I had to scroll to find it. I had to search for it. You want at the top .” Recall Notification E-mail System Sign-Up Some consumers suggested improvements to make the Recall Notification E-mail System Sign-Up easier to locate on the homepage. NHTSA first made its Recall Notification E-mail System Sign-Up available in March 2008. Of the 94 consumers in our testing sessions, 66 found it either “somewhat” or “very easy” to find the Recall Notification E-mail System Sign-Up—making this the least easy of the three tasks we asked consumers to perform. Specifically, several consumers said the Recall Notification E-mail System Sign-Up should include a clearer description, be easier to find, and be located at the top of the homepage (see fig. 5). These improvements are particularly important because some consumers in our focus group sessions told us that the ability to sign up for auto recall e-mail notifications was the most useful part of the auto recall areas of NHTSA.gov. The website evaluation conducted by website usability professionals recommended that NHTSA streamline its homepage with more of a focus on primary website tasks. The evaluation also found that users must move through too many pages to sign up for recall e-mails. Federal key website design and usability practices state that agencies should put important items closer to the top of the page, where users can better locate the information. Key practices also state that agencies should design their websites so users can successfully complete the most common tasks in the fewest number of steps. The website usability difficulties that consumers in our focus groups experienced may be due to the fact that NHTSA has not studied the website’s usability since the agency redesigned NHTSA.gov in late 2016 and, therefore, may have been unaware of these difficulties prior to our review. NHTSA plans to conduct a website usability study with consumers after the consolidation effort, discussed above, is complete. However, NHTSA could not provide a general time frame for conducting the study because it has not yet determined when the consolidation effort will be complete. We have previously reported that it is essential for organizations to effectively guide their information technology efforts by establishing timelines to complete them, among other strategic planning best practices. Without establishing a completion date for its website consolidation effort, the website usability difficulties we identified may persist and limit the effectiveness of NHTSA’s primary means of providing consumers with safety recall information about their vehicles on NHTSA.gov. In January 2016, NHTSA launched a national advertising campaign encouraging consumers to check for open recalls using the agency’s VIN and year, make, and model look-up tools. Through March 2017, NHTSA spent about $1 million on its Safe Cars Save Lives campaign, which sponsors advertisements on Google, Facebook, and other media platforms. For example, Google might place NHTSA’s advertisement above other search results, when a consumer typed certain keywords— such as “recall,” “airbag recalls,” or “safercar.gov”—into the search. NHTSA evaluated the campaign’s effectiveness by monitoring website traffic performance reports to determine how frequently consumers clicked on NHTSA-sponsored advertisements and ultimately searched for open recalls using the agency’s look-up tools. NHTSA also compared results across media platforms and adjusted the campaign’s strategy to improve performance. For example, NHTSA optimized advertisements on mobile devices, since mobile-device users performed more recall searches than other users. According to NHTSA data, the awareness campaign resulted in consumers performing 1.1 million recall searches through March 2017—a cost of about $0.90 per search. Agency data indicate that this cost generally decreased as NHTSA improved the campaign’s strategy. Agency officials told us NHTSA plans to spend another approximately $1.8 million on Safe Cars Save Lives from September 2017 through September 2018 due to the campaign’s effectiveness in raising the public’s awareness about auto recalls. NHTSA began implementing a mandated 2-year pilot grant program intended to evaluate the feasibility and effectiveness of informing consumers about open auto recalls during state vehicle registration. In September 2016, NHTSA solicited applications to participate in the program, wherein selected states would inform consumers—at no charge—about open recalls using all means that permit consumers to register vehicles within the state (e.g., in person, Internet, and mail). According to NHTSA, only one state applied for the grant. In September 2017, NHTSA awarded the sole applicant $223,000. Under the program, the grantee needs to collect and report program performance data, including the extent to which open recalls have been identified and repaired. In addition, the grantee must report whether certain notification means were more effective than others and what could be done to improve the program. Upon completion of the pilot program, NHTSA is required to evaluate the extent to which open recalls identified have been remedied. Auto manufacturers we met with were generally supportive of the program. Specifically, representatives from 9 of the 10 manufacturers told us that notifying consumers about open recalls during vehicle registration can raise consumer awareness or improve recall completion rates. In September 2016, NHTSA issued a Notice of Proposed Rulemaking (NPRM), which proposes to require auto manufacturers to notify consumers about open recalls by electronic means—such as e-mails, phone calls, and text messages, in addition to First-Class Mail. As we described earlier, auto manufacturers are currently required to notify consumers about safety recalls affecting their vehicles via First-Class Mail. According to NHTSA, the NPRM aims to aid in efficiently and effectively improving recall completion rates, by proposing that manufacturers provide notification using electronic means in addition to First-Class Mail. Consumers in our focus groups as well as auto manufacturers and consumer associations we interviewed generally supported additional notification using electronic means. Consumers in our focus groups: As we discussed earlier, 69 of the consumers in our focus group discussion sessions reported they would prefer to receive additional notification by at least one type of electronic means. However, only 7 consumers actually received such notifications—suggesting a gap between industry notification practices and notification preferences for these consumers. Auto manufacturers: Representatives from 9 of 10 manufacturers we interviewed told us they generally support providing notification using electronic means. Although the NPRM proposes a broad definition of electronic means to give manufacturers flexibility to determine the most effective means, these representatives also shared implementation concerns. For example, representatives from 1 of the 9 manufacturers told us that—although the company collects e- mail addresses from some customers for other purposes—not all customers provide e-mail addresses, and those collected are not always accurate. As we discussed previously, most manufacturers we met with currently use supplemental electronic means notification on a case-by-case basis. Consumer associations: Similarly, both consumer associations we interviewed told us additional electronic notification can help reach consumers who do not complete repairs after receiving initial mailed notification. NHTSA’s proposal would maintain manufacturer reporting requirements, though it may result in additional reporting. This additional information could help the agency evaluate the effectiveness of various means of consumer notification. We previously found that NHTSA may be able to use manufacturers’ data to identify what factors make some recalls more or less successful than others. We recommended that NHTSA use the recall data it collects to analyze particular patterns or trends that may characterize successful recalls and determine whether these factors represent best practices. If the NPRM is finalized, manufacturers would provide NHTSA with representative copies of the newly required electronic notifications, in addition to mailed notifications, and would specify the electronic means used, such as e-mail or text message. According to NHTSA officials, this information could allow the agency to track and evaluate the effectiveness of various notification means used by manufacturers by, for example, comparing completion rates across means—a key step in identifying best practices that could encourage consumers to complete repairs. However, it is too early for NHTSA to conduct such an evaluation, because the agency has not issued a final rule. NHTSA officials told us the agency is working with the administration on NHTSA’s regulatory portfolio and priorities, including this rulemaking. NHTSA has also taken steps to collaborate with industry stakeholders and explore consumer education best practices. For example, in April 2015 NHTSA hosted a day-long workshop that brought together auto industry stakeholders to examine public education of the recall process. During the workshop, participants identified current barriers to the public’s awareness of auto recalls and discussed potential solutions to address them, such as using text messages and social media to communicate with younger consumers and using different delivery methods for recall notices. Similarly, in January 2016 NHTSA and 18 auto manufacturers adopted a set of Proactive Safety Principles to explore and employ new ways to increase safety recall participation rates. For example, NHTSA and auto manufacturers agreed to share industry best practices and policies based on lessons learned from ongoing safety recalls. The Independent Monitor of Takata in conjunction with NHTSA has also issued a set of coordinated communications recommendations based on consumer research, best practices observed during the Takata recall, and discussions with manufacturers. For example, the recommendations encourage manufacturers to: pursue a “multi-touch” communications strategy that employs non- traditional means, such as e-mail and text messages; convey risk in clear, accurate and urgent terms; and include a clear “call to action” designed to facilitate prompt and efficient scheduling of repairs. According to NHTSA officials, the agency relies on auto manufacturers to evaluate the effectiveness of these efforts. However, agency officials told us NHTSA reviews manufacturers communication plans as part of the Takata recall’s Coordinated Remedy Program and provides ongoing recommendations on manufacturers’ communication language, approach, and strategies. With the recent steep increase in safety defect vehicle recalls and continued low recall completion rates, it is vital for consumers to be able to easily access and use publicly available auto recall information. NHTSA has taken important steps to improve its website—which provides safety recall information to consumers—resulting in most consumers in our focus groups finding the website easy to use. However, the difficulties some experienced in attempting to complete essential auto recall tasks demonstrated that NHTSA.gov does not always reflect key website usability practices for website design. Although NHTSA plans to conduct a website usability study with consumers after consolidating its websites, it has not determined a completion date for this effort—an essential step for organizations to effectively guide their information technology efforts. Without such a date, the website usability difficulties may persist and limit the effectiveness of NHTSA.gov in providing consumers with recall information about their vehicles. By addressing these difficulties in the interim, NHTSA can better assure that consumers obtain this information, which can be vital to their safety. We are making the following two recommendations to NHTSA: The Administrator of NHTSA should determine a completion date for the agency’s website consolidation effort. (Recommendation 1) The Administrator of NHTSA should, while the agency continues its website consolidation effort, take interim steps to improve the usability of the auto recall areas of NHTSA.gov by addressing the website usability difficulties we identified. (Recommendation 2) We provided a draft of this report to DOT for review and comment. In its written comments, reproduced in appendix VI, DOT stated that it concurred with our recommendations. The department also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to relevant congressional committees, the Secretary of Transportation, and the Administrator of NHTSA. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. This report examines the use of publicly available auto recall information for safety defects affecting passenger vehicles. The report addresses the following objectives: (1) How do consumers and industry stakeholders use publicly available auto recall information? (2) How easy or difficult to use do consumers find the auto recall areas of NHTSA.gov? (3) What steps, if any, has the National Highway Traffic Safety Administration (NHTSA) taken to raise consumer awareness about auto recalls and how has NHTSA evaluated the effectiveness of these steps? We define publicly available auto recall information to include information on the auto recall areas of NHTSA.gov, such as examples of notification letters that manufacturers mail to consumers. This report focuses on safety defect vehicle recalls affecting passenger vehicles that are initiated when a defect in a vehicle or vehicle equipment creates an unreasonable safety risk, as determined by NHTSA or a manufacturer. To determine how consumers use publicly available auto recall information, we conducted and analyzed transcripts and questionnaires from 12 consumer focus groups we conducted with used and new vehicle owners who had experienced an auto recall in the last 24 months. Each focus group was split into two sessions: (1) a discussion session to explore participants’ thoughts, experiences, and preferences about auto recall information and (2) a website usability testing session. Also, we administered a questionnaire as part of each of these sessions. For the discussion session, we asked consumers about the recall notification process and how they used the recall information, and for the website usability testing session, we asked consumers to fill in a questionnaire during the session itself as they assessed the usability of the auto recall areas of NHTSA’s website. We conducted the 12 focus groups at six locations across the country, with each group including 7 or 8 consumers for a total of 94 participants. Half of the focus groups were comprised of consumers who had completed the repair and the remaining half included consumers who had not completed the repair. We selected the six focus group locations to provide population and geographic dispersion. To ensure geographic dispersion, we selected at least one location in each U.S. Census region (see table 3). To ensure population dispersion, we selected Metropolitan Statistical Areas representing a range of population sizes based on 2015 U.S. Census estimates. To ensure our selection included the perspectives of vehicle owners in geographically distant or isolated communities, we also selected a rural location, which we defined as a city or town that has a population of less than 50,000 inhabitants and is not an urbanized area contiguous and adjacent to a city or town that has a population of greater than 50,000 inhabitants. Using information provided by the participants, we selected focus group participants based on age, income, gender, education level, race, and ethnicity to ensure we collected a range of perspectives on auto recall information use. However, since we did not select a representative sample of participants, focus group results are not generalizable to all vehicle owners. During focus group discussion sessions, we asked participants to discuss factors they considered when deciding whether to repair their recalled vehicle and then to select the single most influential factor. Each of the 12 focus group sessions was audio recorded and transcriptions were created; transcripts served as the record for each group. We then evaluated those transcripts using systematic content analysis to identify the factors consumers considered when deciding whether to complete repairs and any suggested improvements to the auto recall communication process. The analysis was conducted in three steps. First, two analysts independently developed a code framework and then worked together to resolve any discrepancies. Second, each transcript was coded independently by analysts using the framework and any discrepancies were resolved by both analysts agreeing on the coding of the associated statement by a participant. Third, if needed, another analyst adjudicated any continued disagreement between coders. Because the transcripts did not include a unique identifier for each focus group participant, we conducted our analysis of focus group session discussions at the group level (i.e., of the 12 focus groups we conducted). We also administered and analyzed a questionnaire as part of each discussion session to quantify responses regarding consumers’ use of auto recall information, including how they received and preferred to receive auto recall notifications. Our analysis of the questionnaire responses was conducted at the individual consumer level (i.e., of the 94 consumers who participated). These focus group sessions were structured, guided by a moderator who used a standardized list of questions to encourage participants to share their thoughts, experiences, and preferences. We also conducted two pretest focus groups at our headquarters and made some revisions to the focus group guide prior to beginning the sessions with consumers. Methodologically, focus groups are not designed to demonstrate the extent of a problem or to generalize results to a larger population or provide statistically representative samples or reliable quantitative estimates. Instead, they are intended to generate in-depth information about the reasons for the focus group participants’ thoughts, experiences, and preferences on specific topics. The projectability of the information produced by our focus group sessions is limited. For example, the information includes only the responses from the vehicle owners from the 12 selected groups and their individual responses to questions we asked. The experiences and preferences expressed may not reflect other vehicle owners’ thoughts and preferences. In addition, while the composition of the groups was designed to ensure a range of age and education levels, among the other criteria mentioned previously, the groups were not constructed using a random sampling method. To determine how industry stakeholders use auto recall information, we interviewed selected auto manufacturers, selected franchised and independent auto dealerships, NHTSA program officials, and other industry stakeholders. Specifically, we interviewed representatives from the following 10 auto manufacturers, selected based on each manufacturer’s sales market share (small, medium, and large), place of ownership (foreign and domestic), and experience with auto recalls (lower to higher based on the average annual number of auto recall campaigns and average market share of each manufacturer from 2010 to 2014) to collect a range of perspectives on how manufacturers use auto recall information: Tesla Motors, Inc. To understand the perspective of auto dealers, we interviewed four franchised dealerships, one in each of the four U.S. Census regions where we conducted focus groups with consumers. We also interviewed three independent auto dealerships in two U.S. Census regions. The results of these interviews are not generalizable to all auto manufacturers and dealerships, but provide insights about how some industry stakeholders use auto recall information. We conducted interviews with NHTSA program officials to understand NHTSA’s role in the auto recall process. In addition, we interviewed other stakeholders, including the Independent Monitor of Takata, which assists NHTSA in overseeing the Takata recall, as well as officials from consumer associations and other industry groups (see table 4). To evaluate how easy or difficult consumers find the auto recall areas of NHTSA.gov to use, we reviewed various website usability resources to understand federal and industry key website usability practices for making websites easy to use, such as focusing on design and how easily users can find information. In addition, we reviewed federal standards for internal control related to communicating quality information externally. During our usability testing sessions, we asked consumers to attempt to complete auto recall tasks—the primary means NHTSA.gov provides for consumers to access information about auto recalls affecting their vehicles—and discuss their experiences. We then compared consumers’ experiences with the usability of the website against these practices. To identify key website usability practices, we analyzed guidance documents from NHTSA and other federal agencies. For example, we analyzed the General Services Administration’s (GSA) and the Department of Health and Human Services’ Research-Based Web Design & Usability Guidelines, which includes quantified, peer-reviewed guidelines intended to help federal agencies improve the design and usability of their information-based websites. We also analyzed GSA’s Requirements for Federal Websites and Digital Services, and the U.S. Digital Services Playbook to identify key practices for making websites easy to use. Identified key practices are: (1) design and content— focusing on the layout, headers, and design; (2) navigation—how easily users can find information; (3) clarity—the ability to read and digest content; (4) identity and purpose—whether the site clearly presents its purpose; and (5) accessibility—the ability of people with physical or mental disabilities to use the site. To analyze the results of focus group website testing sessions, we performed a systematic content analysis of the session transcripts using the same content analysis methods described above and an analysis of the questionnaire we administered to each participant during the website usability sessions. Specifically, we analyzed the transcripts from the website usability testing sessions to account for consumers’ experiences, including their initial impressions of the website and any suggested usability improvements. We also analyzed the results of the questionnaire that each participant completed where participants were asked to mark responses regarding their experience including an assessment of the usability of the auto recall areas of NHTSA.gov. Our analysis of the results from the questionnaire responses was conducted at the individual consumer level (i.e., of the 94 consumers who participated) while our analysis of focus group session discussions was conducted at the group level (i.e., of the 12 focus groups we conducted). To corroborate the results of usability testing sessions we conducted with the consumers in our focus groups, we requested that five website usability professionals from GSA’s Federal User Experience Community conduct an independent evaluation of the auto recall areas of NHTSA.gov against federal and industry key website usability practices (described above). The website usability professionals developed a website usability evaluation form, which they used to individually evaluate the auto recall areas of NHTSA’s website. The website usability professionals then met to form a consensus and provided us with one final group evaluation of the website usability of the auto recall areas of NHTSA.gov. Also, although neither our usability testing nor the website usability evaluation conducted by website usability professionals directly addressed accessibility, we interviewed responsible agency officials about how the agency assesses the accessibility of NHTSA.gov. We also requested and analyzed website data provided by NHTSA to understand how consumers access and use NHTSA.gov. Requested data included the number of subscribers to NHTSA’s Recall Notification E-mail System Sign-up; the number of weekly vehicle identification number (VIN) searches performed on NHTSA.gov from August 2014 through May 2017; and NHTSA.gov usage data by device (i.e., usage by mobile devices, tablets, and desktop computers). We assessed the reliability of these data by reviewing any supporting documents provided by the agency and interviewing responsible NHTSA officials, and concluded the data were sufficiently reliable for our reporting purposes. While we did not independently review the usability of auto manufacturers’ auto recall websites, we requested and reviewed the results of any audits that NHTSA performed of these websites, including whether the websites met statutory and regulatory requirements for providing auto recall information to the public. We then corroborated any audit findings by reviewing the auto recall websites of the selected auto manufacturers that we interviewed. To determine any steps NHTSA has taken to raise consumer awareness about auto recalls and how NHTSA evaluates the effectiveness of any steps, we reviewed relevant statutes, regulations, and proposed rules, including the Fixing America’s Surface Transportation Act and a Notice of Proposed Rulemaking related to recall notification methods. We also reviewed agency and other documents that describe or evaluate NHTSA’s public awareness activities. For example, we analyzed NHTSA’s strategic planning documents—such as NHTSA’s Strategic Plan 2016–2020—to identify ongoing public awareness activities along with their related goals, objectives, or performance metrics. Similarly, we requested and analyzed any documents NHTSA uses to evaluate the effectiveness of its public awareness activities, including performance reports for NHTSA’s ongoing Safe Cars Save Lives campaign. To assess the reliability of data included in these performance reports— such as VIN searches performed—we reviewed agency documentation and interviewed agency officials about the reliability, accuracy, and completeness of the data and determined the data were sufficiently reliable for our reporting purposes. We reviewed performance management practices as provided in the Government Performance and Results Act of 1993 (GPRA), the GPRA Modernization Act of 2010, and standards for internal control in the federal government to identify any opportunities for improvement. We also performed a literature review to identify any related published articles and research studies. To understand how NHTSA implements and evaluates any public awareness activities, we also interviewed responsible agency officials from NHTSA’s Office of Communications and Consumer Information and other offices. In addition, we discussed NHTSA’s public awareness efforts during interviews with industry stakeholders, including selected auto manufacturers, selected franchised and independent auto dealerships, and other industry stakeholders. We analyzed the results of these interviews along with the focus group discussions we conducted with consumers (discussed above) to identify perspectives on the effectiveness of NHTSA’s public awareness steps. We conducted this performance audit from October 2016 to December 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The National Highway Traffic Safety Administration (NHTSA) is required to conduct a biennial analysis of vehicle safety recall completion rates and submit the results of its analysis in a report to certain congressional committees. The report must include, among other things, the annual recall completion rate by vehicle type and vehicle component (such as brakes, fuel systems, and air bags) for each of the 5 years preceding the year the report is submitted. According to NHTSA’s May 2017 report, completion rates for all vehicles combined ranged between 63 percent and 67 percent between calendar year 2011 and calendar year 2014 (see table 5). However, NHTSA reported wider variation when the recall completion rates are broken down by vehicle type. Similarly, the report found that completion rates for most component categories fall within a range of 60 percent to 75 percent (see table 6). The annual completion rate is a volume-based, weighted metric, such that the more vehicles affected by the recall, the more weight or influence it has on the computed rate. Focus group participants responded to a questionnaire we administered to collect information on consumers’ auto recall notification preferences during our discussion sessions. Table 7 shows participants’ responses to the administered questionnaire, by age group. We present these responses by age group, because consumers’ notification preferences may vary according to their ages. Focus group participants responded to a questionnaire we administered to collect information on the usability of NHTSA.gov during our usability testing sessions. Table 8 shows focus group participants’ responses to the administered questionnaire, by age group. We present these responses by age group, because consumers’ website usability needs or preferences may vary according to their ages. In addition to the individual named above, H. Brandon Haller (Assistant Director); Katherine Blair; Jason Blake; Melissa Bodeau; Alicia Cackley; William Colwell (Analyst in Charge); Lacey Coppage; Elizabeth Dretsch; Jaci Evans; Marcia Fernandez; Sarah Kaczmarek; Malika Rice; Todd Schartung; and Andrew Stavisky made key contributions to this report.

Question: Write a one-page summary of the report 1.

Answer: BOP is the largest employer within DOJ and is responsible for the care and custody of an inmate population of about 186,000. BOP has faced challenges retaining staff at correctional facilities, although it has used retention incentives, along with other human capital flexibilities. GAO was asked to review BOP's use of retention incentives. This report addresses: (1) how BOP used its authority to pay retention incentives; (2) internal controls BOP has in place for the use of retention incentives; and (3) the extent to which BOP plans for and evaluates the use of retention incentives. GAO obtained employee-level retention incentive expenditure data from DOJ's Justice Management Division for fiscal years 2012 through 2016. GAO also reviewed agency documentation, such as policy statements and 40 randomly selected retention incentive application packet case files from fiscal years 2014 through 2016. GAO also interviewed officials from BOP's Central Office and four correctional facilities that use retention incentives, selected to reflect variation in the number and types of employees receiving retention incentives, BOP regions, and BOP institution security levels. From fiscal years 2012 to 2016, the Department of Justice's (DOJ) Federal Bureau of Prisons' (BOP) total retention incentive expenditures generally increased from $10.7 to $14.0 million and the number of employees receiving retention incentives increased from 2,024 to 2,460. During those five years, BOP spent more than 97 percent of its total retention incentive expenditures on employees at four BOP institutions in California and for medical professionals nationwide. Further, total retention incentive expenditures for medical professionals increased by an average of 21 percent per year (see figure). According to BOP officials, BOP uses retention incentives, for example, to supplement BOP's medical professionals' salaries which are generally lower than private sector salaries. BOP has a variety of internal controls in place throughout the retention incentive process that help ensure retention incentive applications and approvals meet requirements. For example, each application goes through multiple levels of review to verify its accuracy and completeness. BOP takes steps to determine workforce needs and how to fill those needs, but has not strategically planned for and evaluated its use of retention incentives. According to BOP, planning for human capital needs is conducted at institutions during quarterly meetings, but discussions about these incentives respond to short-term staffing situations rather than proactively addressing future staffing needs. Including human capital goals and strategies in BOP's human capital plan would create a roadmap so the agency could move from being reactive to its current workforce needs to being strategic in trying to achieve its long-term workforce goals. Additionally BOP has not evaluated the effectiveness of its use of retention incentives in retaining staff. As a result, BOP does not know whether retention incentives have contributed to employees' retention in relation to other incentives used by BOP. Consistent with key principles for strategic human capital planning, planning for and evaluating the use of retention incentives could help BOP better determine if these incentives are an efficient and effective means by which to retain staff. GAO recommends that BOP (1) include human capital goals and how retention incentives will be used to achieve these goals in its human capital plan; and (2) evaluate the use of retention incentives. BOP concurred with GAO's recommendations.