Oil and Gas Oversight: Interior Has Taken Steps to Address Staff Hiring, Retention, and Training but Needs a More Evaluative and Collaborative Approach
Released: 2016-10-31T00:00:00

Letter

Background

BLM, BSEE, and BOEM are directly overseen by the Assistant Secretary for Land and Minerals Management, who is responsible for guiding Interior’s management and use of federal lands and waters and their associated mineral and nonmineral resources. In addition, human capital programs at the bureaus and elsewhere in the department are overseen by Interior’s Assistant Secretary of the Office of Policy, Management and Budget, which is broadly responsible for employee training and development; part of the office’s mission is providing high-quality, innovative, efficient, and effective training. The Office of Policy, Management and Budget comprises multiple offices, including the Office of Human Resources, which has primary responsibility for evaluating the effectiveness of Interior’s personnel management program, and the Office of Strategic Employee and Organization Development, which is responsible for delivering efficient and effective training across the department.

In fiscal year 2014, BLM, BSEE, and BOEM employed over 900 key oil and gas staff who oversee onshore and offshore oil and gas activities. Onshore land use planning is handled by BLM’s petroleum engineers, natural resource specialists, geologists, and other scientists. Offshore resource planning is handled by BOEM’s petroleum engineers, geoscientists, and other specialists. Operators that are awarded leases for oil and gas development can then submit to BLM (onshore) or BSEE (offshore) an application for a permit to drill. Petroleum engineers, inspectors, natural resource specialists, geologists, and other scientists review and approve applications for permits to drill. The application for permit to drill contains a detailed set of forms and documents that specify requirements that the operator must follow when drilling. Once operators’ planning for oil and gas operations commence, BLM and BSEE inspectors, petroleum engineers, and natural resource specialists carry out a variety of oil and gas inspections. For example, BLM’s inspectors conduct production inspections, drilling inspections, and environmental compliance inspections. Similarly, BSEE inspectors conduct drilling and production inspections to ensure that operators comply with all regulatory requirements. However, Interior and others have stated that offshore inspections in a marine environment are generally more complex and difficult than onshore inspections and require helicopters or boats to reach inspection sites, making the planning and performance of duties more difficult and hazardous. Further, offshore facilities have large amounts of equipment and personnel in relatively confined spaces, more sophisticated safety systems and requirements, and high production volumes, pressures, and temperatures, as well as more limited access to some equipment and piping, especially in deep water areas that are far from shore.

In addition to GAO, Interior’s Inspector General and the Outer Continental Shelf Safety Oversight Board have reported on Interior’s challenges related to hiring and retention of such key oil and gas staff. For example, Interior’s Inspector General concluded in December 2010 that the Bureau of Ocean Energy Management, Regulation and Enforcement (BOEMRE)—which was replaced by BSEE and BOEM in 2011 and which oversaw offshore oil and gas activities—faced considerable hiring challenges in the Pacific Region because of increased hiring by the oil and gas industry in that area due to the industry’s significant salary advantage over federal service. In addition, the report found that engineers in BOEMRE’s Gulf of Mexico Region had to work extra hours to keep up with increased workloads because of staffing shortages, resulting in their inability to attend training or take annual leave. It stated that continued shortages could lead to significant employee burnout and the possibility of less comprehensive reviews as employees attempted to keep pace with demands. In a second 2010 report, Interior’s Inspector General reported that BLM risked losing its trained inspectors because oil and gas operators commonly recruit BLM inspectors by offering high salaries during successful business periods. In that report, the Inspector General recommended, among other things, that BLM consider developing and implementing a continued service agreement requiring newly certified inspectors to stay with the bureau for a specified period of time. Further, the Outer Continental Shelf Safety Oversight Board reported in 2010 that Interior did not have a formal program to train its inspectors.

In terms of training, the Outer Continental Shelf Safety Oversight Board also noted in its 2010 report that almost half of the offshore inspectors it surveyed said they did not receive sufficient training. Further, BOEMRE did not have an inspection certification program that combined classroom and on-the-job experience, as well as a formal technical review or exam. By contrast, the report pointed out that BLM had a certification program that combined classroom instruction, on-the-job experience, and a formal technical review or exam. Among the board’s recommendations were for Interior to implement a bureau-wide certification or accreditation program for inspectors; consider partnering with BLM and its National Training Center to establish an Interior oil and gas inspection certification program, with training modules appropriate to the offshore environment as needed; develop a standardized training program similar to other Interior bureaus to ensure that inspectors are knowledgeable in all pertinent regulations, policies, and procedures; and ensure that annual training keeps inspectors up-to-date on new technology, policies, and procedures.

Interior’s Inspector General came to similar conclusions and made similar recommendations in 2010.

To address hiring and retention challenges, the federal government has a variety of tools available to use. For example, to address staffing problems caused when nonfederal employers pay significantly higher salaries than what the federal government pays, an agency may request special salary rates from OPM that establish higher minimum rates of basic pay for positions in one or more geographic areas. Agencies may also use incentive payments to recruit and retain employees. Incentive payments can come in the form of recruitment incentives, retention incentives, and relocation incentives. Recruitment incentives can be paid to new employees in certain difficult-to-fill positions; retention incentives can be paid to certain current employees holding high or unique qualifications; and relocation incentives can be paid to certain current employees who must relocate to accept a position in a different geographic area and whose position is difficult to fill. To receive an incentive payment, the employee must agree to complete a specified period of service with the agency. In general, total incentive payments may not exceed 25 percent of the employee’s original annual rate of basic pay multiplied by the number of years of service the employee agrees to complete. Agencies may also repay federally insured student loans in order to recruit or retain highly qualified candidates or employees through the Student Loan Repayment Program. Through this program, agencies may make payments to the loan holder of up to a maximum of $10,000 for an employee in a calendar year and a total of not more than $60,000 for any one employee. Employees receiving this benefit must sign an agreement to remain in the service of the agency for at least 3 years. Federal agencies can use special salary rates, incentive payments, and student loan repayments in combination to increase an employee’s overall compensation.

Interior Has Taken Steps to Improve Hiring and Retention but Has Not Evaluated Its Efforts and Has Missed Opportunities to Collaborate on Challenges

Since 2012, Interior has taken steps to resolve its hiring and retention challenges for key oil and gas staff, but it has not evaluated the effectiveness of its efforts. In addition, Interior has missed opportunities to facilitate collaboration among the bureaus, and as a result, the bureaus have sometimes acted in a fragmented, overlapping, and potentially duplicative fashion to resolve similar hiring and retention challenges.

Interior Has Taken Steps to Address Hiring and Retention Challenges but Has Not Evaluated the Effectiveness of Its Efforts

Since 2012, Interior has taken steps to address two underlying factors— lower salaries and a lengthier hiring process compared with the oil and gas industry—that have impeded its ability to hire and retain key oil and gas staff, but it has not evaluated the effectiveness of its efforts.

Salaries

Interior has increased the compensation for certain key oil and gas staff through use of special salary rates, incentive payments, and student loan repayments since fiscal year 2012, but the department has not evaluated the effectiveness of this compensation in resolving its hiring and retention challenges. During fiscal years 2012 through 2016, Interior had special salary rates, authorized by Congress in annual appropriations acts, that allowed it to pay certain staff up to 25 percent more than their basic pay. Interior stated that in 2013 the Office of Policy, Management and Budget met with officials from OPM, the U.S. Department of Agriculture, the Department of Defense, and the U.S. Army Corp of Engineers to discuss the impacts of expanding oil and gas extraction activities on their recruitment and retention efforts. Interior also stated that the Office of Policy, Management and Budget worked with officials from BLM, BSEE and BOEM to (1) ensure that the three bureaus had the capacity to fund special salary rates through the budget process, (2) develop an integrated special salary rate request to OPM and (3) issue guidance that would provide instruction to human resource officials and hiring managers on its use. Further, Interior stated that, beginning in fiscal year 2013, the Office of Policy, Management and Budget submitted applications to OPM requesting to increase the base salaries for staff in certain positions and geographic locations through a special salary rate. In fiscal years 2015 and 2016, OPM approved Interior’s requests to provide key oil and gas staff in 11 states up to 35 percent more than their basic pay.

In addition, some of the bureaus increased compensation through other tools, such as incentive payments and student loan repayments. For example, for fiscal years 2012 through 2014, BLM and BSEE substantially increased the number of staff receiving a retention incentive payment from a total of 14 to a total of 346 employees. During the same period, BSEE and BOEM increased the number of staff receiving a student loan repayment from 25 to 66 employees. (See fig. 3.) As noted earlier in this report, employees receiving incentive payments and student loan repayments must sign an agreement to remain working for the agency for a certain period of time. Service agreements, in addition to the actual monetary payment, may also play a role in retaining staff.

Officials from the three bureaus said that these efforts to increase the compensation paid to key oil and gas staff, along with the industry downturn that reduced private sector hiring, had likely helped them fill vacancies. In May 2015, BLM officials said that anecdotally they know that the incentive payments and special salary rates have proven to be somewhat effective and were particularly helpful in recruiting and retaining inspectors. Similarly, in May 2015, BSEE officials said that they had hired more staff in the first part of fiscal year 2015 than in fiscal year 2014, although they noted that they had the most difficulty recruiting petroleum engineers and inspectors in the Gulf of Mexico Region because the pool of prospective candidates was smaller than for other positions. BSEE officials also said that while they lost a fair number of staff in fiscal year 2014, many of those who left did so because of retirements. Senior BOEM officials also reported success in hiring staff, and senior officials said that as of May 2015 the bureau was fully staffed; however, several months later BOEM officials in the Gulf of Mexico Region did report some vacancies. Senior BOEM officials said they had the most difficulty recruiting petroleum engineers, geologists, and geophysicists.

Outside of these anecdotal observations, Interior and the bureaus have not evaluated whether these efforts, and the specific tools they used, were effective in hiring and retaining staff. In prior work, we have found that strategic workforce planning requires evaluation of an agency’s progress toward its human capital goals. In November 2014, Interior senior officials told us that they would implement a performance measure framework to evaluate the effectiveness of incentives on a quarterly basis beginning in April 2015. However, as of July 2016, a senior official from the Office of Policy, Management and Budget said these quarterly reviews had not begun yet. In September 2016, officials said they had developed initial performance metrics and gathered data for the first three quarters of fiscal year 2016 and would continue to track and monitor the data on a quarterly basis. However, the agency has not yet used this data to evaluate the effectiveness of incentives. In the absence of these evaluations, Interior cannot determine the extent to which the tools it is using are effective in meeting its goals of hiring and retaining key staff or whether it is expending funds on tools that are not the best use of its limited resources. In addition, without regular evaluations, Interior may not have the information it needs to determine if or how it should alter the tools it uses as the oil and gas market shifts, potentially increasing Interior’s competition with industry for oil and gas staff.

Bureau officials acknowledged that retaining newly hired staff may prove difficult when oil and gas market conditions change again and companies increase their hiring efforts. In April 2016, BLM officials noted that while there have been some market-based changes that have proved to be advantageous to the bureaus’ hiring efforts in some locations, the potential for a resurgence in private sector demand for qualified petroleum engineers and inspectors remains a likely probability over the next 12 to 18 months. BLM further noted that since it takes 12 to 18 months to recruit, train, and certify entry-level petroleum engineers and inspectors, losing these staff after they are hired and trained could undermine much of the progress the bureau had made.

Because of the importance of key staff for Interior’s oversight of oil and gas development, we developed a statistical model to examine the main factors associated with the likelihood that federal employees in key positions—petroleum engineers, inspectors, geologists, geophysicists, natural resource specialists (or biologists), environmental protection specialists—would leave those positions. While not definitive, the model illustrates the type of analysis that Interior could potentially perform itself—using more detailed and current data—to evaluate the effectiveness of specific tools in retaining key oil and gas staff. For our analysis, we used data mainly from OPM’s EHRI data set, which contains personnel data for civilian federal employees. We supplemented our analysis with data from BLM so that we could identify employees in key positions who were responsible for oil and gas oversight. We used data on approximately 29,000 federal employees throughout the federal government, all of whom were hired into one of the key oil and gas positions during fiscal years 2003 through 2014. Our model estimated the effect that differences in salaries and other compensation had on the likelihood that a federal employee would leave his or her position, while controlling for factors such as the employee’s age, gender, geographic location, and length of time working in that position. We also examined the effect of the performance of the oil and gas market on employee retention.

Our results showed that federal employees who received higher adjusted basic pay (which could include a special salary rate), retention payments, student loan repayments, and other additional compensation were less likely to leave than their counterparts working in the same positions who did not receive such compensation. We also found that when the oil and gas market was performing well, federal employees in these positions were more likely to leave their positions. Specifically, for federal employees working in key oil and gas oversight positions, we found the following:

Higher adjusted basic pay was significantly associated with a lower likelihood of leaving, with each additional $1,000 reducing the relative odds of leaving by about 2.0 percent.

All the categories of other compensation in our model—retention payments, student loan payments, cash awards, and time-off awards —were significantly associated with reducing the likelihood of leaving. Among these categories, the strongest effects were from retention and student loan payments.

A higher percentage growth rate of the oil and gas market was significantly associated with a higher likelihood of employees leaving their position. Interior officials we interviewed said that they have difficulty retaining key employees when the oil and gas market is performing well, and our results support this assertion. Conversely, a slower growth of the oil and gas market was associated with fewer employees leaving their positions.

Our analysis also showed that natural resource specialists, biologists, and environmental protection specialists were more likely than inspectors to leave their positions. In addition, our analysis showed that BSEE and BLM employees were more likely to leave their positions than federal employees working in the same positions in other federal agencies and other Interior bureaus. This effect was stronger at BSEE than at BLM, with BSEE employees responsible for oil and gas oversight being 50 percent more likely to leave than their counterparts at BLM. However, our results are based on EHRI data from fiscal years 2003 through 2014, the most current EHRI data available to us at the time of our analysis. In comparison, Interior has other data available to it that are more current and detailed. For example, Interior has access to current fiscal year information, which are not yet available in EHRI, on the types and amounts of payments it has given its employees, which would allow the department to conduct a more thorough and precise evaluation of the effect of these payments on retention of key oil and gas staff.

Hiring Process

Each of the three bureaus has taken steps to begin to address their lengthy hiring process. For example, in 2015 the three bureaus adopted new human resources software that officials said will provide them with better data to track their hiring process. In June 2016, officials from the three bureaus said that they had started analyzing data extracted from this new system to identify steps in the hiring process that may be causing delays. Also in 2016, BSEE and BOEM issued new hiring process guidance to clarify steps in the hiring process for its managers. BSEE and BOEM also provided multiple training classes on the new guidance to ensure that managers understood the process. In addition, in a July 2015 memorandum, BOEM summarized the results of an analysis of its hiring process and identified some improvements that could be made. However, in reviewing the analysis, we identified problems with the data used, such as missing and inaccurately recorded dates.

In June 2016, a senior official from Interior’s Office of Policy, Management and Budget said that they were aware of the bureaus’ efforts to analyze their hiring process time. Officials from the three bureaus said that their hiring processes continue to exceed OPM’s goal of 80 days. Some bureau officials also told us that their hiring process sometimes took as long as 190 days. As noted previously, we recommended in January 2014 that Interior systematically collect data on hiring times for key oil and gas positions, ensure the accuracy of the data, and analyze the data in order to identify the causes of delays and expedite the process. However, senior officials from the Office of Policy, Management and Budget did not indicate any plans to look across the bureaus’ efforts in order to help address their shared challenge of a lengthy hiring process. In the absence of such action to address the lengthy hiring processes for the bureaus, they may be losing qualified applicants who accept a different job. We continue to believe that having accurate hiring data and finding ways to reduce the lengthy hiring process are important steps toward resolving Interior’s hiring challenges and may prove especially important if the oil and gas market shifts.

Interior Has Missed Opportunities for Collaboration on Resolving Challenges in Hiring and Retaining Key Staff

Interior’s Office of Policy, Management and Budget has missed opportunities to facilitate collaboration across the three bureaus in addressing their shared challenges in hiring and retaining staff. For example, officials from this office said that they assembled the three bureaus’ requests to OPM for a special salary rate, but we found that they did not facilitate collaboration among the bureaus about which staff should receive a special salary rate. BOEM officials requested the 35 percent special salary rate for certain key oil and gas staff but did not request this special salary rate for its biologists (also referred to as natural resource specialists). In contrast, BLM requested this 35 percent special salary rate for its natural resource specialists along with other positions. BOEM regional managers said that they were not aware that BLM was requesting the special salary rate for its natural resource specialists and did not know that they could request the special salary rate for these staff. BOEM managers said that they learned of this after OPM had already approved these requests. Some of these managers said that had they known BLM was going to request a special salary rate for its natural resource specialists, they probably would have done so too. Some officials said that the bureaus compete with each other for the same pool of applicants and staff. The fact that BLM can pay a natural resource specialist 35 percent more than BOEM may place BOEM at a disadvantage in its recruitment efforts and its ability to retain staff if its natural resource specialists leave to take a comparable position at BLM. In addition, BOEM may also be particularly vulnerable to losing its natural resource specialists to industry, based on the results of our statistical model and comments from BOEM managers, both of which indicated that these staff were more likely to leave their position relative to other key oil and gas staff.

Senior officials in Interior’s Office of Policy, Management and Budget did not identify any collaboration mechanisms that they used to bring the three bureaus together to discuss their shared human capital challenges. These officials said the bureaus’ senior managers interact through the meetings of the Deputies Operating Group and Principals Operating Group. However, in our review of the topics discussed by these groups in fiscal year 2015, we found that the bureaus’ hiring and retention challenges were not discussed. In prior work, we have found that collaborative efforts can enable organizations to produce more public value than could be produced when they act alone. To facilitate collaboration, agencies can use a variety of mechanisms, such as interagency groups, communities of practice, and liaison positions. Further, as we have concluded in prior work, leadership is a necessary element for successful collaborative working relationships. Officials from the three bureaus said that they do not have a mechanism, such as a workgroup, in place to collaborate with each other on their shared hiring and retention challenges. In the absence of such a collaboration mechanism, the bureaus have sometimes acted in a fragmented, overlapping, and potentially duplicative fashion to resolve similar hiring and retention challenges. For example, some members of the BSEE and BOEM recruitment teams told us that while they sought to hire staff with similar skills, they participated in recruitment events, such as job fairs, separately and did not give prospective applicants information about career opportunities available at the other bureaus. Officials also said the fact that the bureaus maintained separate recruitment tables was confusing to prospective applicants. Some officials noted that greater collaboration could be useful. For example, some BOEM officials said it would be beneficial if the bureaus had a single booth that could represent all the job opportunities at Interior because the broader range of opportunities and locations might generate more interest among prospective applicants. However, without further leadership from the Office of Policy, Management and Budget to create or use an existing mechanism to facilitate collaboration in addressing hiring and retention, the bureaus may continue to address their shared challenges through fragmented and potentially duplicative efforts, which can waste resources.

Interior Has Not Fully Evaluated Training Needs or Effectiveness, According to Officials, and Has Provided Limited Leadership in Facilitating the Bureaus’ Shared Use of Training Resources

Interior and its bureaus have trained key oil and gas staff without fully evaluating the bureaus’ staff training needs or the training’s effectiveness, according to officials, and Interior has provided limited leadership in facilitating the bureaus’ sharing of training resources. Specifically, Interior has not evaluated training needs or effectiveness as required by law and regulations, according to officials, and its bureaus have not evaluated training needs or effectiveness as directed by departmental policy. Further, Interior’s Office of Policy, Management and Budget has provided limited leadership in facilitating the sharing of training resources across the bureaus, appearing to miss opportunities that could improve the use of these resources.

Interior Has Not Evaluated Training Needs or Effectiveness, According to Officials

Interior’s Office of Policy, Management and Budget has not evaluated the three bureaus’ training efforts, contrary to federal law and regulations, according to officials. The Federal Workforce Flexibility Act of 2004 requires agencies to regularly evaluate their training at the department level with respect to accomplishing specific performance plans and strategic goals in performing the agency mission and then modify the training as needed. Similarly, OPM has stated that training and the effective evaluation of training are critical within the federal government, and OPM regulations require agencies to evaluate their training programs annually to identify training needs and assess how well training efforts contribute to accomplishing the agency mission. However, senior officials from the Office of Policy, Management and Budget said that they have not performed these annual evaluations of the bureaus’ staff training needs. In addition, senior officials from this office said they have not requested or received these annual training evaluations from the bureaus even though Interior’s Departmental Manual states that bureaus should conduct such evaluations and submit them to the office. These officials explained that they thought that the 2008 Departmental Manual was old and needed to be revised. However, based on our review of the manual and discussion with an official in Interior’s Office of the Solicitor, we determined that the manual is still in effect.

Similarly, the bureaus have not evaluated their oil and gas staff’s training needs to the extent directed by Interior’s policies, according to officials. For example, as noted above, Interior’s Departmental Manual directs each bureau to conduct an annual evaluation of its training program; these evaluations are to determine if the program is effectively meeting identified needs. The manual also states that training programs should identify and address competency gaps, including for technical competencies. Similarly, our guide for assessing training efforts in the federal government states that well-designed training programs are linked to agency goals and to the skills and competencies needed for the agency to perform effectively. However, none of the bureaus have consistently evaluated training needs, according to officials, and only one of the bureaus developed competencies for their key oil and gas staff.

The bureaus’ efforts to evaluate training needs and develop competencies include the following:

BLM most recently evaluated training needs for its oil and gas staff in 2012 and 2013. BLM evaluated the training needs for its natural resource specialists and environmental protection specialists in 2012, followed by its petroleum engineers, inspectors, and geologists in evaluations that spanned 2012 and 2013. In so doing, BLM did not follow the direction of Interior’s Departmental Manual to conduct annual evaluations. In addition, BLM has not developed technical competencies for its oil and gas staff per OPM and Interior definitions.

BSEE has not formally evaluated the training needs of its key oil and gas staff, according to officials. Instead, BSEE officials told us that these training needs are discussed by managers, subject matter experts, and other staff who use this information to identify training courses for staff to take. In addition, BSEE has not developed technical competencies for its key oil and gas staff per OPM and Interior definitions.

BOEM has relied on its offices within its three regions to implement its training efforts, and on individual supervisors to evaluate training needs, according to BOEM officials, but BOEM officials told us that the bureau has not formally evaluated the training needs of its key oil and gas staff bureau-wide. These supervisors evaluate training needs of individual employees at the beginning of each fiscal year, and BOEM seeks to address these needs of its staff through vendor- based training, training taught by BOEM staff, and mentoring, according to officials. BOEM has, however, developed competencies per OPM and Interior definitions for its geologists, geophysicists, and petroleum engineers by using ones already published by other sources.

Officials from each of the bureaus told us they have not performed annual evaluations of their training needs because officials from the Office of Policy, Management and Budget have not requested them. Without evaluating training needs and developing competencies, Interior cannot ensure that the training it provides for key oil and gas staff is linked to the competencies needed for the agency to perform effectively and that the training addresses any competency gaps.

The bureaus also have not evaluated the effectiveness of the training provided to their key oil and gas staff as directed by Interior’s Departmental Manual. The manual states that all formal training courses sponsored by departmental bureaus or offices are expected to be evaluated, and it recommends that bureaus use a five-level evaluation system to assess the effectiveness of their training, with targets for the percentage of courses that should be evaluated at each level. (See fig. 4.) For example, the guidance recommends that all training courses receive level 1 evaluations, which measure student satisfaction and identify ways to improve the training; successively lower percentages of courses are recommended to receive successively higher levels of evaluation.

Officials from each of the bureaus told us they have not fully evaluated the effectiveness of their training efforts because either they did not have staff to perform them or Interior did not request it. Collectively, the bureaus conducted varying levels of evaluations, and none reported doing evaluations above level 3, as discussed below:

BLM conducts level 1 and 2 evaluations for each course, as well as level 3 evaluations and proficiency examinations for certain courses, according to BLM officials. For example, for its inspector certification training program, comprised of six modules, each inspector is to complete a proficiency examination and complete related field work, according to a BLM handbook. BLM’s inspectors must demonstrate proficiency in each module before they can progress to the next module, according to BLM officials. Following the successful completion of all six modules, inspectors are eligible for certification and, once certified, they are allowed to issue citations to operators when appropriate.

BSEE conducts level 1 evaluations for all of its training and its vendors conduct level 2 evaluations to some extent but not to the extent directed by the Departmental Manual, according to BSEE officials. In addition, BSEE has not developed competencies for its inspectors and does not conduct level 3 evaluations for its inspectors to measure how training affected behavior and skills, according to officials. Further, BSEE’s training for inspectors does not include proficiency examinations or certifications, according to officials, as BLM’s training program does. BSEE officials told us that they have not implemented a certification program, although the Outer Continental Shelf Safety Oversight Board and Interior Inspector General recommended it in 2010. By conducting such evaluations and requiring these examinations for certification of inspectors, BSEE could ensure that its inspectors learned and could apply content received in training courses (i.e., were adequately trained). In the absence of such evaluations, BSEE may not be able to verify that its inspectors are adequately trained. BSEE officials told us that they planned to obtain two independent evaluations of their training efforts. According to these officials, the first evaluation, which will review whether the training currently offered to engineers is sufficient, was tentatively scheduled to start in July 2016. The second evaluation will review the bureau’s approach to identifying competencies, training, and possible certification requirements for inspectors and, according to officials, the contract for the work should be awarded by December 2016. As of June 2016, BSEE officials told us that they were finalizing their efforts to initiate the first evaluation and were planning to complete a statement of what work would be included in the second evaluation.

BOEM conducts level 1 evaluations when requested by vendors, but BOEM did not report conducting higher-level evaluations. In addition, BOEM officials stated that BOEM does not systematically evaluate training provided by internal BOEM staff, vendors, or others because the bureau does not have staff assigned to training, such as to develop training curricula or evaluate training efforts.

None of the bureaus reported conducting level 4 or 5 evaluations, which would give the bureaus information about the overall effectiveness of their training efforts by measuring the impact of training courses on staff’s job performance and comparing program benefits to training costs. During our review, key oil and gas staff we interviewed told us that some courses provided for inspectors were not always effective. For example, BSEE inspectors at four local offices told us in September 2015 that the training courses BSEE provided them, which were primarily led by contractors, did not adequately prepare them to perform inspections because the courses focused on how equipment operates and did not teach them how to inspect the equipment. Similarly, managers from four BSEE offices told us that inspector courses were not entirely relevant and not tailored to inspectors’ responsibilities. For example, one manager said that these training courses do not familiarize inspectors with information they need to perform inspections, such as what to look for when inspecting the equipment. A BSEE training official told us in January 2016 that she had heard this same feedback. In response, BSEE created an extra day of training for some courses, such as their Cranes and Rigging Inspections course that would be led by a BSEE instructor, not a contractor, who would teach the inspectors how to inspect the equipment covered in these courses. Without evaluating its bureaus’ training efforts, Interior may not be able to ensure that its key oil and gas staff are being adequately trained to execute their oversight tasks, and it may not be spending training funds effectively and efficiently.

Interior Has Provided Limited Leadership in Facilitating the Sharing of Training Resources across Bureaus

Interior’s Office of Policy, Management and Budget has provided limited leadership in facilitating the sharing of training resources across the bureaus. The Office of Strategic Employee and Organization Development—housed within the Office of Policy, Management and Budget—has objectives that include improving training across the bureaus and facilitating the sharing of training resources, such as training staff expertise and course curricula. However, we identified areas where it appears that the Office of Strategic Employee and Organization Development has missed opportunities to improve the bureaus’ training efforts and facilitate the sharing of training resources. For example, BOEM, which is the smallest of the three bureaus, does not have staff assigned to developing curricula or evaluating training efforts across the bureau and, as discussed earlier, it therefore relies on external vendors for training and evaluates the training when requested by the vendors. In addition, BSEE, which had 6 full-time staff in their Offshore Training Program as of July 2016, according to officials, also relies on external vendors for training and conducting level 2 evaluations. In contrast, as of July 2016, BLM had 59 full-time staff in its National Training Center, and has the capacity to evaluate their training efforts, according to officials.

In 2010, the Outer Continental Shelf Safety Oversight Board and Interior’s Inspector General recognized strengths in BLM’s training program for inspectors and recommended that BSEE and BLM consider partnering to establish an Interior-wide inspection certification program. However, neither Interior’s Office of Policy, Management and Budget nor the bureaus evaluated the need for or viability of a joint inspector certification training program, according to officials.

Similarly, Interior’s Office of Policy, Management and Budget has not pursued potential opportunities for BOEM and BSEE to share training resources, according to officials. Recognizing that BOEM is a smaller bureau than BSEE, and recognizing the benefits of economies of scale, BOEM has arranged since 2011 to have BSEE’s human resources department service BOEM for select human resource functions, but not training, according to a senior BOEM official. In January 2016, officials from the Office of Policy, Management and Budget said that they were in favor of BOEM using BSEE’s training program, but they had not yet taken any steps toward encouraging such collaboration to facilitate the sharing of resources.

In addition, to develop training courses specific to their bureau, BSEE training officials said they would need curriculum developers, which they do not have. As a result, BSEE officials said they rely almost exclusively on external off-the-shelf courses taught by contractors. In contrast, BLM’s training center has about six full-time curriculum developers, according to officials. BLM training officials said that these curriculum developers would be able to develop training curricula for BSEE if they worked alongside subject matter experts from BSEE. However, officials told us that the Office of Policy, Management and Budget has not taken any steps to encourage collaboration in this area.

Senior officials from the Office of Policy, Management and Budget acknowledged that their office has not effectively facilitated the sharing of training resources across the bureaus as of June 2016. As we mentioned earlier, we found in prior work that to facilitate collaboration, agencies can use a variety of mechanisms, such as interagency groups, communities of practice, and liaison positions; that leadership is a necessary element for successful collaborative working relationships; and that collaborative efforts can enable organizations to produce more public value than could be produced when they act alone. In January 2016, a senior official from the Office of Policy, Management and Budget said that their focus in the previous fiscal year had been to assist the bureaus in obtaining a special salary rate for their key oil and gas staff. Another senior Interior official said that in January 2016 their Interior Training Directors Council— composed of senior training officials across Interior—would begin reviewing training across the bureaus and seek to identify opportunities to share training resources. According to its charter, the goal of the council is to facilitate a partnership across the bureaus in order to maximize the effectiveness and efficiency of training efforts throughout the Department of the Interior. In March 2016, the council, which had previously operated as a community of practice since 2001, shifted to a more formal structure that would allow it to develop policy and make recommendations to Interior’s Human Capital Officers, according to a senior official. However, as of June 2016, officials had not reported any progress made by the council, and it is unclear what, if any, steps the office has taken to review training and identify opportunities to share training resources. Without further leadership from the Office of Policy, Management and Budget to create or make better use of an existing mechanism that effectively facilitates collaboration across the bureaus and helps them identify opportunities to share training resources, Interior and its bureaus may not be spending training funds effectively and efficiently.

Conclusions

Since 2012, Interior has taken steps toward resolving its challenges in hiring and retaining key oil and gas staff, who are the front line in providing effective oversight of activities related to federal oil and gas resources. Notably, to hire and retain such staff, Interior’s bureaus have invested increasing resources into compensating them through special salary rates, incentive payments, and student loan repayments—tools that can help bridge the gap between federal salaries and those paid by industry. We recommended in January 2014 that Interior explore the expanded use of existing authorities, such as recruitment incentives, and develop clear guidance for how the effectiveness of their use will be assessed, among other things. Interior has partially responded to this recommendation by its increased use of incentives, but it has not evaluated their effectiveness. Interior also has not evaluated the effectiveness of other tools, specifically the special salary rates and student loan repayments. We developed a statistical model that Interior could expand upon to analyze the effectiveness of specific tools. In the absence of such evaluations, Interior cannot know the extent to which the increased use of incentive payments, special salary rates, and student loan repayments have been effective in hiring and retaining key staff. In addition, without regular evaluation, Interior may not have information it needs to determine if or how it should alter its approach when the oil and gas market shifts and industry begins hiring more employees, potentially increasing Interior’s competition with industry for oil and gas staff.

Further, Interior continues to face a lengthy hiring process, according to officials. In January 2014, we also recommended that Interior systematically collect data on hiring times for key oil and gas positions, ensure the accuracy of the data, and analyze the data to identify the causes of delays and expedite the hiring process. All three bureaus have adopted new human resources software that may provide them with better data to track their hiring process, and the bureaus have started to analyze these data to identify what steps are causing delays in the hiring process. We continue to believe that having accurate hiring data and finding ways to reduce the lengthy hiring process are important steps toward resolving Interior’s hiring challenges and may prove especially important if the oil and gas market shifts.

Concerning training, Interior has not evaluated the bureaus’ training needs or the training’s effectiveness as required by federal law and regulations, and the bureaus’ have not fully evaluated their training efforts as directed by Interior policy. None of the bureaus have consistently performed annual evaluations of their training needs for all key staff, and only one of the bureaus has developed technical competencies that are critical to successful performance by these staff, as directed by Interior’s Departmental Manual. Without evaluating training needs and developing such competencies, Interior cannot ensure that the training it provides for key oil and gas staff is linked to the competencies needed for the agency to perform effectively and that the training addresses any competency gaps. In addition, none of the bureaus have evaluated the effectiveness of the training as directed by the Departmental Manual. Because Interior and its bureaus have not fully evaluated their training efforts, Interior may not be able to ensure that its key oil and gas staff are being adequately trained to execute their oversight tasks, and it may not be spending training funds effectively and efficiently. BLM’s inspector certification training program stands out as an exception to these general findings because BLM has evaluated inspectors’ training to ensure that they have learned and can apply skills critical to their oversight duties. In contrast, BSEE does not give inspectors proficiency examinations to measure learning or application of skills, and does not certify them, as recommended by two oversight bodies in 2010. Although BSEE officials said they were finalizing their efforts to initiate the first evaluation of their training efforts and were planning to complete a statement of what work would be included in the second evaluation, unless they follow through with and complete these efforts, the bureau cannot verify that its inspectors are adequately trained.

Moreover, the Office of Policy, Management and Budget, which is responsible for managing Interior’s human resources and addressing cross-cutting issues, has not effectively facilitated collaboration among the bureaus in addressing their shared hiring, retention, and training challenges. Senior officials in Interior’s Office of Policy, Management and Budget did not identify any collaboration mechanisms currently being used to bring the three bureaus together to discuss their shared human capital challenges and to share training resources. In the absence of such a collaboration mechanism, the bureaus have sometimes acted in a fragmented, overlapping, and potentially duplicative fashion to resolve similar challenges. Without further leadership from the Office of Policy, Management and Budget to create or make better use of an existing mechanism, such as the Deputies Operating Group, Principals Operating Group, or the Interior Training Directors Council, to facilitate collaboration in hiring, retention, and training, the bureaus may continue to address their shared challenges through fragmented and potentially duplicative efforts.

Recommendations for Executive Action

To help ensure Interior can hire, retain, and train staff it needs to provide effective oversight of oil and gas activities on federal lands and waters, we recommend that the Secretary of the Interior take the following five actions: Direct the Assistant Secretary for Policy, Management and Budget to:

Regularly evaluate the effectiveness of its available incentives, such as special salary rates, the student loan repayment program, and other incentives in hiring and retaining key oil and gas staff.

Annually evaluate the bureaus’ training programs, including: staff training needs, potential opportunities for the bureaus to share training resources.

Direct the Assistant Secretary for Land and Minerals Management to:

Develop technical competencies for all key oil and gas staff.

Evaluate the need for and viability of a certification program for BSEE inspectors.

Direct the Assistant Secretary for Policy, Management and Budget to coordinate with the Assistant Secretary for Land and Minerals Management to create or use an existing mechanism, such as the Deputies Operating Group, Principals Operating Group, or the Interior Training Directors Council, to facilitate collaboration across the three bureaus in addressing their shared hiring, retention, and training challenges.

Agency Comments and Our Evaluation

We provided our draft report to Interior for review and comment. Interior provided written comments, in which it agreed with one of the five recommendations in the draft report, partially agreed with three others, and disagreed with the remaining recommendation. Interior’s comments are reproduced in appendix II, and key clarifying points from the department are discussed below in the context of our recommendations. Interior also provided technical comments, which we incorporated as appropriate.

Interior agreed with our first recommendation, which would have the Assistant Secretary for Policy, Management and Budget regularly evaluate the effectiveness of its available incentives. Interior also submitted several points of clarification and comments regarding our related findings: Interior clarified that it now has full approval for the special salary rates. Interior also provided documents showing performance metrics it would use to track and monitor the impact of special pay rates and other pay flexibilities, such as incentive payments. We added language to our report to further acknowledge these actions.

Interior disagreed (1) with the accuracy of how the report portrayed the Office of Policy, Management and Budget’s role and (2) that the office had missed opportunities to collaborate across the bureaus, especially as it related to special salary rates for key positions. Interior stated that the office was an integral partner, collaborator, and coordinator among the departmental stakeholders and the bureaus' leadership, human capital and budget teams. In response to Interior’s comments, we added language to specifically identify the Office of Policy, Management and Budget’s role and actions in the special salary process. Regarding missed opportunities, Interior disagreed that BOEM was excluded from the collaborative process for the special salary requests. In the draft report, we did not state that BOEM was excluded but rather that BOEM regional managers said they were not aware that BLM was requesting the special salary rate for its natural resource specialists and did not know that they could do so. Therefore, while Interior stated that officials from the Office of Policy, Management and Budget said that the office collaborated and coordinated between departmental stakeholders, it appears not all stakeholders were equally informed.

Interior stated that BLM's inclusion of natural resource specialists does not have a negative impact on BOEM mission delivery. We did not address such an impact in our report. We did state, however, that since BLM can pay a natural resource specialist 35 percent more than BOEM can, this difference may place BOEM at a disadvantage in its recruitment efforts and its ability to retain staff if its natural resource specialists leave to take a comparable position at BLM.

Interior stated that the data demonstrated that the greatest need for BLM to acquire natural resource specialists was within the North Dakota region and that BOEM does not maintain offices in that region. However, BLM also offers the special salary rate for natural resource specialists in other states where BOEM does maintain offices. In addition, federal employees could relocate from one state to another state in order to take a new job.

Interior partially agreed with our second recommendation, to have the Assistant Secretary for Policy, Management and Budget annually evaluate the bureaus’ training programs, including training needs, training effectiveness and potential opportunities for the bureaus to share training resources. Interior said that the Office of Policy, Management and Budget would ensure that the three bureaus are coordinating their training needs and that its Office of Strategic Employee and Organizational Development can validate the bureaus’ engagement in this activity and provide support in fulfilling these recommendations. While these steps may be useful, as stated in the report, Interior has not evaluated the bureaus’ training needs or the training’s effectiveness as required by federal law and regulations, and the bureaus have not fully evaluated their training efforts as directed by Interior policy. We continue to believe that the Office of Policy, Management and Budget is required by law and regulation to evaluate the bureaus’ training programs. Without evaluating the bureaus’ training programs, Interior cannot ensure that the training provided is sufficient to support the required oversight duties.

Interior also submitted several points of clarification and comments regarding related findings: Interior stated that our report assumed that BOEM and BSEE should be acquiring technical training from BLM, which, according to Interior, does not accurately reflect the analysis conducted to determine the training needs for offshore development or recognize the training coordination that does occur. Relatedly, Interior stated that we did not acknowledge the vastly different skill sets needed to inspect or permit equipment needed for onshore versus offshore facilities. However, our draft report did not state or assume that BLM would be training these bureaus, and we did not recommend such an action. We did state that it appears that Interior missed opportunities to improve the bureaus’ training efforts and facilitate the sharing of training resources in areas, such as curricula development, which led to our second recommendation. With regard to the differences in skill sets needed for inspections, our interviews with agency officials support the point that there are differences in these two inspection environments. We added language to our report to better acknowledge these differences. Nonetheless, our interviews also indicate that there are common skills and knowledge used to inspect onshore and offshore facilities. This point is illustrated by the fact that 15 BSEE staff took one or more of BLM’s inspector certification training modules from fiscal year 2012 through fiscal year 2015, according to BLM documentation. We added language to our report to recognize that BSEE staff took this BLM training.

Interior stated that our draft report did not recognize the training and coordination that occurs and described collaborative efforts between BSEE and BLM regarding training. We added language to our report to recognize the BSEE staff who took BLM training. Nonetheless, it appears that the Office of Policy, Management and Budget has missed opportunities to facilitate the sharing of training resources, and we continue to believe that there is a need for the type of evaluations called for in our recommendation. Once the bureaus have made these evaluations, they should be better able to identify overlapping skill sets which could then be addressed by sharing training resources.

Interior also noted, with regard to BSEE training, that it would be difficult and expensive to continuously update standard certification modules and tests to keep pace with the technology changes in the offshore oil and gas industry. Interior stated that BSEE therefore chose to rely on vendors, rather than in-house expertise, to provide classroom training. However, based on our review, none of the bureaus has performed a level 5 evaluation, which would compare the benefits and costs of training. As a result, the bureaus do not know whether it would be cost effective to update certification modules rather than continue the current reliance on vendors.

Interior partially agreed with our third recommendation that directed the Assistant Secretary for Policy, Management and Budget to develop technical competencies for all key oil and gas staff. In its comments, Interior said that because oil and gas occupations are highly technical positions, the bureaus would be best positioned to identify technical competencies. We agree and have redirected our recommendation to the Assistant Secretary for Land and Minerals Management, where the three bureaus are housed.

Interior disagreed with our fourth recommendation that directed the Assistant Secretary for Policy, Management and Budget to evaluate the need for and viability of a certification program for BSEE inspectors. Regarding this recommendation, Interior said that oil and gas inspection is highly technical and that BSEE was in the best position to evaluate the technical training needed to carry out its authorities and responsibilities. Based on this comment, we have redirected this recommendation to the Assistant Secretary for Land and Minerals Management. Concerning our related findings, Interior stated that the report does not recognize that although BSEE Level II inspectors do not receive a formal certificate, they receive a hands-on personal evaluation and approval from a supervisory inspector. According to Interior, this supervisory approval confirms that the Level II inspector attained all of the knowledge necessary through course work and supervised on-the-job training—and, more importantly, that the inspector sufficiently demonstrated these skills in the field—to become a Level III inspector. Although our current review of training focused on technical training delivered through classroom instruction and did not directly include an evaluation of on-the-job training, we agree that such efforts are an important part of an inspection training program. However, in July 2012, we reported that senior and regional office officials stated that relying on a combination of on-the-job training, which included pairing senior inspectors with newly hired inspectors, and some classroom instruction produced inconsistent results because some senior inspectors proved to be less effective trainers than others. We believe that BLM’s model of training inspectors through a certification program may offer some advantages over BSEE’s current approach, and we continue to believe that the need for and viability of such a program for BSEE inspectors should be evaluated.

Interior partially agreed with our fifth recommendation that directed the Assistant Secretary for Policy, Management and Budget to coordinate with the Assistant Secretary for Land and Minerals Management to create or use an existing mechanism to facilitate collaboration across the three bureaus in addressing their shared hiring, retention, and training challenges. Interior stated that coordination already exists among the bureaus and that, as part of the Office of Policy, Management and Budget’s quarterly review of performance data, the office will ensure that the bureaus continue to coordinate on hiring, retention, and training. However, Interior disagreed with our statement that the Office of Policy, Management and Budget has missed opportunities to collaborate across bureaus to address recruitment and retention challenges. Our report identifies examples of missed opportunities for collaboration, including BSEE and BOEM recruitment teams who, according to team members, participated in recruitment events such as job fairs separately and did not give prospective applicants information about career opportunities available at the other bureaus, even though they sought to hire staff with similar skills. Because of these findings, we continue to believe that the Office of Policy, Management and Budget should take a greater leadership role in facilitating collaboration to address shared challenges across the bureaus.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of the Interior, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Statistical Methodology Used to Examine Main Factors Associated with the Retention of Federal Employees in Key Oil and Gas Positions

To examine the Department of the Interior (Interior) efforts to resolve its hiring and retention challenges for key oil and gas staff, we developed a statistical model to examine the main factors that would reduce the likelihood that federal employees in key positions—those that corresponded to the positions of key oil and gas staff at the Bureau of Land Management (BLM), the Bureau of Safety and Environmental Enforcement (BSEE), and the Bureau of Ocean Energy Management (BOEM) —would leave those positions.

Data

We developed a model to examine the main factors associated with employee retention for key oil and gas-related employees at Interior. We analyzed the probability of retention of federal employees hired on a permanent basis into key oil and gas occupations from fiscal years 2003 through 2014. We used the Enterprise Human Resources Integration (EHRI) database, which contains information on variables such as adjusted basic pay, occupation, the agency where the employee worked, hiring, separation, and awards. We supplemented the EHRI data with data from the Standard & Poor’s 500 Energy Index (to measure demand from the private sector for these key employees) and with data from BLM to identify specifically those employees working in oil- and gas-related positions.

Methods

Our model considered only federal employees who were hired as either career competitive, conditional competitive, or career excepted; thus, other type of hires, such as transfers-in or temporary hires, were not included. We included only employees in our list of “key occupations” throughout the federal government. In order to simplify our analysis, we did not include employees with multiple periods of employment; that is, we only considered those employees who were hired one time from 2003 through 2014. Employees who were hired more than once accounted for only about 2 percent of the total number of hires during that time. In order to be comprehensive and include separations other than just resignations, we also included as “Quits” employees who had an inter-agency transfers, either horizontal (same grade) or upward movement (higher grade). Employees who separated for other reasons, such as retirement or death, or who were still employed at the end of fiscal year 2014, were treated as “Censored” by the model and no account was taken of the difference in these types of “Exits” from the analysis. However, in order to mitigate the effect on our model of possible separations due to retirement or death, we excluded employees who were 50 or older at the time they were hired.

Econometric Model

is the cumulative logistic probability distribution describing the probability of i-th employee quitting at time (month) t and z(i,t) is a list (vector) of variables that are believed to be associated with the i-th employee’s probability of quitting at time t. Each employee is in the study for Tmonths and the data comprise each employee-month between the time an employee was hired and the time that they either quit or they were censored out of the study.

We used the following explanatory variables in our model:

The employee’s age at the time they were hired.

The employee’s gender.

The organization where the employee worked. We split this category

BLM employees in the key occupations who were also identified by BLM as performing oil- and gas-related work.

Other BLM employees in the key occupations.

BOEM employees in key occupations for post-2011.

BSEE employees in key occupations for post-2011.

BOEMRE/MMS employees in key occupations through 2011. Note that since we are using time-varying covariates, this category changed starting in 2012 for any employee who was employed during the redefinition of sub-agency organizations and consequent reorganization.

Other Department of Interior employees in key occupations.

Federal government agencies other than Interior employees in key occupations.

The frequency with which an employee received an award; specifically, the number of awards in a given fiscal year per month employed (at risk) in that fiscal year. We included the following award categories:

Adjusted basic pay (salary) for the fiscal year.

Geographic location; specifically, the U.S. Census Division where the employee’s duty station was located.

A set of time dummy variables indicating the employment duration quarter for a given employee; that is, a dummy for any employee in their first quarter of employment, a dummy for any employee in their second quarter of employment and so on, up to a maximum of 47 dummies (there are 48 quarters from the start of 2003 to the end of 2014 so this allowed for 48 minus one dummy variables).

The percentage growth rate of the Standard & Poor’s 500 Energy Index, which measured the health of the private energy sector and consequent source of possible demand for federal employees in the key occupations.

Results

A detailed set of results is shown in table 4. The main results pertinent to our study were as follows:

All the awards variables except for student loan payments were significantly associated with lowering the probability of quitting. The student loan payments were significant at about the 6 percent level but we hypothesized that these loan payments are more likely to go to younger employees. This hypothesis was supported by our results when we ran a second model that included an interaction term between student loan payments and employees’ age when they were hired. In this second model, the student loan payments were significant and associated with a lower probability of quitting and the interaction term was positive, suggesting the effect on reducing the probability of quitting is greater for younger employees.

Higher adjusted basic pay (salary) was significantly associated with a lower probability of quitting with the odds ratio higher by 1.8 percent for each additional $1,000 in salary.

A faster growing private energy sector, as measured by the growth in the Standard & Poor’s 500 Energy Index, was significantly associated with a higher probability of quitting. This supports the hypothesis that key occupation employees are attracted away from federal employment when the private energy sector is performing well.

Organization results, relative to the base case; namely, key occupation employees outside Interior, the following groups had a significantly higher likelihood of quitting;

BLM employees identified by BLM as key oil- and gas-related employees.

BSEE key occupation employees.

Other (outside BOEM, BSEE, and BLM) Interior key occupation employees.

Occupation results, relative to the base case; namely, General Inspection, Investigation and Compliance occupation, the following occupations had a higher likelihood of quitting:

General Natural Resource Management and Biological Scientists.

Environmental Protection Specialists.

Appendix II: Comments from the Department of Interior

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Dan Haas (Assistant Director), John Barrett, Mark Braza, Scott Bruckner, Antoinette Capaccio, Michael Kendix, Angela Miles, and Cynthia Norris made significant contributions to this report. Also contributing to this report were David Bennett, Andrew Berglund, Ashely Chaifetz, Eric Charles, Keya Chateauneuf, Clifton Douglas, Glenn Fischer, Tom Gilbert, Paige Gilbreath, Holly Hobbs, Steven Lozano, Sarah Martin, Gloria Ross, Lillian Slodkowski, Matt Tabbert, Sarah Veale, Amy Ward-Meier, Michelle Wong, and Arvin Wu.