Human Capital: Monitoring of Safeguards and Addressing Employee Perceptions Are Key to Implementing a Civilian Performance Management System in DOD
Published: 2009-10-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 2004, the Department of Defense (DOD) began implementing the National Security Personnel System (NSPS)--a human capital system for DOD civilians. NSPS significantly redesigned the way DOD civilians are hired, compensated, and promoted. Pub. L. No. 110-181 mandated that GAO conduct reviews of the NSPS performance management system in calendar years 2008, 2009, and 2010. In this report, GAO assessed (1) the extent to which DOD has implemented certain internal safeguards to ensure the fairness, effectiveness, and credibility of NSPS, and monitored their implementation, and (2) how DOD civilian personnel perceive NSPS, and the actions DOD has taken to address those perceptions. GAO analyzed relevant documents and employee survey results, interviewed DOD officials, and conducted discussion groups with DOD employees at eight locations outside of the continental United States. Toward the end of GAO's review, both Houses of Congress passed proposed legislation that, if enacted, would terminate NSPS and require any future performance management system for DOD civilians to include certain internal safeguards
section_title: What GAO Found
paragraphs: DOD continues to take steps to implement internal safeguards as part of NSPS, but implementation of some safeguards could still be improved, and continued monitoring of all safeguards' implementation is needed. In general, DOD has taken some steps to meet the intent of each of the safeguards, and it has implemented some of the recommendations from GAO's September 2008 report. However, opportunities exist for DOD to improve implementation of some safeguards. For example, DOD has not yet evaluated the effectiveness of the training employees receive, although doing so could help DOD measure the impact of its training and its progress toward achieving agency goals. In addition, DOD has not specified in its guidance what process the components should follow to investigate and eliminate potential barriers to fair and equitable ratings. Consequently, the components may not follow a consistent approach when investigating potential barriers, which could hinder their efforts to eliminate them. Further, GAO previously noted that continued monitoring of the safeguards was needed to ensure that DOD's actions were effective. While DOD monitors some aspects of the system's implementation, it does not monitor how or the extent to which the safeguards specifically are implemented across the department. As a result, decision makers lack information that could be used to determine whether the department's actions are effective and whether the system is being implemented in a fair, equitable, and credible manner. DOD civilian personnel have mixed perceptions about NSPS, and while the department has taken some steps toward addressing employee concerns, it has not yet developed and implemented an action plan to address areas where employees express negative perceptions of the system, as GAO recommended in 2008. DOD's survey data from 2008 revealed that overall, NSPS employees responded positively about some aspects of performance management, such as connecting pay to performance, and negatively about others, such as the performance appraisal process. According to the most recent survey data, the negative perceptions of employees who worked under NSPS the longest remain largely unchanged from what was reported by GAO in 2008. Further, as GAO reported in 2008, employees and supervisors continue to express negative perceptions in discussion groups about NSPS--for example, voicing concerns about the negative impact of NSPS on employees' motivation and morale, and about the excessive amount of time spent navigating the performance management process. Such negative perceptions are not surprising given that large-scale organizational transformations often require an adjustment period to gain employees' trust and acceptance. DOD has taken some steps to address employees' perceptions of NSPS--for example, by issuing a memorandum with suggested actions the components could take to address employee concerns. However, DOD has not yet developed and implemented an action plan that fully meets the intent of GAO's 2008 recommendation. Specifically, DOD has not yet specified such things as its intended actions, who will be responsible, and the time frames for these actions. GAO continues to believe that implementing such a plan has merit.

section_title: Letter
paragraphs: In 2004, the Department of Defense (DOD) began implementing the  National Security Personnel System (NSPS)—a human capital system for  DOD civilian employees. NSPS significantly redesigned the rules,  regulations, and processes that govern the way in which civilian  employees are hired, compensated, and promoted within the department.  DOD began converting employees to NSPS in 2006 and, according to the  department, by early 2009 had converted more than 200,000 employees to  the system. However, a series of events transpired in 2009 leaving the  system’s future uncertain. Specifically, in February 2009, the Chairman of  the House Committee on Armed Services and the Chairman of the  committee’s Subcommittee on Readiness urged DOD to halt conversions  to NSPS, highlighting concerns over a lack of transparency and  widespread discontent with the system. Subsequently, in March 2009, DOD  and the Office of Personnel Management (OPM) announced that they  would suspend any further conversions of organizations to NSPS pending  the outcome of a review of the system led by the Defense Business Board.  In July 2009, the Defense Business Board presented the Secretary of  Defense with the report of its review of NSPS, which included  recommendations to initiate a reconstruction of the system and to  continue the moratorium on conversions to NSPS until reconstruction is  complete. Most recently, in October 2009, the proposed National Defense  Authorization Act for Fiscal Year 2010, which passed both Houses of  Congress, contains provisions that would terminate NSPS and convert  DOD civilian employees currently under the system to previously existing  civilian personnel systems no later than January 1, 2012.
Before the enactment of the legislation that authorized NSPS, we raised a  number of critical issues about the proposed regulations for NSPS in a  series of testimonies in 2003. Since then, we have provided congressional  decision makers with insight into DOD’s processes to design and  implement NSPS and the most significant challenges the department faces  in implementing the system. While GAO supports human capital reform in  the federal government, a key challenge is ensuring sustained and  committed leadership for such a major transformation effort. Further,  moving too quickly or prematurely to implement such systems, whether at  DOD or elsewhere, can significantly raise the risk of problems  implementing them. We have also reported that DOD and other federal  agencies must ensure that their performance management systems contain  appropriate accountability mechanisms and internal safeguards to ensure  that the systems are fair, effective, and credible.
To help inform congressional deliberations on the implementation of  NSPS, the National Defense Authorization Act for Fiscal Year 2008  mandated that GAO conduct reviews of NSPS in calendar years 2008, 2009,  and 2010 to determine the extent to which DOD has effectively  incorporated certain accountability mechanisms and internal safeguards  into the system and to assess employee attitudes toward the system. In  September 2008, we issued our first review of NSPS, and in April 2009 we  testified on that assessment before the House Committee on Armed  Services’ Subcommittee on Readiness. We used the following safeguards  to assess NSPS:  Involve employees, their representatives, and other stakeholders in the  design of the system, to include employees directly involved in  validating any related implementation of the system.
Assure that the agency’s performance management system links  employee objectives to the agency’s strategic plan, related goals, and  desired outcomes.
Implement a pay-for-performance evaluation system to better link  individual pay to performance, and provide an equitable method for  appraising and compensating employees.
Provide adequate training and retraining for supervisors, managers,  and employees in the implementation and operation of the  performance management system.
Institute a process for ensuring ongoing performance feedback and  dialogue between supervisors, managers, and employees throughout  the appraisal period, and setting timetables for review.
Assure that certain predecisional internal safeguards exist to help  achieve consistency, equity, nondiscrimination, and nonpoliticization  of the performance management process (e.g., independent  reasonableness reviews by a third party or reviews of performance  rating decisions, pay determinations, and promotions before they are  finalized to ensure that they are merit-based, as well as pay panels who  consider the results of the performance appraisal process and other  information in connection with final pay decisions).
Assure that there are reasonable transparency and appropriate  accountability mechanisms in connection with the results of the  performance management process, including periodic reports on  internal assessments and employee survey results relating to  performance management and individual pay decisions while  protecting individual confidentiality.
Assure that the agency’s performance management system results in  meaningful distinctions in individual employee performance.
Provide a means for ensuring that adequate agency resources are  allocated for the design, implementation, and administration of the  performance management system.
We initially developed a list of safeguards based on our extensive body of  work looking at the performance management practices used by leading  public sector organizations both in the United States and in other  countries. These safeguards were then merged with the list of safeguards  specified in section 9902(b)(7) of Title 5, U.S. Code, to develop the  aforementioned safeguards. It is important to keep in mind that the list of  safeguards we are using to assess DOD’s efforts is not intended to cover  all attributes of a modern, results-oriented performance management  system; therefore, other safeguards may exist. Rather, as we previously  reported, the items on the list cover possible safeguards that help ensure  those systems are fair, effective, and credible.
In responding to the congressional mandate for the 2008 review of NSPS,  we reported, in September 2008, that while DOD has taken some steps to  ensure that NSPS is fair, effective, and credible, implementation of certain  safeguards could be improved. We further reported that although DOD  civilian employees under NSPS responded positively regarding some  aspects of performance management, the department does not have an  action plan to address employees’ negative perceptions of NSPS. To  address our findings, we recommended that DOD improve the  implementation of certain safeguards and develop and implement an  action plan to address employees’ concerns about NSPS. DOD generally  concurred with our recommendations, with the exception of one— requiring a predecisional analysis of its rating results. We discuss the steps  DOD has taken to address our previous recommendations later in this  report.
To address the congressional mandate for the 2009 review of NSPS, we  established the following objectives: (1) To what extent has DOD  implemented safeguards to ensure the fairness, effectiveness, and  credibility of the NSPS performance management system and monitored  the implementation of the safeguards? (2) How do DOD civilian personnel  perceive NSPS, and what actions has DOD taken to address these  perceptions?
For our first objective, we reviewed the safeguards specified in section  9902(b)(7) of Title 5, U.S. Code, for the NSPS performance management  system, as well as other safeguards we have previously identified as key  for performance management systems. We also obtained and analyzed  regulations, guidance, and other documentation provided by officials in  DOD and the NSPS program offices of the four components—the Army,  the Air Force, the Navy, and the Fourth Estate—and interviewed  knowledgeable officials in all of these offices to obtain a comprehensive  understanding of their efforts to implement the NSPS performance  management system and the safeguards as well as the processes,  procedures, and controls used for monitoring and overseeing  implementation of the system. In addition, we visited eight DOD  organizations—two for each component—located outside the continental  United States to assess the extent to which the department has  implemented safeguards for NSPS. Because our 2008 review focused on  assessing implementation of NSPS and the safeguards at locations that  were geographically distributed throughout the United States, we focused  our efforts for this review on visiting locations outside the continental  United States. During our site visits we reviewed documents, such as pay  pool business rules, and met with the performance review authority, pay  pool managers, pay pool panel members, rating officials, the NSPS  program manager or transition team, and DOD civilian employees under  NSPS to obtain their perspectives on the extent to which the safeguards  had been implemented. The organizations we visited were selected based  on a number of factors, such as the presence of a large number or  concentrated group of civilian employees under NSPS. For our second  objective, we obtained and analyzed the results of DOD’s May 2006,  November 2006, May 2007, and February 2008 Status of Forces Survey of  Civilian Employees, which gauge employee attitudes toward NSPS, among  other things. We used these data to identify indications of movement or  trends in employee perceptions. As we reported in our first assessment of  NSPS, we reviewed the results of prior administrations of DOD’s Status of  Forces Survey of Civilian Employees and found the results to be  sufficiently reliable for the purposes of our report. For our current  assessment, we found no substantial changes in the methodology for the  2008 Status of Forces Survey of Civilian Employees. In addition, we  obtained the perspectives of DOD civilian personnel at the eight locations  we visited by conducting small group discussions with nonsupervisory and  supervisory employees under NSPS and administering a short  questionnaire. Because the locations we visited are not representative of  all DOD locations, the results from our discussions and the comments  provided are not generalizable to the entire NSPS civilian population.  However, the composition of our discussion groups was designed to  ensure that we spoke with employees from each of the four DOD  components at locations outside the continental United States. Further  details about our scope and methodology can be found in appendix I.
We conducted this performance audit from November 2008 through  September 2009 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The General Schedule (GS) is the federal government’s primary pay and  classification system for white-collar employees. Under this system,  federal employees are paid according to governmentwide rules and  procedures, and federal positions are classified according to the difficulty  and responsibility of the work performed. The GS system was created in  1949, when most federal positions involved clerical work or revolved  around the execution of established, stable processes—for example,  posting census figures in ledgers or retrieving taxpayer records from vast  file rooms. Over time, however, federal positions have become  increasingly specialized and more highly skilled. In light of this change, a  number of federal agencies have attempted to provide managers with  greater flexibility in hiring and awarding pay raises to employees by  implementing human capital initiatives, such as performance management  systems, that reward employees’ performance and contribution to the  agency’s mission. The need for human capital reform regarding these  systems has been the subject of a number of previous GAO reviews.  These reviews have noted, for example, that federal agencies must have  modern, effective, credible, and validated performance management  systems that are capable of supporting pay and other personnel decisions,  and that pay for performance works only with adequate safeguards and  appropriate accountability mechanisms in place to ensure that the  safeguards are implemented in a fair, effective, and credible manner.
In November 2003, the Congress included a provision in the National  Defense Authorization Act for Fiscal Year 2004 providing DOD with the  authority to establish a pay-for-performance management system as part  of NSPS. In April 2004, the Secretary of Defense appointed an NSPS  Senior Executive to, among other things, design, develop, and implement  NSPS. Under the Senior Executive’s authority, the PEO was established as  the central policy and program office for NSPS. The PEO’s responsibilities  include designing the human resource/pay-for-performance systems,  developing communication and training strategies, modifying personnel  information technology, and preparing joint enabling regulations and  internal DOD implementing regulations, called implementing issuances. As  the central DOD-wide program office, the PEO also directs and oversees  the four components’ NSPS program managers, who report to their parent  components and the PEO. These program managers serve as their  components’ NSPS action officers and also participate in the development,  planning, implementation, and deployment of NSPS.
Beginning in April 2006, DOD began phasing (or spiraling) civilian  employees into NSPS; however, in January 2008 the National Defense  Authorization Act for Fiscal Year 2008 prohibited the Secretary of Defense  from converting more than 100,000 employees to NSPS in any calendar  year and excluded Federal Wage System employees from coverage under  NSPS. Further, in March 2009, DOD announced that it would delay the  conversion of additional organizations to NSPS pending the outcome of a  joint review of the system by DOD and OPM. According to DOD, this  decision affected roughly 2,000 employees in organizations scheduled to  convert to NSPS during the spring of 2009. As a result of these and other  legislative changes that resulted in revisions to the NSPS regulations, the  PEO has not developed new timelines for phasing any additional civilian  employees into NSPS. As table 1 shows, according to DOD, almost 220,000  civilian employees have been phased into NSPS as of September 2009.
The NSPS performance management process is ongoing and consists of  several phases that are repeated during each annual performance cycle.  The process begins with a planning phase that involves supervisors (or  rating officials) and employees working together to establish performance  plans. This includes developing job objectives—the critical work  employees perform that is aligned with their organizational goals and  focused on results—and identifying contributing factors—the attributes  and behaviors that identify how the critical work established in the job  objectives is going to be accomplished (e.g., cooperation and teamwork).  After the planning phase comes the monitoring and developing phase,  during which ongoing communication between supervisors and employees  occurs to ensure that work is accomplished; attention is given to areas  that need to be addressed; and managers, supervisors, and employees have  a continued and shared understanding of expectations and results. In the  rating phase, the supervisor prepares a written assessment that captures  the employee’s accomplishments during the appraisal period. In the  final—or reward—phase, employees should be appropriately rewarded or  compensated for their performance with performance payouts. During this  phase, employee assessments are reviewed by multiple parties to  determine employees’ ratings and, ultimately, performance payouts.
The performance management process under NSPS is organized by pay  pools. A pay pool is a group of employees who share in the distribution of  a common pay-for-performance fund. The key participants that make up  pay pools are the employee, supervisor, higher-level reviewer, pay pool  panel, pay pool manager, performance review authority, and, in some  instances, the sub-pay pool, as shown in figure 1.
Within a pay pool, each participant has defined responsibilities under the  performance management process:    Employees are encouraged to be involved throughout the performance  management cycle. This includes initially working with their  supervisors to develop job objectives and identify associated  contributing factors; identifying and recording accomplishments and  results throughout the appraisal period; and participating in interim  reviews and end-of-year assessments, for example, by preparing a self- assessment of their performance during the annual appraisal period.
Supervisors (or rating officials) are responsible for effectively  managing the performance of their employees. This includes clearly  communicating performance expectations; aligning performance  expectations and employee development with organization mission  and goals; working with employees to develop written job objectives  that reflect expected accomplishments and contributions for the  appraisal period and identifying applicable contributing factors;  providing employees meaningful, constructive, and candid feedback  relative to performance expectations, including at least one  documented interim review; making meaningful distinctions among  employees based on performance and contribution; and providing  recommended ratings of record, share assignments, and payout  distributions to the pay pool.
The higher-level reviewer, typically the rating official’s supervisor, is  responsible for reviewing and approving job objectives and  recommended employee assessments. The higher-level reviewer is the  first step in ensuring consistency of ratings because this individual  looks across multiple ratings.
The pay pool panel (or, in some cases, the sub-pay pool panel) is a  board of management officials who are usually in positions of line  authority or in senior staff positions with resource oversight for the  organizations, groups, or categories of employees making up the pay  pool membership. The primary function of the pay pool panel is the  reconciliation of ratings of record, share distribution, and payout  allocation decisions. For example, the pay pool panel may adjust a  supervisor’s recommended rating of record in order to help ensure the  equity and consistency of ratings across the pay pool.
Each pay pool has a manager who is responsible for providing  oversight of the pay pool panel. The pay pool manager is the final  approving official of the rating of record. Performance payout  determinations may be subject to higher management review by the  performance review authority or equivalent review process.
Finally, the performance review authority provides oversight of several  pay pools and addresses the consistency of performance management  policies within a component, major command, field activity, or other  organization as determined by the component.
section_title: DOD Continues to Make Progress, but Implementation of Some Safeguards Could Be Improved, and Continued Monitoring of the System’s Implementation, Including the Safeguards, Is Needed
paragraphs: DOD continues to take some steps to implement each of the safeguards we  reported on in September 2008. However, opportunities exist to  continually involve employees in the system’s implementation, and  implementation of some safeguards—for example, providing effective  training—could be improved. Also, we previously reported that continued  monitoring of the safeguards was needed to help ensure that DOD’s efforts  were effective as implementation of NSPS proceeded. However, we found  that while DOD monitors some aspects of the implementation of NSPS, it  does not monitor how the safeguards specifically are implemented across  the department. Because DOD does not monitor the safeguards’  implementation, decision makers in DOD lack information that could be  used to determine whether the department’s actions are effective and  whether the system is being implemented in a fair, equitable, and credible  manner.
subsections: section_title: Implementation of Some Safeguards Could Be Improved
subsections: section_title: Involve Employees in the Design and Implementation of the System
paragraphs: DOD has taken a number of steps to involve employees and stakeholders  in the design and implementation of NSPS. In our September 2008 report,  we noted that DOD solicited comments from employees and unions  representing DOD employees during the design of NSPS. We also noted  that DOD involved employees during the implementation of the system in  workshops and other efforts aimed at refining the system’s performance  factors. DOD continues to take such steps. For example:    DOD solicited comments from employees and unions on the system’s  final rule, which was published in the Federal Register in September  2008. According to the Federal Register, the final regulations, which  became effective in November 2008, include revisions based on 526  comments submitted during the public comment period and on  comments from 9 of the department’s 10 unions with national  consultation rights.
DOD involved employees in efforts to improve the usability of the  automated tools that support the NSPS performance and pay pool  management processes. Specifically, the PEO and the department’s  Civilian Personnel Management Service held a series of meetings with  employees, rating officials, pay pool managers, and human resource  practitioners in early 2008 to address concerns regarding the usability  of the automated tools. These meetings allowed the department to  gather requirements for the next version of the NSPS automated tools  based on lessons learned and user input. Subsequently, DOD  established six separate working groups to develop and evaluate the  requirements for each of its automated tools. In addition, DOD  initiated separate efforts to enhance the usability of the Performance  Appraisal Application—the DOD-wide tool for employee self- assessments and appraisals. Specifically, the contractor that developed  the Performance Appraisal Application enlisted the assistance of  software usability experts to evaluate the tool and recommend  changes that would enhance users’ experience with it. As a part of this  effort, the contractor observed and worked with employees and rating  officials to identify changes that could be made to the Performance  Appraisal Application to make it more user-friendly. DOD also tested  the functionality and usability of the enhancements that were made to  the Performance Appraisal Application with over 300 users.
DOD has taken steps to involve the components in the implementation  of NSPS through biweekly conference calls held at key phases of the  performance management process. According to the PEO, during these  calls, PEO and Civilian Personnel Management Service representatives  discuss topics submitted by the components, respond to questions  regarding such things as NSPS policy and the system’s automated  tools, and share lessons learned with participants. Further, according  to the PEO, these conference calls allow participants to address  systemic problems through feedback shared between different levels  of the organization.
At the locations we visited outside the continental United States, we found  that some steps were generally being taken to involve employees in the  implementation of NSPS as well. For example, officials at each of the eight  locations reported that organizations identified lessons learned that were  generally based upon employee feedback gathered following the mock pay  pool. According to these officials, lessons learned were used to make  changes to, among other things, training materials, business rules, and the  use of job objectives and contributing factors. For example, two locations  limited the number of contributing factors employees should use in their  performance plans based upon lessons learned, while one location  responded to employee feedback regarding a lack of time to devote to  NSPS by mandating that employees be allowed to take time to complete  NSPS training.
While DOD has taken a number of steps to involve employees in the design  and implementation of NSPS thus far, as stated above, we note that one  way the department could continue to involve employees as  implementation of the system proceeds is to develop and implement an  action plan to address employees’ perceptions of NSPS, as we  recommended in our first assessment of NSPS. However, DOD has not yet  done so, which we discuss further in the second objective of this report.
section_title: Link Employee Objectives to the Agency’s Strategic Goals and Mission
paragraphs: DOD continues to take steps to link employee job objectives to the  agency’s strategic goals, mission, and desired outcomes. As we noted in  our 2008 report, DOD’s automated tool for employee self-assessments and  appraisals—the Performance Appraisal Application—provides a  designated area for the mission of the employee’s command to be inserted  as a guide while employees compose their job objectives and self- assessments. In May 2009, DOD published its evaluation of NSPS for 2008,  entitled National Security Personnel System (NSPS) – 2008 Evaluation  Report, which included an evaluation of employee performance plans to  determine the extent to which employee job objectives are aligned with  higher-level organizational goals and thus ensure that employee  performance contributes to the achievement of organizational success.  The evaluation included 240 employee performance plans encompassing a  range of job series, pay schedules, pay bands, and organizations within  each of the four components. The evaluation found that 95 percent of the  objectives were strongly aligned to higher-level goals and demonstrated a  clear, direct, and strong linkage to the organizational mission or relevant  strategic goals.
During our site visits, we found that the organizations were taking steps to  ensure that employees understood how their job objectives link to the  organization’s strategic goals. This was generally accomplished through  documentation requirements in the Performance Appraisal Application  and reinforced during employees’ discussions with their supervisors. Some  organizations have taken additional steps to ensure that employee job  objectives link to the organization’s strategic goals. For example, at one  location we visited the commanding general issued a memorandum to  managers emphasizing the importance of ensuring that employee  objectives are linked to the command’s mission and objectives and that  employees understand their roles in achieving those objectives. However,  officials at five locations also reported that employees do not always  understand this linkage.
section_title: Training and Retraining in the System’s Implementation and Operation
paragraphs: DOD continues to take steps to provide employees with required and other  training on the implementation and operation of the NSPS performance  management system, but has not yet evaluated the effectiveness of the  training that it provides. In our September 2008 report, we noted that DOD  encouraged employees who were transitioning to NSPS to receive training  that covered skills and behaviors necessary to implement and sustain  NSPS; foster support and confidence in the system; and facilitate the  transition to a performance-based, results-oriented culture. DOD and each  of the components continue to take such steps to provide employees with  required and other training on the system, including introductory training  for employees converting to NSPS and sustainment training for employees  already under the system. While the components are responsible for  providing employees with training on the NSPS performance management  system, the PEO supports their efforts by offering a variety of  departmentwide training courses and other materials. For example, Web- based training modules that the PEO has developed, such as its NSPS 101  and iSuccess courses (see fig. 2 for sample screen shots), provide  employees with basic knowledge about NSPS and performance  management principles in general, and are used by employees to develop  their job objectives. As another example, the PEO developed training  guides to educate employees on changes to the NSPS classroom materials  resulting from the revised NSPS regulations and implementing issuances.  The PEO also developed a Web site for accessing NSPS learning materials,  resources, and other tools. In addition, we found that the Air Force has  begun incorporating training on NSPS as a normal part of its operations  and is working to embed NSPS topics in the regular training provided to  Air Force civilians and servicemembers.
Although DOD and the components continue to take steps to provide  employees with training on NSPS, the department has not yet evaluated  the effectiveness of the training provided. We previously reported that it is  increasingly important for agencies to measure the real impact of training  and thus evaluate their training—for example, by establishing clear goals  about what the training is expected to achieve along with agreed-upon  measures or performance indicators to ascertain progress toward  achieving those goals. DOD has outlined the fundamental requirements,  or goals, of the training that it provides to employees on NSPS.  Specifically, DOD has noted that for NSPS, a training program must be  implemented that enables employees to understand better how to succeed,  and enables supervisors to communicate performance expectations to  their employees, provide feedback to them on their performance against  these expectations, and tell them what steps they can take to improve  their performance and competencies and manage their careers. However,  while DOD has undertaken efforts to understand employees’ perceptions  of its training, the department has not yet evaluated the effectiveness of  the training that it provides. For example, in early 2009 the PEO conducted  what PEO officials describe as an ad hoc study of training needs. The  PEO’s study consisted of conducting sensing sessions with 120 human  resource practitioners at 11 locations across the department. According  to the PEO, the purpose of these sessions included obtaining feedback on  existing NSPS learning products and support and exploring options for  next generation products. While the PEO’s study identified some needed  improvements, it does not constitute an evaluation of the department’s  training—for example, because it did not assess the department’s progress  toward achieving the goals specified for the training.
As another example, DOD’s 2008 evaluation report also highlighted  deficiencies with the department’s training on NSPS. Specifically, the  report notes that without effective communication and training, NSPS  cannot achieve its goal of being a credible and trusted system. Further,  three of the report’s six key recommendations focus on the need to  enhance the effectiveness of the training provided to employees on NSPS:  (1) provide more training on the performance management system; (2)  provide high-level training for employees and supervisors that explains the  pay pool process; and (3) hold mock pay pool panels, which serve as  refreshers for continuing panel members and as training for new members.  However, like the PEO’s study, DOD’s 2008 evaluation report does not  constitute an evaluation of the department’s training—for example,  because it did not include an in-depth assessment of DOD’s training and  also did not assess the department’s progress toward achieving the goals  for the training, per agreed-upon measures or performance indicators. Of  the components, we found that only the Army has taken some steps to  evaluate the training it provides to employees on the system. Specifically,  the Army assesses the adequacy of NSPS training during on-site reviews  that it conducts as part of its implementation of the system. The Army  conducted three such assessments during 2008 and an additional four such  assessments during 2009.
With respect to our site visits, although we found that each of the eight  locations provided training on NSPS to employees, officials at each  location also expressed concerns over the effectiveness of the training  provided. For example, officials at seven locations told us that additional  training was needed on writing job objectives or employee self- assessments under the system, while other officials noted that additional  training was needed on the pay pool process, use of the system’s  automated tools, and how the reconsideration process works. Similarly,  officials at two locations raised concerns that the training they received  did not provide them with the skills they needed to use the Performance  Appraisal Application. For example, officials told us that they received  training too early and had either forgotten it or the training had become  outdated by the time they actually used the Performance Appraisal  Application. Further, some program officials raised concerns about their  organizations’ ability to provide employees with adequate training on the  system when the employees are located outside the continental United  States. For example, program officials at one location in Germany  reported challenges providing employees located in the field with training  on NSPS because they lack the resources to send NSPS trainers to those  locations. However, until DOD evaluates its training, it will not be able to  determine whether the training meets the needs of its employees or the  department is making progress toward achieving the goals it specified for  the training.
section_title: Ongoing Performance Feedback and Dialogue between Supervisors and Employees
paragraphs: DOD continues to take steps to ensure that employees receive timely  performance feedback that is meaningful, constructive, and in accordance  with the department’s requirements. In our first assessment of NSPS we  noted that DOD’s implementing issuances required at least one  documented interim performance review and an annual performance  appraisal and that the Performance Appraisal Application allowed  supervisors and employees to document these feedback sessions. Since  then, DOD has taken additional steps to enhance the Performance  Appraisal Application by modifying the tool to allow supervisors and  employees to identify where they are in the performance appraisal process  and help them accomplish required actions in a timely manner.
During our site visits, officials at all eight locations told us that NSPS helps  ensure the occurrence of performance feedback between employees and  supervisors. For example, officials noted that use of the Performance  Appraisal Application encourages employee feedback by allowing  employees to document and track feedback sessions, and that NSPS  encourages direct discussions about performance-related issues, such as  developing effective job objectives and establishing performance  expectations.
section_title: System to Better Link Individual Pay to Performance in an Equitable Manner
paragraphs: DOD continues to take steps to better link individual pay to performance  as well. As we noted in our 2008 report, the NSPS performance  management system uses a multirating system of five rating categories—of  which the lowest rating is “1” (unacceptable performance) and the highest  rating is a “5” (role model performance)—that allows distinctions to be  made in employee performance and therefore compensation. Since then,  DOD added details to the NSPS regulations to facilitate uniform, equitable  practices across the department that accord with merit system principles.  These include specifying specific share assignment ranges, rounding rules  for converting raw performance scores to ratings, and formulas for  determining share value and the calculation of performance payouts under  NSPS. DOD also clarified that a common share value should apply  throughout an entire pay pool. According to the PEO, these changes, in  addition to the higher-level review of performance expectations,  recommendations for ratings of record, share assignment, and payout  distribution have helped to promote a more equitable method for  appraising and compensating employees.
However, during our site visits, officials at seven of the eight locations told  us that they saw the potential for factors other than performance to  influence employees’ ratings, such as the quality of employees’ and  supervisors’ writing skills. For example, rating officials at one location  commented that NSPS does not reward employees based on their  performance but rather on how well employees and supervisors can  communicate in writing what the employee achieved during the  performance management cycle. Similarly, at another location, a pay pool  panel member told us that the paperwork submitted to the panel by both  the employee and the supervisor must be of very high quality because it  can be difficult to defend a high rating recommended for an employee if  the assessments are poorly written. DOD’s 2008 evaluation report also  highlighted concerns from employees and managers over the department’s  success in linking pay to performance under NSPS. For example, DOD’s  report noted that while some employees believed that they saw some level  of pay for performance under NSPS, others were uncertain. Further,  DOD’s report also noted that some managers observed that the quality of  written assessments contributed significantly to the way in which ratings  were substantiated.
section_title: Means to Ensure That Adequate Agency Resources Are Allocated for System Design, Implementation, and Administration
paragraphs: We found that DOD also continues to take steps to ensure that adequate  agency resources are allocated to NSPS. According to 5 U.S.C. §  9902(e)(4), to the maximum extent practicable, for fiscal years 2004  through 2012, the overall amount of money allocated for compensation of  civilian employees in organizations under NSPS shall not be less than the  amount that would have been allocated under the GS system. To meet the  requirements of 5 U.S.C. § 9902(e)(4), DOD’s implementing issuances  require that the components certify in writing to the Deputy Secretary of  Defense through the Under Secretary of Defense for Personnel and  Readiness that the amount expended for NSPS performance-based pay  increases is no less than what would have been expended had these  positions not been converted into NSPS. Following the 2008 NSPS  performance management cycle, each of the components certified that it  met this requirement.
DOD also continues to capture NSPS implementation costs, and for fiscal  year 2008, the PEO reported that NSPS implementation costs were about  $61.8 million. According to the PEO, continuing implementation of NSPS  will result in some additional program implementation costs, such as for  training for NSPS, conducting outreach to employees and others, and  improving the system’s automated tools. However, the PEO estimates that  once the conversion of all non-bargaining unit employees is complete, the  system’s implementation costs will decrease significantly unless there is a  decision to convert bargaining unit employees. Thereafter, according to  the PEO, the cost of administering NSPS will be no different than that of  any other personnel system.
section_title: Predecisional Internal Safeguards to Determine if Rating Results Are Consistent, Equitable, and Nondiscriminatory
paragraphs: In our 2008 report, we recommended that DOD take steps to better ensure  the consistency and equity of the performance management process by  requiring a third party to perform predecisional demographic and other  analysis of the pay pool results. DOD did not concur with our  recommendation, stating that its postdecisional analysis of the rating  results was useful for identifying barriers and any needed corrective  action. DOD also stated that if the information obtained from a  postdecisional demographic analysis demonstrated that the results were  not fair or equitable, for any reason, the process used to achieve those  results could be examined with the intent to identify and eliminate barriers  to a fair and equitable outcome.
However, our review of the postdecisional analyses that the PEO and each  of the components completed following the 2007 NSPS performance  management cycle and the analyses that each of the eight organizations  we visited for our review completed following the most recent  performance management cycle in 2008 found these analyses were  inconsistent, did not always include an analysis of the rating results by  demographics, and were generally conducted at too high a level to provide  decision makers with sufficient information to identify potential barriers  or corrective actions. For example, we found that following the 2007  performance management cycle, the PEO analyzed the rating results for  more than 100,000 employees by select demographic groups, such as race,  gender, ethnicity, age, veteran status, and targeted disability, but limited  its analysis to the aggregate data from its pay pools. That is, the PEO did  not analyze the rating results at the level where decisions are made—in the  case of NSPS ratings and payouts, the pay pool level. Similarly, in  analyzing the postdecisional analyses that the components conducted  following the 2007 performance management cycle, we found  inconsistencies in their approaches, primarily because the components  were allowed to develop their own approaches for conducting this  analysis. For example, only the Army and Fourth Estate included an  analysis of the rating results by demographics as part of their respective  postdecisional analyses. However, we also found that neither the Army’s  nor the Fourth Estate’s demographic analysis of the ratings provided  decision makers with sufficient information to identify possible barriers or  corrective actions that could be taken to address such barriers. As with  the PEO, this problem results because neither the Army nor the Fourth  Estate conducted its analysis at the pay pool level. Of the eight locations  we visited for our review, we found that only one organization’s  postdecisional analysis following the 2008 performance management cycle  included an analysis of its ratings results by demographics.
Since we issued our 2008 report, DOD has taken steps to promote a degree  of consistency in its postdecisional analysis of NSPS ratings and payout  data. Specifically, in December 2008, DOD revised its implementing  issuance to require the heads of DOD components to conduct an annual  analysis of NSPS performance ratings and payouts for subordinate  elements, and issue guidance to lower echelons and otherwise act to  identify, examine, and remove barriers to similar rating and payout  potential for demographic and other groups in the workforce. Further, in  May 2009, the PEO issued guidance, entitled Guidance for Conducting  Annual Analysis of NSPS Performance Ratings and Payouts, in order to  provide the components with a framework and suggested approaches for  conducting their annual analysis and to serve as a starting point for  identifying and examining barriers. For example, the guidance notes that  the NSPS performance management system’s rating and payout process  has four main outcomes—the rating of record, number of shares awarded,  payout, and the distribution of the payout—and that each outcome should  be reviewed. The guidance also notes that analysis is best done at the level  where decisions are made—in the case of NSPS ratings and payouts, the  pay pool level. Further, the guidance  expresses the expectation that as the components conduct their  analyses, changes and improvements to the guidance will be made;  is careful to ensure that the components understand base parameters  for conducting the analysis so it is conducted in a manner that is  methodologically sound;  encourages consultation with experts, such as statisticians and human  resources researchers, to assist with determining the most suitable  analytical models to employ, the statistical tools to utilize, and the  standards to adopt in relation to understanding, measuring, and  reporting significant findings; and    makes responsibility for conducting the analysis a shared  responsibility between various offices, including the components’  legal, equal employment opportunity, and human resources offices, but  notes that the components should consider tasking their Office of  General Counsel or Office of the Judge Advocate General, whose staff  are well positioned to ensure that the components are in compliance  with applicable statutes, regulations, and policies, with primary  responsibility for conducting the analysis and ensuring that adequate  resources are provided in support of the function.
While issuance of the May 2009 guidance represents a noteworthy step, the  guidance does not address all steps suggested by the Equal Employment  Opportunity Commission for identifying and addressing potential barriers  to fair, consistent, and equitable ratings. The Equal Employment  Opportunity Commission’s Instructions to Federal Agencies for EEO  Management Directive 715 provides guidance that agencies can use in  identifying and addressing potential barriers. The instructions state that  “barrier identification and elimination is the process by which agencies  uncover, examine, and remove barriers to equal participation at all levels  of the workforce.” Management Directive 715 further states that “where it  is determined that an identified barrier serves no legitimate purpose with  respect to the operation of an agency, this Directive requires that agencies  take immediate steps to eliminate the barrier.” In conducting their  analysis, the components’ data may uncover barriers or other potential  problems. However, understanding why the barrier or problem exists, or  what to do to address it, may require that the components take additional  steps. To identify and eliminate potential barriers, the directive outlines a  four-step process: (1) analyzing workforce data to identify potential  barriers; (2) investigating actual barriers and their causes; (3) eliminating  barriers, which includes devising a plan for improvement and developing  overall objectives for barrier elimination, with corresponding action items,  responsible personnel, and target dates; and (4) assessing the success of  the plan. The PEO’s guidance aims to promote a degree of uniformity and  standardization in conducting postdecisional analyses. However, the  PEO’s guidance does not specify what process the components should  follow to investigate potential barriers and their causes, nor does it specify  a process for eliminating barriers that are found. By not specifying such  steps in its guidance, the components may not follow a consistent  approach when investigating barriers, which could hinder their efforts to  eliminate them.
While not predecisional, we recognize that DOD’s approach does provide  some benefits, some of which are similar to those of a predecisional  analysis. For example, DOD’s approach lays out a method of analyzing  ratings that would address some of the merit principles in 5 U.S.C. §  2301—for example, that employees should receive fair and equitable  treatment in all aspects of personnel management and that employees  should be protected against arbitrary action, personal favoritism, or  coercion for partisan political purposes. However, as stated previously,  because DOD does not specify what process the components should  follow to investigate and eliminate potential barriers, the components may  not follow a consistent approach, which could hinder their efforts to  ensure fair, consistent, and equitable ratings.
section_title: Reasonable Transparency of the System and Its Operation
paragraphs: While DOD continues to take steps to ensure a reasonable amount of  transparency in its implementation of NSPS, concerns about the overall  transparency of the system continue to exist. To improve the transparency  of the NSPS performance management system, our September 2008 report  recommended that DOD require commands to publish the final rating  results to employees. DOD concurred with our recommendation and, in  November 2008, amended its NSPS regulations and implementing  issuances to require commands to publish the final rating results to  employees. Under DOD’s revised guidance, performance review  authorities are required to communicate the general pay pool results to the  NSPS workforce in writing. At a minimum, this includes the number of pay  pools (if aggregate pay pool results are necessary), the number of  employees rated, the rating and share distributions, the average rating, the  average share assignment, the share value (or average share value), and  the average payout expressed as a percentage of base salary. At the eight  locations we visited, we found that each of pay pools shared this  information with employees following the 2008 NSPS performance  management cycle.
DOD continues to take other steps to ensure a reasonable amount of  transparency of the NSPS performance management system. In May 2009,  the PEO launched “NSPS Connect,” a centralized Web portal for  employees, managers, and others to access NSPS products, such as online  training courses, fact sheets, tips sheets, and information on the system’s  automated tools. The PEO also continues to take steps to collect and share  lessons learned on the department’s experiences implementing NSPS and  facilitate lessons learned briefings with the components following each  performance management cycle. Further, the PEO continues to report  periodically on internal assessments and employee survey results relating  to the NSPS performance management system. For example, in May 2009,  the PEO published the results of its 2008 evaluation of NSPS. Similarly, as  we previously reported, DOD posts the results of its survey of civilian  employees on a Web site that is accessible to DOD employees,  supervisors, and managers.
Officials at each of the eight locations we visited told us that efforts were  being made to help ensure transparency of the NSPS performance  management system and the rating process. According to officials, among  the steps being taken to help ensure transparency, for example, were  establishing multiple communities of practice, disseminating business  rules and other guidance on NSPS to employees and managers under the  system, and publishing monthly newsletters on NSPS. At seven of the  locations, officials told us that town hall meetings were held to keep  employees informed of NSPS-related developments, and officials at six  locations told us that mock pay pool panels were held to show employees  how the pay pool process works.
However, our site visits revealed some concerns about the overall  transparency of the system. For example, at three locations officials  expressed concerns over a lack of transparency with regard to their pay  pools’ business rules, indicating that their business rules either had not  been published or were published well after the performance management  cycle had begun. At one location, pay pool panel members told us that  though it was 6 months into the current performance management cycle,  they did not yet have copies of their business rules. Similarly, rating  officials at four locations told us that they did not understand what  constituted a rating of “4” because neither their pay pools nor business  rules provided clear criteria.
section_title: Meaningful Distinctions in Individual Employee Performance
paragraphs: Although our site visits revealed concerns over the extent to which  meaningful distinctions in individual employee performance are being  made under NSPS, DOD has taken some steps that include addressing a  recommendation we made in our 2008 report aimed at encouraging the use  of all available rating categories. Specifically, at all eight locations we  visited, officials told us that they did not believe NSPS was being  implemented in a manner that encouraged meaningful distinctions in  individual employee performance. For example, officials at four locations  told us that they were hesitant to give ratings lower than a “3,” and  officials at six locations told us that they believed that there was a forced  distribution of the ratings or manipulation of the ratings in order to  achieve a predetermined quota. Further, at one location we found that  management stressed the importance of maintaining employee share value  at above 2.0, which would result in a higher payout, regardless of the  employee’s rating. According to the PEO, any effort to limit share value is  a roundabout way to establish preset limits on ratings and would  constitute forced distribution, which the NSPS regulations prohibit.  However, in response to concerns about the potential for the forced  distribution of performance ratings under NSPS, in April 2009 the  department posted to its NSPS Web site a fact sheet emphasizing that the  forced distribution of ratings is prohibited under NSPS and describing how  meaningful distinctions in performance are made under the system. DOD’s  fact sheet provides guidance specifying what constitutes the forced  distribution of ratings, why the forced distribution of ratings is prohibited,  how use of standard performance indicators minimizes the potential for  individual bias or favoritism, and how organizations can best apply this  information when rating and rewarding employee performance under  NSPS. DOD has also noted that if employees believe their rating did not  result from meaningful distinctions or are unfair, the system affords them  the opportunity to challenge their ratings through a formal process known  as reconsideration. See appendix II for further information on the NSPS  reconsideration process.
section_title: Although DOD Monitors Some Aspects of the Implementation of NSPS, It Does Not Monitor How the Safeguards Specifically Are Implemented across the Department
paragraphs: Within DOD, both the PEO and the components are responsible for  monitoring the implementation of NSPS. As part of its efforts to monitor  the implementation of NSPS, the PEO conducts broad, annual evaluations  of NSPS to determine whether the system is on track to achieve certain  goals, or key performance parameters, by, among other things,  monitoring employee perceptions of the system using DOD’s survey of  civilian employees. In May 2009, the PEO published its first evaluation of  NSPS, which focused on determining whether NSPS, as implemented in  spiral 1 organizations, was on track to achieve some of the goals specified  by DOD and if any improvements were needed. While some of DOD’s  goals for the system lend themselves to comparisons to the safeguards— for example, one of its goals is ensuring a credible and trusted system  (which could align with the transparency safeguard)—PEO officials stated  that to date, their focus has been limited to understanding how the  components have generally implemented NSPS and not on monitoring or  assessing the components’ implementation of the safeguards.
With respect to the components, DOD’s implementing issuances state that  the heads of DOD components are accountable for the manner in which  management in their organizations carries out NSPS policy, procedures,  and guidance. However, we found that only the Army has taken steps  similar to the PEO to assess whether it is on track to achieve DOD’s goals  for the system—the other components have not done so. Furthermore,  none of the components monitors how the safeguards specifically are  implemented within their organizations because there is no requirement to  do so.
We have previously reported that transitioning to a more performance- oriented pay system is a huge undertaking that requires constant  monitoring. Further, in our 2008 assessment of NSPS, we noted that  continued monitoring of the safeguards was needed to ensure that DOD’s  actions were effective as implementation of NSPS proceeded. While DOD’s  efforts to date have helped provide decision makers with some  information on how the department is implementing NSPS, including  potential areas for changes or improvements, they do not provide decision  makers in DOD and the Congress with information to determine whether  the safeguards specifically have been implemented effectively. Without  monitoring the safeguards’ implementation, decision makers in DOD and  the Congress lack information that could be used to determine whether  the department’s actions are effective and whether the system is being  implemented in a fair, equitable, and credible manner. For example, in  conducting our review, we identified some issues related to the  implementation of the safeguards that illustrate the need to monitor their  implementation. These include the following:    Ensuring that adequate agency resources are allocated for the  system’s design, implementation, and administration. We found  that each of the components generally lacks visibility over the reasons  why organizations have supplemented their pay pool funds. For  example, Air Force NSPS program officials told us that for the 2009  payout, while they knew that 8 of the Air Force’s 18 major commands  supplemented their pay pool funds, they did not know the specific  reasons why. According to PEO officials, organizations might elect to  supplement their pay pool funds for a variety of reasons—for example,  to recruit or retain employees or to compete with other organizations  for talent. However, because they do not understand the reasons why  pay pools supplement their pay pool funding—which could help DOD  and the components understand the extent to which adequate  resources have been allocated to the system—decision makers cannot  identify or assess any trends in these practices. Indeed, DOD’s 2008  evaluation report notes that some employees in organizations that  supplemented their pay pools’ funding questioned whether the higher  funding levels could be sustained over the long term.
Ensuring reasonable transparency of the system and its  operation. We found evidence that three of the pay pools we visited  deviated from their business rules during the last performance  management cycle, indicating a lack of transparency of the  performance management process in some instances. DOD’s guidance  states that a pay pool’s business rules are the guiding principles or  ground rules that are used throughout the pay pool process, that pay  pool panels should establish these principles and hold one another  accountable for following them, and that a pay pool’s policies—which  would include its business rules—will be made available to employees  before the end of the performance cycle. However, at three of the  locations we visited, we found evidence that the pay pool had deviated  from its business rules during the course of the last performance  management cycle. For example, at one location the pay pool’s  business rules required all recommended ratings be reviewed, noting  that the pay pool panel will ensure that all employees receive  appropriate consideration and that ratings are fair and consistent.  However, officials we spoke with at that location told us that they did  not review all recommended ratings in accordance with their business  rules. Rather, only the recommended ratings of “4” or “5” were  reviewed. As another example, we found evidence that a pay pool at  another location used different criteria than what was specified in its  business rules for allocating the number of shares to employees.  According to component-level NSPS program officials, in order to  ensure transparency of the system, pay pools should not deviate from  their business rules once those rules are published. However, none of  the components requires its pay pools to notify it when such an event  occurs, or of the reasons why, though doing so could help provide  decision makers with information on the extent to which pay pools are  implementing the system in a manner that is transparent to employees.
section_title: Although DOD Civilian Employees under NSPS Have Mixed Views about the System, DOD Has Not Yet Developed and Implemented a Plan to Address Employees’ Negative Perceptions of Some Aspects of the System
paragraphs: DOD civilian personnel have mixed perceptions about NSPS, and although  DOD has taken some steps toward addressing employees’ concerns, it has  not yet developed and implemented an action plan to address areas where  employees express negative perceptions of the system. DOD’s most recent  survey of civilian employees reveals that NSPS employees have mixed  perceptions about NSPS. The responses to questions specifically asking  about NSPS show positive perceptions about some aspects of  performance management under NSPS, including connecting pay to  performance, but show negative perceptions about other aspects of  performance management, such as the appraisal process. Further, the  most recent data indicate that the perceptions of those employees who  have worked under NSPS the longest appear to have remained largely  unchanged from the negative perceptions we reported in 2008. Moreover,  during discussion groups we held at eight locations outside the continental  United States, civilian employees and supervisors expressed consistent  concerns and negative perceptions of NSPS that are similar to those  identified in our 2008 report, although they also identified positive aspects  of the system. DOD has taken some steps to address employees’ negative  perceptions of the system; however, the department has yet to develop  and implement an action plan that meets the intent of our prior  recommendation because it does not specify such things as the actions  DOD intends to take, who will be responsible for taking the action, and the  timelines for doing so. We continue to believe that implementing such an  action plan is important and note that doing so would be a step that DOD  could take to involve employees in the system’s implementation—which is  one of the safeguards we previously discussed.
subsections: section_title: NSPS Employees Express Mixed Perceptions about the System
subsections: section_title: NSPS Employees in All Spirals Have Positive Views of Some Aspects of Performance Management but Negative Views of Other Aspects
paragraphs: According to DOD’s most recent survey data, some NSPS employees  recognize that positive aspects of performance management, such as  connecting pay to performance, exist under the system. For example, as  shown in table 2, DOD’s survey data for 2008 indicate that an estimated 38  percent of NSPS employees agree that differences in their performance are  recognized in meaningful ways, as compared with an estimated 33 percent  of non-NSPS employees. Further, an estimated 42 percent of NSPS  employees agree that pay raises depend on how well employees perform  their jobs, as compared with an estimated 25 percent of non-NSPS  employees. When asked about how poor performers are dealt with, an  estimated 34 percent of NSPS employees, versus an estimated 27 percent  of non-NSPS employees, agreed that steps are taken to deal with poor  performers.
In comparison, an estimated 47 percent of non-NSPS employees, as  compared with an estimated 44 percent of NSPS employees, agreed that  their current performance appraisal system motivates them to perform  well. Further, an estimated 34 percent of non-NSPS employees, as  compared with an estimated 29 percent of NSPS employees, agreed that  their performance appraisal system improves organizational performance.  Table 3 shows additional comparisons between NSPS and non-NSPS  employee responses to questions about performance appraisals.
In our first assessment of NSPS, we reported that the results of DOD’s  Status of Forces Survey of Civilian Employees indicated that the  perceptions of employees who had been under the system the longest had  become more negative on questions related to performance management.  However, the results of DOD’s most recent administration of the survey in  2008 indicate that spiral 1.1 employee perceptions are about the same as  the May 2007 survey, as shown in table 4. For example, from the  November 2006 through February 2008 administrations of DOD’s survey,  the percentage of spiral 1.1 employees that agreed that they understood  what they had to do to be rated at a different performance level declined  from an estimated 59 percent in November 2006 to an estimated 53 percent  in May 2007, then remained largely unchanged in February 2008 at an  estimated 54 percent.
In addition, when asked about the overall impact that NSPS will have on  personnel practices in DOD, spiral 1.1 employees’ perceptions have  become significantly more negative since first converting to NSPS in 2006,  but showed little change between the May 2007 and February 2008  surveys. Specifically, the results of the 2008 survey indicate that that an  estimated 22 percent of spiral 1.1 employees responded that the overall  impact of NSPS on the department’s personnel practices would be  positive, as compared to an estimated 23 percent in May 2007 and an  estimated 25 percent in November 2006. Table 5 shows a comparison of  spiral 1.1 employee responses over time about the overall impact of NSPS  on personnel practices in DOD.
section_title: NSPS Employees and Supervisors in Our Discussion Groups Expressed Consistent Concerns
paragraphs: As with our first review of NSPS, DOD civilians in our discussion groups at  locations outside the continental United States continue to express wide- ranging but consistent concerns about the NSPS performance  management system. Although the results of our discussion groups are not  generalizable to the entire population of DOD civilian employees, the  themes that emerged provide valuable insight into employees’ perceptions  about the implementation of NSPS thus far.
Specifically, during these discussion groups, participants at six locations  told us that they were initially optimistic about the intent of NSPS and the  concept of pay for performance. Further, some participants indicated that  they remain positive about the amount of performance pay and flexibilities  afforded to supervisors to rate their employees under the system.  However, as with our first review, discussion group participants at all  eight locations we visited primarily expressed frustration with and  concern about certain aspects of NSPS implementation and the system.  The prevalent themes that emerged during our discussion groups include  concerns over (1) the negative impact of NSPS on employee motivation  and morale, (2) the excessive amount of time spent navigating the  performance management process, (3) challenges with job objectives,      (4) factors undermining employee confidence in the system, and               (5) factors unrelated to job performance affecting employees’ final  performance ratings. As we noted in 2008, the themes that emerged during  our discussion group sessions are not surprising. Our prior work, as well  as that of OPM, has recognized that organizational transformations, such  as the adoption of a new performance management system, often entail  fundamental and radical changes that require an adjustment period to gain  employees’ trust and acceptance. As a result, we expect major change  management initiatives in large-scale organizations to take several years to  be fully successful.
subsections: section_title: Concerns about a Negative Impact on Motivation and Morale
paragraphs: A prevalent theme from our discussions with both employees and  supervisors was that several aspects of NSPS have had a negative impact  on employee motivation and morale—consistent with our first assessment  of NSPS. Specifically, discussion group participants at all eight locations  discussed how various aspects of NSPS—for example, their perception  that a rating of “3” is only average—have negatively affected their  motivation and morale. Discussion group participants at six of the eight  locations also told us they have negative perceptions of what a rating of  “3” means. At five of those locations, discussion group participants told us  that they continue to believe that a rating of “3” means only “average,” in  contrast to “valued performer,” as it was initially defined to the workforce  by DOD.
Discussion group participants at five locations also discussed how  achieving a rating higher than a “3” seemed to be an unattainable goal. For  example, employees at four locations told us that they felt NSPS either  does not provide incentives for high performance or encourages only  mediocre performance from employees under the system because of the  high number of employees receiving “3”-level ratings each year. As another  example, supervisors at one location noted that across the installation  there is a general feeling that everyone receives a rating of “3,” and  therefore such a rating is considered average, no matter how DOD defines  it. Similarly, discussion group participants at seven locations told us that  they felt it was difficult for employees in certain positions to receive a  rating of “5” because of the nature of their work or the perceived value  their management placed on those positions. At one of those locations,  supervisors told us that they felt such things as how the pay pool’s  business rules were structured affected whether an employee could  receive a high rating. At that location, the pay pool’s business rules  specified that an employee must receive a higher-level award, such as a  command or agencywide award, to receive a rating of “5.” However,  discussion group participants told us that they felt some employees were  not in a position to receive such awards because of their positions or the  type of work they did.
In addition, discussion group participants at all eight locations we visited  expressed frustration over the amount of the annual performance payout  provided under NSPS. For example, discussion group participants noted  that they felt the payout was not significant enough to encourage anything  other than average performance. Discussion group participants at six of  the eight locations also told us that they felt NSPS discourages employees  from seeking additional responsibilities and opportunities that fall outside  the scope of their objectives because their payout may not reflect their  additional work. In addition, discussion group participants at six locations  told us that because supervisory positions under NSPS require such a  significant increase in responsibility and effort, and because the maximum  allowable pay increase for reassignments is capped at 5 percent, some  employees may not seek promotion opportunities. Similarly, a discussion  group participant at another location expressed frustration that some  employees only received their payout in the form of a bonus and not an  increase in salary. Discussion group participants at three of the eight  locations also expressed concerns that they felt performance payouts  under NSPS tended to benefit higher-paid employees at the expense of  lower-paid employees. For example, employees at one location expressed  concerns that in their pay pool, the higher payouts under NSPS seemed to  go to employees at the top of the pay bands.
section_title: Concerns about an Excessive Amount of Time Spent Navigating the NSPS Performance Management Process
paragraphs: Another prevalent theme at seven of the eight locations and also  highlighted in our first assessment was that employees spend an excessive  amount of time navigating the performance management process. While  the discussion group participants’ complaints about the time- and labor- intensive nature of the system were not limited to any one part of the  process, discussion group participants at seven locations pointed out that  the time and effort required to complete the steps of the NSPS  performance management process were significantly greater than what  was required of them under previous systems. For example, one  supervisor we spoke with speculated that his supervisory duties under  NSPS took him six times as long to perform as they had under the GS  system, while another supervisor told us that he may have spent from 45 to  50 hours assessing the performance of three employees, a task he could  have completed in 10 hours under the GS system.
At five of the locations we visited, employees expressed concerns about  NSPS potentially affecting their ability to complete their jobs or affecting  the mission because of what they perceived as an excessive amount of  time required of employees and supervisors in navigating the NSPS  performance management process. In some instances, employees spoke of  impacts NSPS was having on their supervisors, while others spoke of their  own experiences navigating the NSPS performance management process.  At three locations, discussion group participants described how what they  perceived as an excessive amount of time navigating the NSPS  performance management process affected their ability to complete their  job-related duties, requiring completion of some NSPS tasks, such as self- assessments and employee ratings, after work hours or on weekends. One  employee described feeling inundated with information on NSPS and that  it was difficult to stay on top of things while simultaneously performing his  job, while another employee estimated that she spent about 2 hours per  week on NSPS-related tasks. In some instances, discussion group  participants told us that they saw the potential for the excessive time  commitment required by NSPS to affect the missions of their  organizations. According to one supervisor, any task that takes employees  away from their daily work affects the mission, and any task that takes the  time and patience of the command’s leadership detracts from the mission.  Further, in discussing during a site visit the potential for NSPS to impact  the organization’s mission, one general officer we spoke with described  NSPS as “mission ineffective.”
section_title: Concerns about Challenges with Job Objectives under NSPS
paragraphs: Another prevalent theme that emerged from our discussions with both  supervisors and employees at all eight locations was that there are  challenges with employee job objectives under NSPS. According to DOD,  the NSPS performance management system is designed to provide a fair  and equitable method for appraising and evaluating performance. As part  of the system, DOD established the concept of “job objectives,” which are  the required tasks of a given job as determined by managers and  supervisors, and directed that job objectives be developed and used as the  standards for evaluating employee performance. However, supervisors  and employees at each of the eight locations discussed challenges they  experienced developing their job objectives under NSPS. Specifically:    Although DOD guidance encourages employees to develop job  objectives that are specific, measurable, aligned, realistic, and timed— an approach summarized by the acronym S.M.A.R.T.—employees and  supervisors we met at six of eight locations discussed how they found  it challenging to develop job objectives that are measurable or that  follow the S.M.A.R.T. approach. Supervisors at one location objected  to the S.M.A.R.T. approach, particularly the “specific” portion, because  they felt that job objectives needed to be broad enough to allow  employees to discuss any accomplishments they make if they complete  additional job activities or other tasks that might arise during the year.  Supervisors at two locations discussed how the work they did was  nebulous and unpredictable, which made it challenging to develop job  objectives that not only reflected the nature of their work but that they  could exceed. Similarly, supervisors at another location expressed  concerns that employees’ job objectives may not reflect the work they  do by the end of the performance management cycle because of  constant changes within their organization.
According to discussion group participants at four locations, guidance  for developing job objectives is either limited or nonexistent, which  may result in different approaches to developing job objectives across  an organization. At one of these locations, employees told us that their  management had not established consistent ground rules for  developing job objectives and that as a result some employees’ job  objectives were based on out-of-date position descriptions. One  organization we visited used a mixture of mandatory and employee- specific job objectives; but, according to one employee, little guidance  exists to help employees and supervisors when they need to develop  personalized job objectives. Employees at another location told us that  there were significant differences in the amount of involvement they  had in developing their job objectives. For example, one individual told  us that employees in her office develop their own objectives, while  another said employees in her office are assigned mandatory  objectives and were thus unable to provide input into their objectives.
Discussion group participants at six locations expressed concerns that  it can be difficult to achieve a high rating for some job objectives.  Some locations we visited used mandatory job objectives, which left  employees concerned that their job objectives did not accurately  capture the full responsibilities of the work they performed. For  example, at one location, a uniform, mandatory supervisory objective  accounted for half of supervisors’ ratings, which, according to one  supervisor, diminished the value of the other responsibilities they had.  The supervisor expressed further concern that some mandatory job  objectives, such as those assigned to government purchase card  holders, require a pass-fail evaluation, making it difficult, if not  impossible, for the employee to receive a high rating. In one instance, a  location we visited required all employees to be rated against a  mandatory safety objective. However, according to some supervisors,  it did not make sense for everyone to have the mandatory safety  objective because for many employees, safety issues were out of their  control.
section_title: Concerns about Factors That Undermine Employee Confidence in the System
paragraphs: During our discussion groups, participants at all eight locations also  discussed how various factors undermine employees’ confidence in the  system and its implementation thus far. For example, discussion group  participants at six locations commented that they do not believe that the  NSPS performance management system has the ability to rate employees  fairly. At the locations we visited, discussion of these concerns centered  on such things as the perception of subjectivity and the potential for  favoritism under NSPS; a lack of transparency surrounding the pay pool  panel process, including a lack of understanding of what employees  needed to do to receive higher ratings; and the perception of  inconsistencies in interpretations of the standards used to determine  employee ratings.
One prevalent theme at all eight locations involved perceptions of  subjectivity, such as the potential for favoritism under NSPS during the  rating and pay pool panel processes. At five locations, participants  discussed their frustration with how NSPS takes the responsibility for  rating employees out of the hands of supervisors and places it in the hands  of the pay pool panel members, who may or may not have any direct  knowledge of individual employees’ performance. One supervisor told us  that NSPS may inadvertently favor employees who work closely or are in  direct contact with members of the pay pool panel because those  individuals have direct knowledge of the employees and, sometimes, their  performance. Similarly, supervisors at another location told us that they  did not feel that their pay pool panel understood their jobs and what they  do and expressed frustration that the pay pool panel did not seem to be  reaching out to their supervisors and higher-level reviewers for additional  input on their performance. At five of the eight locations, discussion group  participants also told us that they saw the potential for the employee- supervisor relationship to affect an employee’s rating—either to the  benefit or detriment of the employee.
Another prevalent theme at six of the eight locations—a theme also  highlighted in our first assessment of NSPS—was a lack of transparency  and understanding of the pay pool panel process. Specifically, supervisors  at two locations commented that their organizations’ pay pool panel  processes were not transparent. A supervisor at one location commented  that everything “goes into a black vacuum,” while another likened the  process to a “black box.” Employees at that same location referred to the  organization’s pay pool panel process as a “star chamber,” where decisions  are made but are not explained to employees. Employees and supervisors  at five locations expressed concerns about the amount of information they  received from their pay pools and about the process itself; some desired  further information to help them better understand the pay pool panel  process. In addition, at six of the eight locations, discussion group  participants told us that they did not understand what they needed to do to  receive a higher rating. For example, an employee at one location told us  that she was told by her supervisor that all employees had to receive a  rating of “3” because they would have had to “walk on water” to receive a  higher rating. Discussion group participants at two other locations also  discussed how “walking on water” was a perceived standard for receiving  a high rating under NSPS. At three locations, supervisors commented that  they were unclear about what they could do to help their employees  receive better ratings, while employees at four locations were unclear  about what they could do to achieve higher ratings.
Discussion group participants at six locations also raised concerns about  inconsistent interpretation of the standards used when evaluating civilian  employees under NSPS. Discussion group participants reported concerns  that military supervisors may rate employees using more stringent  standards than their civilian counterparts. Discussion group participants  also reported concerns that some military supervisors may not value the  NSPS performance management process and sometimes devote less time  and effort to the process, which could affect employees’ ratings. One  civilian supervisor told us that some military supervisors with whom he  attended NSPS training had a much harsher perspective of employee  performance than their civilian counterparts. For example, he noted that  the military supervisors indicated that giving a rating of “1” or “2” was  acceptable, whereas he believed civilian supervisors would be more  inclined to give an employee a rating of “3.” Employees also told us that  they do not believe some military supervisors value the work of employees  who perform certain job functions, such as providing child care on an  installation.
section_title: Concerns about Factors Unrelated to Performance Affecting Final Ratings
paragraphs: A prevalent theme expressed by discussion group participants at all eight  locations we visited is that factors unrelated to performance may affect  employees’ final performance ratings. Such factors include the existence  of a forced distribution or quota of ratings, the writing ability of employees  and supervisors, and pay pool panel members’ knowledge of employees.  For example:    Discussion group participants at all eight locations expressed concerns  that their pay pool panels used a forced distribution or quota for  ratings, which dictated the number of ratings in each category that  could be awarded. Employees at one location told us that they were  aware of their management’s attempts to artificially preserve a higher  share value for employees by primarily awarding ratings of “3,”  regardless of the employees’ performance. Further, at three locations  discussion group participants told us that their management told them  that all employees should expect to receive a rating of “3.” Moreover,  some discussion group participants told us that they doubted that their  actual performance had the bearing it was supposed to have on their  final ratings, while others felt the use of a forced distribution or quotas  was in direct conflict with the principles of pay for performance under  NSPS. While no discussion group participants we met with were aware  of any explicit guidance provided to pay pool panels or supervisors  that limited the number of certain ratings they assigned employees,  employees and supervisors from at least three locations believed that  informal guidelines existed or that pay pool panels or supervisors were  encouraged to limit the number of certain ratings they could assign.
Discussion group participants at all eight locations also expressed  concerns that the writing ability of employees and supervisors may  affect ratings—a theme also highlighted in our first assessment of  NSPS. Supervisors at one location likened the process of developing  employees’ assessments under NSPS to a writing contest. Moreover,  supervisors told us that they felt their writing ability could  unintentionally affect their employees’ ratings, noting, for example,  that a supervisor’s ability to articulate an employee’s achievements in  writing plays a significant role in supporting a higher rating for that  employee. Employees shared the supervisors’ concerns, noting that  they believed that succeeding under NSPS depended on the quality of  their written assessments, rather than their job performance, and that  their ratings could suffer if their supervisors did not provide the pay  pool panel with well-written assessments. In discussing the potential  influence that employees’ and supervisors’ writing skills may have on a  pay pool panel’s assessment of an employee, officials at seven of the  eight locations acknowledged that in some instances writing skills had  affected employees’ ratings and could overshadow employees’  performance.
Discussion group participants at seven locations also expressed  frustration that employee ratings were potentially affected by the  extent to which pay pool panel members have personal knowledge of  employees or understand the nature of their work in general. Some  discussion group participants felt that pay pool panel members’  personal knowledge of employees helped some employees receive  higher ratings, while others told us that they felt that members of the  pay pool panel were too far removed from them and lacked direct  knowledge of the work they performed. One employee believed that  individuals who were involved in implementing NSPS worked closely  with pay pool panel members, or were senior managers who were  more likely to receive higher ratings under NSPS than others. Other  employees told us that they were concerned about the potential for  pay pool panel members to advocate in some way for employees they  personally know—for example, by encouraging the pay pool panel to  contact a specific employee’s supervisor to seek additional  information or justification for a rating. As a result, they felt that pay  pool panel members’ personal knowledge of employees could benefit  some employees, but not others.
section_title: DOD Has Not Yet Developed a Comprehensive Action Plan to Address Employees’ Perceptions of the System
paragraphs: In our first assessment of NSPS, we recommended that DOD develop and  implement a specific action plan to address employees’ perceptions of  NSPS, based on guidance published by OPM for conducting annual  employee surveys and providing feedback to employees on the results.  The guidance suggests that after an agency’s survey results have been  reviewed, the agency has a responsibility to provide feedback to  employees on the results of the survey, as well as to let employees know  the intended actions to address the results and the progress made on these  actions. Further, the guidance suggests that agencies consider the  following when developing action plans:    who will be responsible for taking action;    who will be responsible for providing oversight;  if the individuals taking the action have the necessary authority to  make things happen;    what coordination, if any, is required, and how it will be accomplished;    how agencies will adjust given any changes or delays in their actions.
Since then, in June 2009, the PEO issued a departmentwide memorandum  entitled “Addressing Key NSPS Workforce Concerns”; however, issuance  of this memorandum does not fully meet the intent of our 2008  recommendation. Specifically, the PEO’s June 2009 memorandum  summarizes key concerns from the department’s 2008 evaluation of NSPS,  summarizes departmentwide actions that had been taken to date to  address employees’ concerns about the system, and suggests approaches  to enhance local efforts to address workforce concerns. The PEO  identified five key areas of concern, which are similar to those identified in  our own discussion group sessions with DOD employees and supervisors:  (1) performance communication and feedback, (2) understanding of  performance management and the pay pool process, (3) trust in the system  and its processes, (4) training and information, and (5) the amount of time  needed to fulfill performance management responsibilities. The PEO’s  memorandum urged the components to leverage information from the  department’s 2008 evaluation of NSPS and focus on the five areas  discussed above as they plan their own actions. Further, the PEO’s  memorandum noted that DOD has taken some steps to address employees’  concerns about NSPS—for example, developing and fielding a pay pool  training course for employees and rating officials, modifying its  implementing issuances to require all performance review authorities to  review pay pool panel results on an annual basis, and providing guidance  to employees on the prohibition against the forced distribution of ratings.  Issuance of the PEO’s memorandum represents an important first step.  However, because the memorandum does not specify actions the  department intends to take, who will be responsible for taking the action,  and timelines for addressing areas where employees express negative  perceptions of the system, it does not fully meet the intent of our 2008  recommendation. In developing an action plan, we note that OPM recently  issued guidance that agencies can use in developing action plans for  improving employee satisfaction. According to OPM, action plans should  clearly (1) state the objectives, (2) identify actions to be taken, (3) provide  outcome measures and improvement targets, and (4) describe how  progress will be tracked. In addition to identifying the specific actions that  will be taken to achieve improvements, OPM’s guidance also suggests that  agencies specify  time frames for accomplishing the actions,    who will be responsible for implementing the actions,    who will be affected by the actions,   the resources required, and   a plan to communicate these actions to managers and employees.
We continue to believe that developing and implementing a plan to  address employees’ perceptions of NSPS could help DOD make changes to  the system that could lead to greater employee acceptance and, ultimately,  the system’s successful implementation. Further, we note that having such  a plan is an approach that DOD could take to involve employees in the  system’s implementation—which is one of the safeguards we previously  discussed.
section_title: Conclusions
paragraphs: As we noted in our first assessment, DOD’s implementation of NSPS  placed the department at the forefront of a significant transition facing the  federal government. However, toward the end of this review, the future of  NSPS became uncertain, given the proposed legislation that, if enacted,  would terminate the system and require any future system created by DOD  to use safeguards similar to those discussed in our report, including  ensuring employee involvement in the system and providing adequate  training and retraining. In light of the contingent nature surrounding NSPS  and the possibility of implementing a different system, sustained and  committed leadership will be imperative to provide focused attention  necessary to implement any pay-for-performance system within DOD. Key  to implementing a fair, effective, and credible system is including  safeguards early on in the design of the system. Since we issued our first  assessment of NSPS in 2008, we note that DOD has continued to take steps  to meet the intent of each of the safeguards. However, with this latest  assessment, we note that the department has not implemented the  safeguards systematically; for example, it has not ensured that the training  provided to employees on the system’s operations is effective. Further,  DOD has not monitored how the safeguards specifically are implemented  by lower-level organizations across the department. As a result, decision  makers in DOD lack information that could be used to determine whether  the department’s actions are effective and whether the system is being  implemented in a fair, equitable, and credible manner. Additionally, while  DOD has gained experience operating under NSPS, at the time of our  review it had not yet developed an action plan for addressing employees’  perceptions of the system, as we recommended in 2008. As DOD moves  forward with implementing a pay-for-performance system—whether NSPS  or another—we believe that it is important for the department to improve  upon its implementation of the safeguards and address employees’  concerns. Left unchecked, these issues could undermine any future human  capital reform efforts within DOD.
section_title: Recommendations for Executive Action
paragraphs: To help implement a fair, effective, and credible performance management  system for its civilian employees—whether NSPS or another—we  recommend that the Secretary of Defense take the following three actions:    Review and evaluate the effectiveness of the department’s training.
Ensure that guidance is in place for conducting a postdecisional  analysis that specifies what process the components should follow to  investigate and eliminate potential barriers to fair and equitable  ratings.
Include, as part of the department’s monitoring of the implementation  of its system, efforts to monitor and evaluate how the safeguards  specifically are implemented by lower-level organizations across the  department.
section_title: Agency Comments and Our Evaluation
paragraphs: In September 2009, we provided DOD with a draft of this report that  included three recommendations to better address the safeguards and  improve implementation of the NSPS performance management system.  Specifically, we recommended that DOD (1) evaluate NSPS training,        (2) review and revise its guidance for conducting postdecisional analysis  of NSPS ratings, and (3) monitor how the safeguards specifically are  implemented. In commenting on a draft of our report, DOD partially  concurred with our three recommendations. DOD’s comments are  reprinted in appendix III.
DOD partially concurred with our recommendations, noting the  expectation that the Congress would require the department to terminate  NSPS by January 1, 2012, and this action, in turn, would require the  department to focus on drawing down NSPS in an orderly manner. DOD  further stated that it would consider acting on our recommendations to  the extent they are relevant as the department moves forward with any  future performance management system. We believe that this is a  reasonable approach. As discussed above, we recognize the contingent  nature surrounding NSPS as a result of provisions in the proposed  National Defense Authorization Act for Fiscal Year 2010, which recently  passed both Houses of Congress. Accordingly, we revised our  recommendations to apply to any future performance management system  for the department’s civilian employees—whether NSPS or another  system. However, we also note that provisions of the proposed legislation  would require DOD to implement certain safeguards and issue regulations  for that system to provide a fair, credible and transparent performance  appraisal system. We therefore continue to believe that our  recommendations have merit.
We are sending copies of this report to the appropriate congressional  committees. We will make copies available to others upon request. The  report also is available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to the report are  listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: As with our first assessment of the National Security Personnel System  (NSPS) in 2008, we limited our scope in conducting this review to the  performance management aspect of NSPS. Therefore, we addressed  neither performance management of the Senior Executive Service at the  Department of Defense (DOD) nor other aspects of NSPS, such as  classification and pay.
subsections: section_title: Determining Implementation of Safeguards and Monitoring Their Implementation
paragraphs: To determine the extent to which DOD has implemented safeguards as  part of the NSPS performance management system and monitored the  implementation of the safeguards, we used the following safeguards,  which we also reported on in our 2008 review:  Involve employees, their representatives, and other stakeholders in the  design of the system, to include employees directly involved in  validating any related implementation of the system.
Assure that the agency’s performance management systems link  employee objectives to the agency’s strategic plan, related goals, and  desired outcomes.
Provide adequate training and retraining for supervisors, managers,  and employees in the implementation and operation of the  performance management system.
Provide a process for ensuring ongoing performance feedback and  dialogue between supervisors, managers, and employees throughout  the appraisal period, and for setting timetables for review.
Implement a pay-for-performance evaluation system to better link  individual pay to performance, and provide an equitable method for  appraising and compensating employees.
Assure that certain predecisional internal safeguards exist to help  achieve consistency, equity, nondiscrimination, and nonpoliticization  of the performance management process (e.g., independent  reasonableness reviews by a third party or reviews of performance  rating decisions, pay determinations, and promotions before they are  finalized to ensure that they are merit-based, as well as pay panels who  consider the results of the performance appraisal process and other  information in connection with final pay decisions).
Assure that there are reasonable transparency and appropriate  accountability mechanisms in connection with the results of the  performance management process, to include reporting periodically on  internal assessments and employee survey results relating to  performance management and individual pay decisions while  protecting individual confidentiality.
Assure that performance management results in meaningful  distinctions in individual employee performance.
Provide a means for ensuring that adequate agency resources are  allocated for the design, implementation, and administration of the  performance management system.
To assess implementation of the safeguards, we reviewed the legislative  requirements and obtained and analyzed regulations and other guidance  for implementing the NSPS performance management system. We also  obtained and analyzed other documents, such as DOD’s rating results and  reconsideration statistics, for the 2007 and 2008 NSPS performance  management cycles. We also interviewed knowledgeable officials in  DOD’s NSPS Program Executive Office and the NSPS program offices of  the four components—the Army, the Air Force, the Navy, and the Fourth  Estate—to obtain a comprehensive understanding of their efforts to  implement NSPS and each of the safeguards, as well as the processes,  procedures, and controls used for monitoring and overseeing  implementation of the system. In addition, we conducted site visits to  select organizations located outside the continental United States to  assess implementation of the safeguards. To allow for appropriate  representation by each component, we visited two organizations per  component, or eight organizations in total. The organizations we visited  were selected based on a number of factors, such as the presence of a  large number or concentrated group of civilian employees under NSPS  and, when possible, the presence of employees who had converted to  NSPS under both spirals 1 and 2. We focused our efforts for this review on  visiting organizations located outside the continental United States  because our 2008 review focused on assessing implementation of NSPS  and the safeguards at locations that were geographically distributed  throughout the United States. We elected to focus our site visits in  Germany and Hawaii because of the civilian employees located outside the  continental United States who had converted to NSPS at the time we  initiated our review, more than half were located in either Germany or  Hawaii. Also, we wanted to determine whether civilian employees located  outside the continental United States were experiencing any unique  problems or challenges with the system. In Germany, the organizations we  visited were the 5th Signal Command; the 435th Air Base Wing; the  Defense Finance and Accounting Service; and the George C. Marshall  European Center for Security Studies, part of the Defense Security  Cooperation Agency. In Hawaii, the organizations we visited were the  Commander, Navy Region Hawaii; Headquarters, Pacific Air Force; the  Naval Facilities Engineering Command, Hawaii; and the U.S. Army Corps  of Engineers, Honolulu District. For each of the organizations we visited,  we met with or interviewed the performance review authority, pay pool  managers, pay pool panel members, rating officials, and the NSPS program  manager or transition team, among others, to discuss the steps they have  taken to implement the safeguards or otherwise ensure the fairness,  effectiveness, and credibility of NSPS. To assess the organizations’  implementation of the safeguards, we compared and contrasted the  information obtained during our interviews and supplemented this  testimonial evidence with the other relevant documentation we obtained,  such as the organizations’ pay pool business rules, lessons learned, and  training materials.
section_title: Determining DOD Civilian Employees’ Perceptions of NSPS
paragraphs: To determine how DOD civilian employees perceive NSPS, we analyzed  two sources of employee perceptions or attitudes. First, we analyzed the  results of DOD’s survey of civilian employees to identify employee  perceptions of NSPS and examine whether and how these perceptions  may be changing over time. Second, we conducted small group  discussions with civilian employees who had converted to NSPS and  administered a short questionnaire to the participants at each of the eight  organizations we visited. As with our first assessment of NSPS, our overall  objective in using the discussion group approach was to obtain employees’  perceptions about NSPS and its implementation thus far.
subsections: section_title: Analysis of DOD Survey Results
paragraphs: We analyzed the results of the Defense Manpower Data Center’s (DMDC)  Status of Forces Survey of Civilian Employees—including the May 2006,  November 2006, May 2007, and February 2008 administrations—to gauge  employee attitudes toward NSPS and performance management in general  and identify indications of movement or trends in employee perceptions.  As we reported in September 2008, we have reviewed the results of prior  administrations of DMDC surveys and found the survey results, including  the results of the Status of Forces Survey of Civilian Employees,  sufficiently reliable to use for several GAO engagements. However, to  understand the nature of any changes that were made to its survey  methods for administering the survey for 2008 as compared with previous  administrations, we also received responses to written questions from and  discussed these data with officials at DMDC. Based on these responses  and discussions, we determined that DMDC’s survey data remain  sufficiently reliable for the purpose of our reports on DOD civilian  employees’ perceptions of NSPS.
section_title: GAO’s Discussion Groups with DOD Civilian Employees under NSPS
paragraphs: We also conducted small group discussions with DOD civilian employees  and administered a short questionnaire during site visits in February and  March 2009. Specifically, we conducted two discussion groups—one with  nonsupervisory employees and another with supervisory employees—at  each of the eight locations we visited, for a total of 16 discussion groups.  As with our first assessment of NSPS in 2008, our objective in using this  approach was to obtain employees’ perceptions about NSPS and its  implementation thus far because discussion groups are intended to  provide in-depth information about participants’ reasons for holding  certain attitudes about specific topics and to offer insights into the range  of concerns about and support for an issue. Further, in conducting our  discussion groups, our intent was to achieve saturation—the point at  which we were no longer hearing new information.
As we previously reported, our discussion groups were not designed to   (1) demonstrate the extent of a problem or to generalize the results to a  larger population, (2) develop a consensus to arrive at an agreed-upon  plan or make decisions about what actions to take, or (3) provide  statistically representative samples or reliable quantitative estimates.  Instead, our discussion groups provide in-depth information about  participants’ reasons for holding certain attitudes about specific topics and  offer insights into the range of concerns about and support for an issue.  Although the results of our discussion sessions are not generalizable to the  entire NSPS civilian population, the composition of our discussion groups  was designed to ensure that we spoke with employees from each of the  four components at locations outside the continental United States.  Because supervisory and nonsupervisory employees have distinct roles  with respect to NSPS, we held separate discussion sessions for these  groups.
To select the discussion group participants, we requested that the  organizations we visited provide us with lists of employees who had  converted to NSPS. From the lists provided, we selected participants  based on their supervisory and nonsupervisory status. To ensure  maximum participation of the selected employees, we randomly selected  up to 20 participants from each group with the goal of meeting with 8 to 12  individuals in each discussion group and provided the employee names  and a standard invitation to GAO’s points of contact to disseminate to the  employees. At the majority of locations, we reached our goal of meeting  with 8 to 12 individuals in each discussion group; however, since  participation was not compulsory, in two instances we did not reach our  goal of 8 participants per discussion group. Table 6 provides information  on the composition of our discussion groups.
To facilitate our discussion groups, we developed a discussion guide to  help the moderator in addressing several topics related to employees’  perceptions of the NSPS performance management system. These topics  include employees’ overall perception of NSPS and the rating process, the  training they received on NSPS, the communication they have had with  their supervisors, positive aspects they perceive of NSPS, and any changes  they would make to the system, among others. Each discussion group was  scheduled for a 2-hour period and began with the GAO moderator greeting  the participants, describing the purpose of the study, and explaining the  procedures for the discussion group. Participants were assured that all of  their comments would be discussed in the aggregate or as part of larger  themes that emerged. The GAO moderator asked participants open-ended  questions related to NSPS, while at least one other GAO analyst observed  the discussion group and took notes. Following the conclusion of all our  discussion group sessions, we performed content analysis of the sessions  in order to identify the themes that emerged and to summarize the  participants’ perceptions of NSPS. We reviewed responses from several of  the discussion groups and created a list of themes and subtheme  categories. We then reviewed the comments from each of the 16  discussion groups and assigned each comment to the appropriate  category, which was agreed upon by two analysts. If agreement was not  reached on a comment’s placement in a category, another analyst  reconciled the issue by placing the comment in either one or more of the  categories. The responses in each category were then used in our  evaluation and discussion of how civilian employees perceive NSPS.
Following each discussion group we administered a questionnaire to the  participants to obtain further information on their background, tenure  with the federal government and DOD, and attitudes toward NSPS. We  received questionnaires from 164 discussion group participants. In  addition to collecting demographic data from participants for the purpose  of reporting with whom we spoke (see table 7), the purpose of our  questionnaire was to (1) collect information from participants that could  not easily be obtained through discussion, for example, information  participants may have been uncomfortable sharing in a group setting, and  (2) collect some of the same data found in past DOD surveys. Specifically,  the questionnaire included questions designed to obtain employees’  perceptions of NSPS as compared to their previous personnel system, the  accuracy with which they felt their ratings reflected their performance,  and management’s methods for conveying overall rating information.  Since the questionnaire was used to collect supplemental information and  was administered solely to the participants of our discussion groups, the  results represent the opinions of only those employees and cannot be  projected across DOD, a component, or any single pay pool we visited.
section_title: Department of Defense
section_title: Department of the Army
section_title: Department of the Air Force
section_title: Department of the Navy
paragraphs: Commander, Navy Region Hawaii, Pearl Harbor, Hawaii    Naval Facilities Engineering Command, Hawaii, Pearl Harbor, Hawaii    NSPS Program Office, Navy Yard, Washington, D.C.    Office of Civilian Human Resources, Navy Yard, Washington, D.C.
We conducted this performance audit from November 2008 through  September 2009 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient and appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: Reconsideration Process
paragraphs: Department of Defense (DOD) civilian employees who receive  performance ratings under the National Security Personnel System (NSPS)  have the option of challenging their ratings through a formal process  known as reconsideration. The reconsideration process is the sole and  exclusive agency administrative process for nonbargaining unit  employees to challenge their ratings. However, DOD’s NSPS regulations  also allow for alternative dispute resolution techniques, such as mediation  or interest-based problem solving, to be pursued at any time during the  reconsideration process consistent with component policies and  procedures. Under the reconsideration process, employees may challenge  their ratings of record or individual job objective ratings; employees  cannot challenge their performance payout, the number of shares  assigned, the share value, or the distribution of their performance payout  between salary increase and bonus, nor can they challenge their  recommended ratings of record, interim reviews, or applicable closeout  assessments. In addition, employees who allege that their performance  ratings are based on prohibited discrimination or reprisal may not use the  reconsideration process; rather, such allegations are to be processed  through the department’s equal employment opportunity discrimination  complaint procedure.
Employees who wish to challenge their rating have 10 calendar days from  the receipt of their ratings of record to submit written requests for  reconsideration to their pay pool managers. Within 15 calendar days of the  pay pool manager’s receipt of an employee’s request for reconsideration,  the pay pool manager must render a written decision that includes a brief  explanation of the basis of the decision. The pay pool manager’s decision  is final, unless the employee seeks further reconsideration from the  performance review authority. Specifically, if the employee is dissatisfied  with the pay pool manager’s decision, or if none is provided within the  prescribed time frames, the employee may submit a written request for  final review by the performance review authority or his or her designee.  This request must be submitted within 5 calendar days of receipt of the  pay pool manager’s decision or within 5 calendar days of the date the  decision should have been rendered. The performance review authority  then is allotted 15 calendar days from receipt of the written request from  the employee to make a decision, which is final. If the final decision is to  change the rating of record or job objective rating, the revised rating takes  the place of the original one, and a revised performance appraisal is  prepared for the employee.
According to DOD’s 2008 evaluation report, for the 2007 NSPS  performance management cycle, 2,302 civilian employees out of the  100,465 employees who were rated under NSPS elected to file a request for  reconsideration of their ratings, and of these, about 33 percent of the  requests were granted. For the 2008 NSPS performance management  cycle, according to the NSPS Program Executive Office, as of June 2009,  4,296 civilian employees out of the 170,149 employees who were rated  under NSPS elected to file requests for reconsideration of their ratings,  and of these, about 52 percent of the requests were granted.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the contact named above, Ron Fecso, Chief Statistician;  Marion Gatling, Assistant Director; Margaret G. Braley; Virginia A.  Chanley; William Colwell; Emily Gruenwald; K. Nicole Harms; Cynthia  Heckmann; Wesley A. Johnson; Lonnie McAllister; Carolyn Taylor; John W.  Van Schaik; Jennifer L. Weber; Cheryl A. Weissman; and Gregory H.  Wilmoth made key contributions to the report.
section_title: Related GAO Products
paragraphs: Human Capital: Continued Monitoring of Internal Safeguards and an  Action Plan to Address Employee Concerns Could Improve  Implementation of the National Security Personnel System. GAO-09-840.  Washington, D.C.: June 25, 2009.
Questions for the Record Related to the Implementation of the  Department of Defense’s National Security Personnel System.   GAO-09-669R. Washington, D.C.: May 18, 2009.
Human Capital: Improved Implementation of Safeguards and an Action  Plan to Address Employee Concerns Could Increase Employee  Acceptance of the National Security Personnel System. GAO-09-464T.  Washington, D.C.: April 1, 2009.
Human Capital: Opportunities Exist to Build on Recent Progress to  Strengthen DOD’s Civilian Human Capital Strategic Plan. GAO-09-235.  Washington, D.C.: February 10, 2009.
High-Risk Series: An Update. GAO-09-271. Washington, D.C.:   January 2009.
Human Capital: DOD Needs to Improve Implementation of and Address  Employee Concerns about Its National Security Personnel System.   GAO-08-773. Washington, D.C.: September 10, 2008.
The Department of Defense’s Civilian Human Capital Strategic Plan  Does Not Meet Most Statutory Requirements. GAO-08-439R. Washington,  D.C.: February 6, 2008.
Human Capital: DOD Needs Better Internal Controls and Visibility over  Costs for Implementing Its National Security Personnel System.   GAO-07-851. Washington, D.C.: July 16, 2007.
Human Capital: Federal Workforce Challenges in the 21st Century.   GAO-07-556T. Washington, D.C.: March 6, 2007.
Post-Hearing Questions for the Record Related to the Department of  Defense’s National Security Personnel System (NSPS). GAO-06-582R.  Washington, D.C.: March 24, 2006.
Human Capital: Observations on Final Regulations for DOD’s National  Security Personnel System. GAO-06-227T. Washington, D.C.:   November 17, 2005.
Human Capital: Designing and Managing Market-Based and More  Performance-Oriented Pay Systems. GAO-05-1048T. Washington, D.C.:  September 27, 2005.
Human Capital: Symposium on Designing and Managing Market-Based  and More Performance-Oriented Pay Systems. GAO-05-832SP.  Washington, D.C.: July 27, 2005.
Human Capital: DOD’s National Security Personnel System Faces  Implementation Challenges. GAO-05-730. Washington, D.C.: July 14, 2005.
Questions for the Record Related to the Department of Defense’s National  Security Personnel System. GAO-05-771R. Washington, D.C.: June 14,  2005.
Questions for the Record Regarding the Department of Defense’s  National Security Personnel System. GAO-05-770R. Washington, D.C.:  May 31, 2005.
Post-hearing Questions Related to the Department of Defense’s National  Security Personnel System. GAO-05-641R. Washington, D.C.: April 29,  2005.
Human Capital: Agencies Need Leadership and the Supporting  Infrastructure to Take Advantage of New Flexibilities. GAO-05-616T.  Washington, D.C.: April 21, 2005.
Human Capital: Selected Agencies’ Statutory Authorities Could Offer  Options in Developing a Framework for Governmentwide Reform.   GAO-05-398R. Washington, D.C.: April 21, 2005.
Human Capital: Preliminary Observations on Proposed Regulations for  DOD’s National Security Personnel System. GAO-05-559T. Washington,  D.C.: April 14, 2005.
Human Capital: Preliminary Observations on Proposed Department of  Defense National Security Personnel System Regulations. GAO-05-517T.  Washington, D.C.: April 12, 2005.
Human Capital: Preliminary Observations on Proposed DOD National  Security Personnel System Regulations. GAO-05-432T. Washington, D.C.:  March 15, 2005.
Human Capital: Principles, Criteria, and Processes for Governmentwide  Federal Human Capital Reform. GAO-05-69SP. Washington, D.C.:  December 1, 2004.
Human Capital: Building on the Current Momentum to Transform the  Federal Government. GAO-04-976T. Washington, D.C.: July 20, 2004.
DOD Civilian Personnel: Comprehensive Strategic Workforce Plans  Needed. GAO-04-753. Washington, D.C.: June 30, 2004.
Human Capital: Implementing Pay for Performance at Selected  Personnel Demonstration Projects. GAO-04-83. Washington, D.C.: January  23, 2004.
Human Capital: Key Principles for Effective Strategic Workforce  Planning. GAO-04-39. Washington, D.C.: December 11, 2003.
DOD Personnel: Documentation of the Army’s Civilian Workforce- Planning Model Needed to Enhance Credibility. GAO-03-1046.  Washington, D.C.: August 22, 2003.
Posthearing Questions Related to Proposed Department of Defense  (DOD) Human Capital Reform. GAO-03-965R. Washington, D.C.: July 3,  2003.
Human Capital: Building on DOD’s Reform Effort to Foster  Governmentwide Improvements. GAO-03-851T. Washington, D.C.: June 4,  2003.
Posthearing Questions Related to Strategic Human Capital  Management. GAO-03-779R. Washington, D.C.: May 22, 2003.
Human Capital: DOD’s Civilian Personnel Strategic Management and  the Proposed National Security Personnel System. GAO-03-493T.  Washington, D.C.: May 12, 2003.
Defense Transformation: DOD’s Proposed Civilian Personnel System  and Governmentwide Human Capital Reform. GAO-03-741T.  Washington, D.C.: May 1, 2003.
Defense Transformation: Preliminary Observations on DOD’s Proposed  Civilian Personnel Reforms. GAO-03-717T. Washington, D.C.: April 29,  2003.
High-Risk Series: Strategic Human Capital Management. GAO-03-120.  Washington, D.C.: January 2003.
Acquisition Workforce: Status of Agency Efforts to Address Future  Needs. GAO-03-55. Washington, D.C.: December 18, 2002.
Military Personnel: Oversight Process Needed to Help Maintain  Momentum of DOD’s Strategic Human Capital Planning. GAO-03-237.  Washington, D.C.: December 5, 2002.
Managing for Results: Building on the Momentum for Strategic Human  Capital Reform. GAO-02-528T. Washington, D.C.: March 18, 2002.
A Model of Strategic Human Capital Management. GAO-02-373SP.  Washington, D.C.: March 15, 2002.
Human Capital: Taking Steps to Meet Current and Emerging Human  Capital Challenges. GAO-01-965T. Washington, D.C.: July 17, 2001.
Human Capital: Major Human Capital Challenges at the Departments of  Defense and State. GAO-01-565T. Washington, D.C.: March 29, 2001.
High-Risk Series: An Update. GAO-01-263. Washington, D.C.: January  2001.