Identifying Talent through Technology


Automated Hiring Systems in Federal Agencies 

A report to the President and the Congress of the United States 

by the U.S. Merit Systems Protection Board 
Identifying Talent through 

Technology 

Automated Hiring Systems in 

Federal Agencies 
LETTER FROM THE ACTING CHAIRMAN 

U.S. Merit Systems Protection Board 

1615 M Street, NW 

Washington, DC 20419-0001 

August 2004 

The President 

President of the Senate 

Speaker of the House of Representatives 

Dear Sirs: 

In accordance with the requirements of 5 U.S.C. 1204(a)(3), it is my honor to 

submit this Merit Systems Protection Board report, “Identifying Talent Through 

Technology: Automated Hiring Systems in Federal Agencies.” 

The placement of “Strategic Human Capital Management” as the first of five 

Governmentwide initiatives in the President’s Management Agenda clearly 

acknowledges the critical importance of a capable, trained, and motivated Federal 

workforce. One aspect of attaining such a workforce is recruiting and selecting 

highly qualified employees. Federal agencies are looking to information 

technology to help them in this endeavor and are making increasing use of 

automated hiring systems to announce jobs, receive applications, and identify 

promising candidates. Consequently, automated hiring systems change how 

agencies assess job applicants. Because this assessment has significant 

implications for the quality of the Federal workforce, MSPB conducted a study of 

how Federal agencies are using these systems. 

We found that automated hiring systems can be beneficial to both agencies and 

job applicants, when used wisely. These systems can help agencies streamline the 

application process, assess applicants fairly and thoroughly, and reduce hiring 

time. However, we also found that investments in information technology alone 

do not guarantee these results. Agency leaders must also invest time and money 

in implementing an automated hiring system to ensure that selection criteria and 
assessment methods are valid and effective. Federal managers and human 

resource professionals must also continue to identify assessment criteria, select 

and use appropriate assessment tools, and exercise informed judgment 

throughout the hiring process. The report provides recommendations to agencies 

on how they can use automated hiring systems most effectively and 

recommendations to the Office of Personnel Management on steps it can take to 

support such use. 

I believe you will find this report useful as you consider how to shape and manage 

the highly qualified Federal workforce required to meet the challenges of the 21st 

century. 

Respectfully, 

Neil A. G. McPhie 
U.S. Merit Systems Protection Board 

Neil A. G. McPhie, Acting Chairman 

Susanne T. Marshall, Member 

Office of Policy and Evaluation 

Director 

Steve Nelson 

Deputy Director 

John Crum, Ph.D. 

Project Team 

James Tsugawa 

Harry C. Redd III 
Executive Summary 

Introduction 

The Business Case for Good Assessment 

Participants in the Hiring Process and Their Roles 

Research Questions 

Standards for Merit-Based Hiring 

Automation Objectives and Initiatives 

Results and Agency Experience 

Summary and Conclusions 

Recommendations 

Appendix A—An Overview of the Federal Hiring 

Process 

Appendix B—A Brief Discussion of Selection Tool 

Validity 

Appendix C—How Automated Hiring Systems 

Perform Assessments 

Appendix D—An Overview of “Multiple Hurdles” 

Appendix E—List of Recommendations Linked to 

Automation Success Factors 

Appendix F—An Argument for Competency-Based 

Qualification Standards 
Executive Summary 

Federal agencies are making increased use of automated hiring 

systems. This use reflects several factors, including the increasing 

power and availability of information technology, a desire to reduce 

time to hire, and the prospect of increased efficiencies. The automated 

hiring systems that agencies are using are much more than electronic 

filing cabinets for “virtual resumes.” These systems have materially 

changed how agencies recruit, receive applications, and assess and refer 

applicants for Federal jobs. For this reason, the U.S. Merit Systems 

Protection Board (MSPB or the Board) conducted a study to better 

understand these systems, their use in the Federal Government, and 

the effects of that use. 

This report discusses how automated hiring systems work, their 

capabilities and limitations, and how Federal agencies use these 

systems to assess job applicants. The report is not a “report card” on 

specific automated hiring systems or their providers, but discusses how 

automated hiring systems can be used in the hiring process, 

particularly to help identify and select the best person for the job. 

Below, we provide an overview of our major findings. 

The Technology 

• Automated hiring systems have a place in Federal hiring—but 

only if used wisely. 

• Automated hiring systems can be used to identify a pool of 

promising candidates. However, people must still determine what 

attributes automated hiring systems will measure, determine how 

these systems will measure those attributes, monitor the 

operation of these systems, and make selection decisions. 

• Automated hiring systems can contribute to material 

improvements in the hiring process. 

• The use of automated hiring systems is not only desirable, but 

increasingly necessary. There is no realistic alternative; manual, 
paper-based hiring processes are too burdensome, labor-intensive, 

and costly to be practical on a large scale. 

Results Achieved 

Our interviews and site visits revealed varying results. Some Federal 

agencies have achieved good results with their automated hiring 

systems. Those agencies have increased access to job opportunities, 

simplified the application process, and reduced time to hire while 

maintaining or improving the quality of new employees. Other 

agencies have struggled to obtain even acceptable results. All agencies 

have understood—or soon learned—that it takes vision, planning, 

expertise, and resources to use an automated hiring system 

successfully. 

Using Automated Hiring Systems Effectively 

As stated above, the key to whether automated hiring systems are 

beneficial is wise use. Whether automated hiring systems contribute 

positively to Government performance does not depend on the 

technology itself. It depends on how Federal agencies use that 

technology, and whether the leaders of Federal agencies make the 

investments in recruitment, assessment tools, and people needed to use 

the technology effectively. 

Effective use begins with an accurate understanding of the importance 

and nature of the hiring process. It is imperative that leaders in the 

Federal Government recognize that: 

• Hiring decisions have long-term consequences for organizational 

performance; 

• Good hiring decisions depend on good assessment. Good 

recruitment and good intentions are necessary but not sufficient; 

• Good assessment requires valid selection criteria and

measurement tools; and 
• An effective hiring process requires managers and human 

resources (HR) professionals who understand and are able to 

perform their respective roles in that process. 

Table 1 summarizes our recommendations to Federal agencies and the 

U.S. Office of Personnel Management regarding the use of automated 

hiring systems and the overall hiring process. The body of the report 

provides the rationale for these recommendations, and the 

Recommendations section of this report discusses each 

recommendation in detail. 
Table 1. Summary of Recommendations 

Subject Recommendations for Agencies 

Leadership 1. Manage hiring as a critical business process. 

2. Manage the introduction of an automated hiring 

system as an organizational change initiative. 

3. Invest appropriate resources in the design, 

implementation, and operation of the hiring process. 

Designing 

the Hiring 

Process 

4. Use automated hiring systems to support recruitment 

programs, not to replace them. 

5. Design a hiring process that emphasizes selection 

quality over less-important outcomes such as cost 

and efficiency. 

6. Evaluate existing assessment tools before automating 

them. 

Implementing 

an Automated 

Hiring 

System 

7. Invest in “content”—selection criteria and methods 

for assessing applicants against those criteria. 

8. Communicate roles and expectations to line 

managers, HR professionals, and applicants. 

9. Ensure the competence of HR professionals. 

Operating 

Practices 

10. Adopt documentation requirements that balance 

applicant burden and agency risk. 

11. Adopt quality control measures to ensure accurate 

ratings and referrals. 

12. Use “triage” to manage volume. 

13. Systematically evaluate and improve the hiring 

process. 

Subject Recommendations for OPM 

Policy 14. Establish (1) a core document that applicants can use 

to apply for jobs in different agencies; and 

(2) Governmentwide “applicant service standards.” 

Leadership 15. Move toward competency-based qualification 

standards. 

We conclude this overview by noting that an effective hiring process is 

a necessity, not a luxury. As one senior Federal official recently 

observed, Federal agencies cannot meet their workforce needs by filling 

jobs as quickly as possible with “warm bodies”: 

Recruiting and hiring the most 

qualified individuals takes time, but is 
critical to the [Securities and 

Exchange] Commission’s success. 

We have refused to hire employees 

simply to fill chairs, but rather are 

focused on hiring the best and most 

appropriate people to fill these 

important positions, and are keenly 

focused on where each staff person can 

do the most good.1 

Nor can Federal agencies afford to abandon the “war for talent,” relying 

solely on contractors, other levels of government, or other 

organizations to hire and provide the talent needed to manage 

Government programs and deliver public services. 
Introduction

This report describes how agencies are using information technology in 

the hiring process, especially to assess job applicants. Thus, when we 

refer to “automation,” we are referring not only to information 

technology and automated hiring systems, but also to Federal agencies’ 

use of that technology.2 

Automation has a significant and expanding role 

in the Federal hiring process. 

Federal agencies were slow to introduce automation to their hiring 

processes,3 but progress has been swift. When we began this study in 

February 2002, we found substantial variation in agencies’ use and 

understanding of automated hiring systems. The scale ranged from 

“interested in learning about automated systems” to “implementing an 

automated system” to “experienced.” Most agencies were at the lower 

end of the scale; very few were at the higher end. By the time we 

completed our initial information gathering, most large Federal 

agencies were actively considering or using one or more automated 

hiring systems.4 This shift is permanent. Manual, paper-based hiring 

processes are rapidly losing ground and will not regain it. Moreover, 

the role of automated hiring systems is expanding. Agencies do not 

simply use them as “electronic filing cabinets,” but also use them to 

assess applicants—to make substantive decisions about applicants’ 

qualifications and to make distinctions among applicants. For these 

reasons, it is in agencies’ and the public’s interest to ensure that 

automated hiring systems are at least as effective as the paper-based 

hiring systems they replace. 
Table 2. Major Steps in the Federal Hiring Process 

Step Action 

1 Define the job and conduct a job analysis to identify job 

requirements. 

2 Select assessment tools. 

3 Develop a rating procedure—a method for distinguishing 

among qualified applicants. 

4 Recruit, publicize the job, and receive applications. 

5 Review applications for legal requirements. 

6 Assess minimum qualifications—determine whether applicants 

meet minimum requirements. 

7 Assess relative qualifications—make distinctions among 

qualified candidates. 

8 Issue a list of candidates to the selecting official. 

9 Assess candidates. 

10 Select a candidate (or candidates). 

Hiring includes many steps that can be 

automated. 

Automated hiring systems can do many things; for example, they can 

distribute job information electronically, receive on-line applications, 

assess applicants, notify applicants of the outcome of the hiring process, 

and eliminate much human labor. As shown in Table 2, hiring typically 

involves many steps and decision rules and much complex information. 

We provide a fuller description of the Federal hiring process and its 

terminology in Appendix A. 

Our primary focus in this report is assessment itself: how agencies use 

technology to evaluate applicants’ qualifications. We emphasize the 

function (assessment) over the means (technology) for two reasons. 

First, because technology is evolving rapidly, a report that simply 

catalogued and rated the features of individual automated hiring 

systems would be obsolete by the time it was published—if not before. 

Second, it makes little sense to examine automated systems apart from 

the underlying hiring process, particularly the key step of assessment 

of qualifications. Automated hiring systems are a means to an end, and 

only have value if they help agencies refer and select the best 
applicants. As we discuss in this report, automated hiring systems 

clearly can help agencies do this, a significant achievement given the 

demands of Federal hiring processes. Yet the outcomes of automation 

ultimately depend very little on the technology and largely on the 

quality of the underlying assessment processes. Until an organization 

is certain that it can consistently distinguish the best candidates from 

good candidates, and good candidates from mediocre (or unqualified) 

candidates, automating the hiring process is a waste of time and other 

vital resources. 

Accordingly, this report not only discusses how automation can be used 

to support assessment of applicants, but also outlines criteria for sound 

and fair assessment. The report also makes recommendations about 

assessment and related aspects of the hiring process. 

Automated hiring systems can support forms of 

hiring other than competitive examining. 

When conducting this study (for example, when interviewing agency 

managers and HR professionals), we usually discussed automation and 

assessment in the context of competitive examining,5 because 

competitive examining is the standard means of entry into the 

competitive civil service.6 Competitive examining is also procedurally 

demanding, because agencies must follow rules governing the 

announcement of job vacancies; receipt of applications; referral, contact, 

and selection of candidates; and maintenance of records.7 Thus, 

competitive examining is an excellent test of an automated system’s 

ability to support other, less procedurally complex forms of hiring, such 

as merit promotion. Nevertheless, in a merit system, all forms of 

hiring—whether internal (such as merit promotion) or external, 

competitive service or excepted service—involve assessment. 

Therefore, our findings and recommendations have relevance for all 

forms of hiring and for all agencies. 

Automated hiring systems are evolving. 

The systems we covered in this study have changed since our data 

gathering began, and will continue to change. Agencies may also adopt 

entirely new automated hiring systems, modify their systems and 
processes, or use existing systems in new ways. In addition, Federal 

hiring rules and processes are always subject to change. For these 

reasons, our observations and conclusions cannot be the final word on 

automated hiring systems or agencies’ use of those systems. 

Nevertheless, this report provides a reasonably accurate description of 

the fundamental capabilities and characteristics of automated hiring 

systems with respect to their relevance for Federal agencies. This 

description should remain relevant and usable for at least a few years. 

Restated, we expect evolution, not revolution. We learned of no major 

technological breakthroughs on the horizon, and we anticipate that 

agencies will direct their efforts to implementing and effectively using 

existing technologies instead of pursuing new and unproven 

technologies. 
The Business Case for Good 

Assessment 

Assessment matters. 

In a merit system, assessment—by which we mean evaluation of a 

candidate’s ability to do a job—determines who is actively considered 

for a job, and who is ultimately selected for a job. And, as noted below, 

selection decisions are critically important: 

Results of wise decisions can range 

from the mere absence of problems to 

genuinely excellent outcomes 

promoting organizational purposes. 

Cumulative effects in hiring decisions 

can result in substantial increases in 

mean performance levels and 

productivity. Consequences of unwise 

decisions can range from 

inconvenience to disaster.8 

The benefits of a good selection are not merely theoretical.9 A recent 

study by the consulting firm Watson Wyatt Worldwide suggests that 

selecting highly qualified candidates is not only consistent with merit 

principles, but profitable. The company surveyed the human capital 

practices of over 400 private-sector companies and found that 

companies that hired workers well-equipped to perform their jobs 

created more value for their shareholders than did less-selective 

companies.10 

Conversely, the costs of a poor selection are also real. As shown in 

Table 3, one organization estimates that a poor selection can cost a 

private sector company as much as three times the employee’s annual 

salary. 
Table 3. Estimated costs of hiring the wrong person 

Type of Employee Estimated Costs 

Entry-level full-time employee $5,000 to $7,000 

$20,000/year FTE $40,000 

$100,000/year FTE $300,000 

Note: Costs include wasted salary, benefits, severance pay, headhunter fees, 

training costs, and hiring time. 

Source: Corporate Leadership Council, Literature Review, “Employee Selection 

Tests,” Catalog No. 070-198-213, Washington, DC, March 1998, p. 2. 

Previous Board studies document additional costs borne by Federal 

agencies that make a poor selection decision. Federal agencies do not 

have the option of simply terminating a poor performer and “starting 

over,” unless the employee is serving a trial or probationary period, and 

the available alternatives are costly.11 Adverse action (i.e., demotion or 

termination) will require substantial management attention and staff 

time. The remaining options—remediation or inaction—are also 

costly. Remediation entails a potentially lengthy period of reduced 

productivity: the organization must devote resources to training and 

counseling the employee while coworkers (or managers) fix or perform 

the employee’s work. Inaction means that the agency will bear the 

costs of a poor selection indefinitely and may also harm productivity: 

* * inaction can create problems far 

beyond that of a single incompetent 

worker. It can turn the [work] unit’s 

better performers into overworked, 

resentful employees who, noticing the 

absence of penalties for inferior 

performance, may reduce their own 

efforts as a result.12 

Good selections require good 

assessments. 

Agencies can significantly improve the likelihood of selecting good 

employees by using assessment methods with high validity. Validity is 

the ability of an assessment method to predict on-the-job performance. 
(Appendix B provides a fuller discussion of the concept of validity.) 

Good assessments, used wisely, are particularly important for agencies 

that conduct competitive examining, a process that is notably 

unforgiving. Competitive examining demands precision, because an 

agency must be able to identify a small number of promising candidates 

from a pool of tens or hundreds of applicants. Such examining also 

demands accuracy, because an agency must not only identify the best 

applicants, but also provide special selection consideration to certain 

categories of applicant.13 Failure to do so may legally compromise any 

appointment(s) made; there is no tolerance for error. Managers also 

have little tolerance for error. For example, a manager will not be 

pleased if the best applicant for a position cannot be selected because 

that applicant was not rated accurately.14 

Assessment methods are not all created equal. 

As illustrated in Figure 1, some assessment methods are more valid 

than others.15 The most valid methods, such as work sample tests and 

mental ability tests, make useful distinctions among applicants and 

provide valuable, if imperfect, predictions of on-the-job performance. 

The worst methods, such as cataloging applicant’s interests (“Do you 

like helping people?”) and graphology (handwriting analysis), are 

useless: they provide no insight into an applicant’s ability to perform. 

The Validity of Selected Assessment Methods16 

W

Structured interviews 

Mental ability tests 

T

Unstructured interviews 

Reference checks 

T

T&E (point method) 

Interests 

0.02 

0.54 

0.51 

0.45 

0.38 

0.35 

0.26 

0.18 

0.11 

0.10 

0.51 

Graphology 

ork sample tests 

&E (behavioral consistency) 

Biographical data measures 

&E (years of job experience) 
Ratings of training and experience can be useful 

or useless. 

Most agencies use ratings of training and experience (T&E) to screen 

and refer applicants. This is understandable: most OPM qualification 

standards contain explicit experience requirements (and sometimes also 

education requirements), and the resume—the closest thing to a 

universal application form—is essentially an applicant’s description of 

his or her training and experience. And Federal supervisors, by choice 

or by default, rely heavily on training and experience when making 

hiring decisions.17 

Yet Federal agencies’ reliance on T&E is not evidence of its validity. 

Figure 1 shows that the validity of a T&E rating depends on how that 

rating is done. The best method, known as behavioral consistency, has 

a validity of 0.45, good enough to be useful. This method is 

information- and labor-intensive: applicants must provide detailed 

descriptions of relevant behaviors and accomplishments, and the 

employer must evaluate those behaviors and accomplishments against 

preestablished benchmarks. The least valuable method, the “point 

method,” assigns points based on an applicant’s years of experience, 

years of education, educational credentials (e.g., degrees), and the like. 

The point method is fast, simple, and—unfortunately—pointless. 

The implications are clear for any agency seeking to use automation in 

the hiring process. First, the referrals produced by an automated 

system will be only as good as the underlying assessment method(s). 

Second, any agency that plans to use an automated hiring system to 

rate training and experience must pay attention to how the system will 

make those ratings to ensure that they are valid and useful. 
Participants in the Hiring 

Process and Their Roles

This section lists the major participants18 in the Federal process, with 

a focus on assessment, and describes their roles and responsibilities. 

We do this (1) to help the reader better understand the hiring process 

and our findings; (2) to emphasize that effective, timely hiring is a 

shared responsibility; and (3) because our findings and 

recommendations are relevant to all these participants. 

The division of labor we describe below is illustrative. Agencies may 

delegate responsibilities and assign tasks differently. That does not 

affect our main point: that the responsibility for good hiring decisions 

does not lie solely with HR. The hiring process, no matter how 

technically sound or technologically advanced, cannot succeed without 

the support and cooperation of all participants, including agency 

leadership, line managers, HR managers and staff, OPM, and 

applicants. 

Agency Leadership. 

“Agency leadership” refers to high-level management (the executive 

layer at the top of an agency, bureau, or major installation). It is 

axiomatic that agency leadership is responsible for the success or 

failure of the agency’s programs. Agency leadership is equally 

responsible for the success of supporting functions such as finance, 

information technology, and human resources. 

These individuals are rarely, if ever, directly involved in the assessment 

and selection of line employees other than those on their immediate 

staffs. Yet they are critically important to the process. They create 

human capital goals and plans; they set priorities and expectations for 

the HR function; they allocate resources to HR functions (such as 

recruiting and assessment); and they establish expectations for line 

managers. For the critical step of assessment, agency leaders decide 

whether assessment is important and whether to spend the time, 

money, and “political capital” needed to do it well. 
Line Managers 

Line managers19 are directly involved in assessment and selection. As 

representatives of management, line managers define jobs; identify the 

knowledges, skills, and abilities (KSA’s) or competencies important to 

the job; and participate in the development of rating procedures. As 

supervisors, line managers conduct interviews and reference checks and 

recommend or select new employees. Automation does not relieve 

managers of these tasks, although it can support these tasks and 

eliminate repetitive work. In fact, automation makes these tasks more 

important, because automated hiring systems, unlike human beings, 

cannot “work around” a deficient job analysis or rating procedure. Nor 

can these systems identify and deal with applicants who misstate or 

falsify their qualifications. 

HR Managers 

HR managers have two distinct roles in the hiring process. As 

functional managers, they oversee the design, development, and 

application of staffing policies and assessment procedures. As line 

managers, they select, train, and evaluate the HR professionals who 

will advise line managers on staffing issues, develop and use assessment 

tools, and interpret and apply staffing policies. 

HR Professionals 

HR professionals are responsible for the day-to-day operation of the 

hiring process. They coordinate recruiting activities, conduct job 

analyses, prepare and post job announcements, assess and refer 

applicants, and respond to applicant inquiries.20 In most agencies, HR 

professionals are also the “end users” of the automated hiring system.21 

The timeliness and quality of the hiring process—and consequently the 

quality of new employees—depends on their expertise, technological 

competence, and responsiveness. 
The Office of Personnel Management 

OPM regulates Federal civilian employment and provides 

Governmentwide leadership for human resources management. First, 

OPM regulates employment under title 5, United States Code—the law 

under which most Federal civilian employees are hired, paid, and 

managed. For example, when agencies fill jobs through competitive 

examination, they must comply with OPM-issued rules and 

instructions.22 OPM’s authority also covers internal placement (e.g., 

merit promotions and reassignments) and excepted service 

employment.23 Moreover, OPM’s influence extends beyond its 

regulatory bounds: even when agencies are not required to follow 

OPM rules or use OPM systems, they may adopt them or develop 

comparable rules or guidelines. 

Second, OPM develops qualification standards—minimum 

requirements that an applicant or employee must meet to be placed in a 

position. These standards determine which applicants agencies may 

consider for jobs (or, more accurately, which applicants they may not 

consider); they also dictate, at least in part, how agencies will assess 

applicants.24 

Finally, OPM manages Governmentwide human resources initiatives 

and establishes expectations for Federal human capital management. 

For example, OPM is leading two hiring-related electronic government 

(“e-gov”) initiatives: Recruitment One-Stop, an effort to standardize 

and simplify the job search and application process; and e-Clearance, an 

effort to improve the security clearance process.25 OPM can also 

encourage change in less formal ways. For example, in 2002 OPM and 

the Partnership for Public Service issued a “Pledge to Applicants” that 

commits OPM and Federal agencies to an open, comprehensible, and 

timely hiring process. 

Job Applicants 

Obviously, the depth and quality of the talent pool depends on job 

seekers. Individual job seekers decide whether to consider Federal 

employment and whether to become applicants for Federal jobs. 

However, applicants’ influence on hiring decisions is not limited to the 

decision to apply. Applicants also provide most of the information that 

agencies use to assess and select new employees. 
Research Questions

In conducting this study, we attempted to answer four questions: 

1. How is the Federal Government using automated hiring 

systems, particularly to assess employees? This question includes 

matters such as: why agencies are seeking to automate the application 

and assessment process; how automated hiring systems operate; and 

how agencies have modified their application and assessment processes 

to accommodate or take advantage of automation. 

2. What are the effects of automation? This question includes 

matters such as: how automation affects (1) the quality of assessment, 

referrals, and new employees; (2) time to hire; and (3) the fairness and 

openness of the hiring process. The standards we used to evaluate the 

effects of automation, and on which we base our findings and 

recommendations, are discussed in the next section of this report. 

3. What role should automated hiring systems have in the 

assessment and selection of Federal employees? As stated 

previously, we did not aim to provide a “buyer’s guide” to automated 

hiring systems but, rather, to determine what these systems can 

contribute to Federal hiring, particularly the assessment process and 

the resulting selection decisions. Restated, we wanted to know 

whether existing systems and technology can meet the requirements of 

Federal agencies and the Federal hiring process, and if so, what hiring 

functions they can usefully perform or support. 

4. What should OPM and Federal agencies do to use automation 

effectively in hiring? Through this question, we explored approaches, 

policies, and practices that are essential or helpful to effective use of 

automated hiring systems. This question also covers broader issues, 

including assessment strategy, the roles of OPM and agency 

leadership, and change management. 

Our answers to the first and second questions appear in the section 

titled “Automation Objectives and Initiatives”; our answers to the third 

and fourth questions appear in the sections titled “Results and Agency 

Experience” and “Summary and Conclusions.” 
Standards for Merit-Based 

Hiring 

Below, we outline ten standards for merit-based hiring, with a focus on 

the assessment process. These standards provide a framework for 

evaluating the role and effects of automation, and for designing and 

evaluating hiring practices in general. We believe that a hiring process 

must meet all of these standards to be successful. Success on one 

standard does not compensate for failure on another. For example, a 

hiring process whose criteria are valid, but whose requirements are so 

burdensome that they deter most qualified applicants, is not successful. 

Similarly, a hiring process that is well-liked by managers, but whose 

criteria are not demonstrably valid, would be problematic. The 

standards are as follows. 

1. Make reasonable demands of applicants. Applying for a Federal 

job should not be a job in itself. Applicant burden (the level of effort 

and documentation needed to apply for a job) is not merely a “customer 

service” issue; it also affects the openness of the hiring process and the 

quality of selections. An application process that is unduly burdensome 

(i.e., complex, time-consuming, or both) or restrictive (e.g., “on line 

application only”) raises the cost of applying and deters applicants who 

lack the time or resources needed to apply. Such a process is not truly 

open, even if the position in question was properly announced and made 

“open to the public.” 

2. Clearly communicate assessment criteria and methods. 

“Openness” does not end with public notice of a job vacancy and a 

simple application process. Employees and applicants generally 

distrust selection processes and decisions they do not understand. 

Accordingly, agencies should communicate assessment criteria (what 

they are looking for in an applicant) and assessment methods (how they 

will evaluate applicants against the criteria) to ensure that their hiring 

process is not viewed as a “black box.”26 

Communicating assessment criteria also makes sense from an 

assessment standpoint, particularly when ratings of training and 
experience are used. Applicants are more likely to provide useful 

descriptions of their experience, education, and accomplishments when 

they know what the employing agency wants. We note that agencies 

must achieve balance in this area: they must disclose enough to let 

applicants understand and participate constructively in the hiring 

process, but not disclose so much that the process is vulnerable to 

manipulation. 

3. Assess applicants in a manner consistent with merit principles 

and professional standards. The merit system principles and related 

regulations require agencies to use job-related selection criteria and 

methods.27 That is, agencies must select assessment criteria (such as 

knowledges, skills, and abilities) based on their importance to the job 

and evaluate applicants against those criteria. Job-related assessment is 

also critical to the perceived fairness of the hiring process.28 As 

illustrated below in comments from a major Federal union, acceptance 

of—or resistance to—automated hiring systems may depend on 

whether their use is appropriately job-related: 

NTEU looks forward to widespread 

automation of the hiring process. 

Ideally, it will enable agencies to fill 

positions faster, to more thoroughly 

analyze candidate qualifications, and 

to more deeply study the impact of 

hiring practices. However, * * * we 

have noticed issues that concern us. 

First, among them is the fact that 

these systems create the opportunity 

for selecting officials to easily review 

irrelevant data about candidates. One 

agency that previewed for us [its] 

system * * * noted a selecting official 

would be able to see which other jobs 

the candidate had not been selected for 

in the past and the comments that 

were included in the candidate’s file 

from those other selections. We do 

not think that is a good thing; in fact, 

we oppose it. Selections should turn 

on your competition for the single 
position in question, not on unverified 

observations, or similarly irrelevant 

information, from another selection 

action.29 

4. Evaluate applicants fairly. Assessment must be even-handed. 

That is, similarly situated applicants should be treated similarly; 

assessment, consideration, and selection should not depend on nonmerit factors such as ethnicity or personal friendship. However, 

uniform and nondiscriminatory assessment is not sufficient. 

Assessment should also be reasonable and reasonably thorough. An 

applicant’s fate should not depend on trivial matters (such as the font 

size used for the application) or the idiosyncrasies of the rater, whether 

that rater is a person or a machine. Where automated hiring systems 

are concerned, even-handedness can be assumed. However, our 

discussions with agencies—and comments from applicants and Federal 

managers—clearly indicate that thoroughness and accuracy cannot be 

assumed. 

5. Consistently refer the best applicants to the hiring manager. 

Earlier in this report, we discussed the importance of valid assessment 

methods. However, the “real world” test of an assessment process is 

not its theoretical soundness, but its success in practice: whether it 

yields good employees. If managers are satisfied (or, better yet, 

delighted) with the quality of the applicants referred to them, that is a 

positive indication that the underlying assessment process is measuring 

the right attributes with reasonable reliability.30 If, on the other hand, 

managers repeatedly complain about interviewing marginal or 

unqualified candidates or complain that “the system” has discarded the 

most promising applicants, then that is a strong indication that 

something is wrong. 

6. Assess, refer, and select applicants in a timely manner. 

Managers and applicants believe that timeliness is very important. We 

agree. First, timeliness affects quality. An organization that takes too 

long to assess applicants, make selections, or extend employment offers 

cannot reasonably expect to hire “the best and the brightest.” Those 

candidates will be lost to other, more nimble employers. Second, 

timeliness is an important element of fairness and due process. The 

axiom “justice delayed is justice denied” applies to the hiring process as 
well as the judicial process. Applicants are unlikely to view an 

excessively long hiring process as fair, even if that process is valid and 

even-handed. 

7. Keep applicants informed. Any agency that aims to have a “fair 

and open” hiring process should do more than simply post job 

opportunities, accept applications, and rate applications. Agencies 

should also let applicants know where they stand. Applicants should be 

able to find out whether their application was received, how they rated, 

when a hiring decision will be made, and whether the job has been 

filled. 

8. Comply with applicable laws, regulations, and rules. Following 

the rules is essential for Federal agencies. As “model employers”—and 

exemplars of the rule of law—Federal agencies have a special 

responsibility to comply with applicable laws, regulations, rules, and 

public policies. Depending on the job (and the legal authority used to 

fill the job), agencies may have to: announce the vacancy through 

USAJOBS;31 provide priority consideration to certain applicants; 

distinguish and refer a predetermined number of qualified candidates; 

and follow veterans’ preference rules in rating, referral, and selection.32 

9. Receive and withstand scrutiny. Automation, contracting, and 

“partnering” give agencies great flexibility in how they examine. For 

example, agencies may use a contractor to analyze a job and identify 

rating criteria, a computer to score applications, and a panel of outside 

experts to conduct interviews. However, agencies remain responsible 

for their employment practices. They cannot delegate or “contract out” 

that responsibility. And that responsibility extends beyond assessment 

and selection. The Uniform Guidelines on Employee Selection 

Procedures33 require Federal agencies not only to use job-related 

employment practices, but also to: 

• Maintain information and records needed to evaluate the effects of 

employment practices along demographic lines (e.g., race, sex, 

national origin); and 

• Defend, modify, or discontinue any employment practice that has 

an adverse impact. 
Maintaining information and records—and rigorously analyzing 

them—is essential, because a successful defense of an employment 

practice (such as assessment tools and processes) must be based on hard 

data. Previously, we noted that managerial satisfaction with the 

assessment process is important, and can be a good indicator of its 

soundness. However, the Uniform Guidelines clearly state that 

managerial satisfaction cannot be used to defend an assessment process: 

Under no circumstances will the general reputation 

of a test or other selection procedures, its author or 

its publisher, or casual reports of its validity be 

accepted in lieu of evidence of validity. Specifically 

ruled out are: assumptions of validity based on a 

procedure’s name or descriptive labels; all forms of 

promotional literature; data bearing on the frequency 

of a procedure’s usage; testimonial statements and 

credentials of sellers, users, or consultants; and other 

non-empirical or anecdotal accounts of selection 

practices or selection outcomes.34 

Agencies should do more than simply defend existing employment 

practices when challenged. Instead, agencies should systematically 

evaluate and improve their assessment processes and tools. Put 

differently, agencies should examine how they assess and select 

employees before they receive a complaint from a manager or a legal 

challenge from a dissatisfied applicant—not after. 

10. Make reasonable demands on HR professionals and hiring 

managers. We include this standard for practical reasons. 

Assessment, including job analysis, is worthy of the time and attention 

of HR professionals and managers. We do not agree with the 

proposition that these people should not be bothered with “trivial” 

matters such as job analysis and assessing applicants. However, we are 

also realists. An assessment process is only as good as its use in 

practice; an assessment process that requires time or money that HR 

professionals and managers do not have—or are unwilling to invest— 

is unlikely to be effective. 

The (Un)Importance of Cost Per Hire 

Our list of standards does not include a low cost per hire. That 

omission is deliberate. Certainly, an efficient hiring process is 
desirable. Some aspects of efficiency, such as applicant burden and 

timeliness, are important because they directly affect fairness and the 

quality of the applicant pool. The same cannot be said of cost per hire: 

applicants neither know nor care how much the agency spends to hire 

an employee. Of course public money should be spent wisely; all else 

equal, it is better to spend $1,000 to select a good employee than to 

spend $10,000. And agencies should continually measure and improve 

their hiring process. But, as the private sector observer quoted below 

recommends, such efforts should be directed at important matters, such 

as hiring the right person—not hiring a person as cheaply as possible: 

Employees are such a bargain, and 

hiring costs are such a small 

percentage of an employee’s value, 

that fretting over the cost of a hire is 

like agonizing over whether the 

gumball machine will give you seven 

or eight gumballs for a nickel. Who 

cares? 

Measuring the cost of each of your 

hires is one way to spend time. It 

takes a lot of it. Don’t do it. It’s a 

waste. Instead, every HR and 

recruiting pro should spend time 

measuring what a top employee is 

worth. Compare that to what an 

average employee is worth, then 

sprint to your CFO’s office with the 

numbers.35 

If this line of reasoning leads agency leaders to conclude that they 

should spend more time or money on assessing and selecting 

employees, so be it. It is not always possible to “do more with less.”36 
Automation Objectives and 

Initiatives 

This section discusses (1) what agencies hoped to achieve through use 

of an automated hiring system, (2) agency automation initiatives—how 

agencies went about automating their hiring processes, and (3) OPM 

initiatives related to assessment and the use of automated hiring 

systems. 

Objectives of Automation 

Agencies’ primary reason for automating the 

hiring process was to achieve faster hiring. 

When we talked to agencies, we asked them why they used automated 

hiring systems. All mentioned speed: to reduce the time to hire by 

announcing jobs and screening, ranking, and referring candidates more 

quickly. 

Other reasons were to reduce workload and increase efficiency. Less 

frequently cited reasons were to expand the candidate pool and to 

improve candidate quality. However, as reflected in a Government 

Computer News (GCN) interview of Kay Coles James, Director of the 

U.S. Office of Personnel Management, such reasons were secondary. 

Timeliness mattered most: 

GCN: One of the biggest stumbling 

blocks to solving the work force crisis 

is the hiring process. People say it 

takes much too long. 

Kay Coles James: And they’re 

absolutely right. You can’t tell me a 

horror story that I would not nod my 

head and say “Yes, I agree.” * * * 

The question isn’t how do you get 

more people to come [apply for a 

Federal job]. The question is how do 

you build the internal capacity to 

respond to those individuals quickly, 

to keep them informed as you go 
through the process and to make a 

quick hiring decision so that the best 

and brightest don’t get annoyed with 

the process and leave?37 

Agencies expected automation to reduce the 

amount of work needed to fill a job. 

Tight budgets, the emphasis on “citizen-centered” government, and a 

desire to reduce administrative overhead costs are driving agencies to 

reduce the staff and work time devoted to administrative functions such 

as staffing.38 Agencies looked to automated hiring systems, which can 

simplify or perform many of the administrative tasks39 associated with 

paper-based hiring, to achieve these reductions. 

Agencies expected automation to improve the 

applicant pool and the quality of new 

employees. 

Agencies did not, however, expect to achieve these gains through 

better assessment. Agencies expected automated systems to be more 

consistent and less prone to bias (and, in this sense, more fair) than 

human examiners. But no agency asserted that the system would be 

more thorough or discriminating (in the positive sense) than a good 

human examiner. Instead, agencies anticipated—reasonably enough— 

that a more accessible, comprehensible, and efficient application process 

would produce a larger, better applicant pool, and that a faster 

assessment process would reduce the loss of good quality, “in-demand” 

candidates to other employers. 

Agencies expected automation to improve the 

collection and use of hiring data. 

Federal agencies, like other employers, are required to collect applicant 

flow data. In most cases, agencies must also maintain records sufficient 

to permit audit and reconstruction of the hiring process. Agencies 

expected that automated hiring systems would make it easier to collect, 
store, and retrieve these records. Agencies also planned to use their 

automated hiring systems to help them solicit customer feedback and 

produce basic measures such as time to hire. 

Agency Initiatives 

An increasing number of agencies are using 

automated hiring systems. 

The “early adopters” of automation were large agencies (such as 

Department of Defense agencies, the Department of the Interior, and 

the National Aeronautics and Space Administration) that had the staff 

and resources needed to undertake a major information technology 

initiative. As automated hiring systems have matured—reducing the 

need for subsequent adopters to develop detailed technical 

specifications and participate in software development—and their 

potential benefits have become apparent, more agencies have adopted 

one or more automated hiring systems. Nevertheless, automation 

remains more common in large agencies than in small ones, for reasons 

of cost and, possibly, return on investment.40 

While some agencies are still considering whether to adopt an 

automated system or which system(s) to use, other agencies with some 

automation experience are already considering changing systems. A 

small number appear to be considering multiple systems, sometimes for 

different occupations and sometimes as sequential processors for the 

same applicants. 

Agencies use automated hiring systems that are 

designed around Federal hiring processes and 

requirements. 

Current automated hiring systems replicate most of the paper-based 

processes they replace. This is not surprising, because the laws, 

regulations, and policies governing Federal hiring are extensive and 

often highly prescriptive. This greatly limits the imagination and 

discretion of agencies and the designers of automated hiring systems. 

Accordingly, the automated systems included in our study are 

“federalized”: they collect information, carry out processes, and apply 
rules particular to Federal staffing.41 Essential features of a 

“federalized” system include a way to: 

• Determine the basic eligibility of each applicant to be considered 

for Federal employment; 

• Adjudicate applicant claims for special consideration based on 

special factors (e.g., veterans’ preference, eligibility for career 

transition assistance programs42); 

• Assess applicants against the requirements of the job and make 

distinctions among them; 

• List qualified applicants in the proper order (including credit for 

special consideration claims) when referred to a manager for 

selection; and 

• Maintain documentation needed to allow third-party audit of 

referrals and selections. 

Most agencies have chosen from among five different automated hiring 

systems. These systems are, in alphabetical order: 

• Avue, a product of Avue Technologies Corporation; 

• Commerce Opportunities On-Line (COOL), developed by the U.S. 

Department of Commerce; 

• QuickHire, a product of QuickHire LLC, a division of Monster 

Government Solutions; 

• Resumix, a product of Yahoo! Resumix; and 

• USA Staffing, developed by OPM. 

Each system follows one of two models. We refer to the first model as 

“question-based.” In the question-based model, the system uses 

questions to determine whether, and how well, the applicants meet the 

requirements of the job being filled. Generally, questions are forcedchoice, requiring the respondent to choose from among a set of 

provided responses (such as “yes” or “no”). Questions of this type may 
be combined with a free-form narrative in which applicants can explain 

or support their response choices. Avue, COOL, QuickHire, and USA 

Staffing follow this model. We refer to the second model as 

“application-based.” In this model, the automated system applies logic 

to “read” an application and identify the applicant’s competencies and 

credentials. Resumix follows this model. Resumix differs from the 

other systems in one other important respect: it is not “federalized.” 

For this reason, agencies that use Resumix supplement it with “bolton” programs to collect information and apply decision rules specific to 

the Federal Government. Appendix C provides a more detailed 

description of how the two models operate. 

Agencies’ automated hiring systems retain much 

of the substance of paper-based hiring. 

We mentioned earlier in this report and have noted in previous 

reports43 that Federal agencies usually assess applicants in terms of 

their training and experience (T&E) instead of assessing them on 

factors such as job knowledge or cognitive ability.44 That pattern 

continues with automated hiring systems: all the systems we reviewed 

in this study are designed to assess training and experience.45 This is 

convenient—and encouraged by the structure of most existing 

qualification standards—but it is not necessarily effective. As discussed 

earlier, the value of a T&E rating depends on the rating method. This 

dependence, combined with the documented variability in the capability 

of agency delegated examining units,46 has significant implications for 

the value and use of automated hiring systems. 

Agencies’ automated hiring systems enable 

agencies to make some systematic changes in 

recruitment and assessment. 

As we noted earlier, Federal agencies have limited discretion in many 

aspects of hiring. Although automation does not change that fact, 

automation does make it easier to use some existing flexibilities. For 

example, automated hiring systems are well-suited to establishing and 

maintaining inventories (also known as standing registers) of qualified 

applicants. Inventories are particularly useful for organizations that 
engage in continuous recruitment for an occupation or have highvolume or “peak-and-valley” hiring needs. 

Automation also enables agencies to improve the assessment process, 

not just make it more efficient. The speed of automated systems makes 

it easier to follow an initial rating of training and experience with other 

rigorous assessments, such as a structured interview or a work sample 

test, and to factor those assessments into referral and selection 

decisions.47 The use of multiple assessments in sequence (an approach 

known as “multiple hurdles” or “phased assessment”) has always been 

desirable, but automation makes it more practicable. We provide 

additional information about the multiple hurdles approach in 

Appendix D. 

However, relatively few agencies have used automated hiring systems 

to reengineer their hiring process. Instead, agencies have tended to 

automate the existing process, an approach familiarly known as “paving 

the cowpath.” Most agencies continue to announce most jobs on a 

case-by-case basis, even in situations where it might be more effective 

to announce a job once and establish an inventory of candidates. Most 

agencies also continue to evaluate and refer applicants based on a single 

rating of training and experience.48 

We have already mentioned the key factors that have contributed to 

this approach: an emphasis on expediency over quality when hiring; 

the highly regulated Federal hiring process (discouraging 

inventiveness); qualification standards written in terms of experience 

(encouraging assessment through T&E); and—all too often—HR 

offices and staff that lack the support, time, or expertise needed to 

significantly change the process. 

Agencies took different approaches to 

implementing an automated hiring system. 

As discussed above, agencies used similar technologies and applied 

those technologies in conceptually similar ways to similar processes. 

However, agencies varied in how they introduced and implemented 

their automated hiring systems. Some agencies took a systematic 

approach and devoted considerable time and money to designing the 
hiring process, planning the introduction of the automated hiring 

system, and introducing the automated hiring system. Other agencies 

directed most of their initial attention and resources toward acquiring 

and configuring the automated hiring system. 

OPM Activities and Initiatives 

OPM is working on several initiatives that should (or could, if fully 

developed and implemented) improve the hiring process and the 

effectiveness of automated hiring systems. 

Governmentwide standards for the application 

process 

First, OPM is leading an e-government initiative called Recruitment 

One-Stop (ROS) that will consolidate and maximize gains from the use 

of technology in the hiring process. ROS has already produced major 

improvements in USAJOBS. Plans include an application builder that 

will work with all major hiring systems and a requirement for agencies 

to provide applicants with on-line access to the status of their 

applications. These steps will do much to simplify the search for a 

Federal job—a quest that is often gratuitously labor-intensive and 

frustrating. 

Second, in 2002 OPM published a “Pledge to Applicants” committing 

OPM and Federal agencies to an application process that will “enable 

rather than deter job seekers.”49 The pledge includes a reasonable 

application process, timely information on the status of job vacancies, 

and timely hiring decisions. This pledge recognizes that Federal 

agencies have obligations to applicants that go beyond compliance with 

the letter of the law and provides constructive goals for agencies’ hiring 

processes. MSPB supports this pledge, which contains several of our 

own standards for the hiring process. 

Experimentation with new methods of 

recruitment and assessment 

In 2002, OPM held a “virtual job fair” for information technology 

occupations. This job fair demonstrated the potential of Internet-based 
recruiting and the ability of automated hiring systems to process large 

numbers of applications quickly. In connection with that job fair, OPM 

also used a brief objective test to evaluate applicants, demonstrating 

that such assessment tools can be used effectively in a format other 

than paper and pencil. 

Experimentation with new qualification 

standards 

Most existing qualification standards for white-collar occupations are 

written in terms of experience and education. Such standards are 

problematic: they are prone to inconsistent application, their value in 

distinguishing good from poor applicants is limited, and they 

emphasize training and experience over other, potentially better, 

indicators of ability. Such standards are also, despite their surface 

simplicity, not always easy to use or to automate: their application can 

require much data and interpretation.50 

For two occupations, accounting and information technology 

management, OPM has developed and tested an alternative: job 

profiles, or qualification standards written in terms of competencies. 

This approach has considerable promise: job profiles can provide more 

detailed and (we believe) more valid descriptions of occupational 

requirements, and are more amenable to automated application than 

traditional qualification standards. However, OPM and agency 

experience with these profiles, although generally positive, clearly 

shows that job profiles are not a “quick fix” solution: they are costly to 

develop and require competent users and sound assessment methods. 

Much more must be done for this approach to become widespread and 

effective. We understand that OPM plans to move forward with 

competency-based qualification standards, building on experience from 

the pilot job profiles. 
Results and Agency 

Experience 

This section describes Federal agencies’ experiences with their 

automated hiring systems, including the results of automation, issues 

raised by the introduction of an automated hiring system, and “lessons 

learned.” 

Results varied. 

Automated systems are tools, not solutions. To use an analogy, a chef’s 

knife is merely a tool; its use does not guarantee good cooking. The 

quality of the cooking ultimately depends on other factors, such as the 

quality of the ingredients and the chef’s competence. Similarly, the use 

of an automated hiring system does not guarantee an easy-to-use 

application process, high-quality referrals, fast hiring, or increased 

efficiency. 

As discussed below, agencies devoted different levels of attention and 

resources to the introduction of an automated hiring system. It is not 

surprising, then, that referral quality—and every other important 

aspect of hiring, from applicant burden to timeliness to fairness— 

depended on how the agency used its chosen system(s). Some agencies 

were able to consistently refer promising candidates to hiring 

managers; others struggled to refer good (or even qualified) candidates. 

Some agencies succeeded in increasing the pool of applicants and 

reducing time to hire (or at least time to referral); others were buried 

under a figurative pile of applications. Some agencies provided a 

reasonable and understandable application and assessment process; 

others provided a process that many applicants considered bureaucratic 

and obscure.51 

Results were affected by how an agency 

introduced its automated hiring system. 

Hiring is a complex process that affects everyone in an organization, 

from executive to manager to employee. Moreover, the hiring process 
does not merely affect these groups, but also depends on each group’s 

constructive participation in the process. Thus, the introduction of an 

automated hiring system is much more complex than a routine upgrade 

to computer hardware or software. 

Agencies that approached this introduction in a systematic, 

comprehensive way—that engaged in what is commonly called “change 

management”—fared much better than agencies that did not.52 

Agencies did not succeed simply by acquiring the “right” hardware or 

software or having the “best” IT and HR professionals. Agencies that 

achieved good results were attentive to both technical issues (such as 

selection criteria, measurement, staffing policies, and IT infrastructure) 

and human issues (such as training, employee roles, and resistance to 

change). 

Results depended on the soundness and quality 

of “content” 

The adage “garbage in, garbage out” continues to apply to the hiring 

process. Automated hiring systems, like their paper-based 

predecessors, depend on what we call “content.” Content comprises (1) 

the job-related attributes (such as competencies) to be assessed, (2) the 

measures of those attributes, and (3) the methods used to translate 

those measurements into applicant ratings (e.g., scores or rating 

categories). Table 4 illustrates two points. First, content is the basis of 

any merit-based hiring process, paper-based or automated. Second, 

automated hiring systems use content in formats unlike those used in 

paper-based processes. 
Table 4. Comparison of Formats for Evaluating Training and 

Experience 

Content 

Element 

System Type 

Paper-Based Question-Driven Application-Driven 

Attributes Knowledges, skills, and abilities and other characteristics 

(KSA’s) or competencies. 

Measurement Rating of 

application 

against 

benchmarks 

Question 

responses 

Analysis of 

application text 

Translations Crediting plan Question 

response values 

and weights 

Keyword values and 

weights 

Agencies indicated that the development of good content required a 

substantial investment. Agencies could develop content in several 

ways, including contracting for technical assistance, convening panels 

of HR professionals and subject matter experts, and using recurring 

discussions between individual HR professionals and subject matter 

experts.53 Each way has advantages and disadvantages, but all require 

a substantial investment of time and money. 

Agency experience makes it quite clear this investment is essential: 

good content is a necessity, not a luxury, and good content cannot be 

acquired cheaply. Organizations that sought short cuts to developing 

content (such as “quick and dirty” adaptation of paper-based content or 

borrowing or buying another organization’s content) were almost 

always disappointed. These organizations found that their content was 

not effective: the assessment process failed to measure important 

attributes, failed to make useful distinctions among applicants, or 

both.54 The result was either inaccurate differentiation (e.g., highly 

qualified candidates rated lower than less qualified candidates) or 

minimal differentiation (e.g., candidates of varying quality levels all 

received similar ratings). 
Automated hiring systems, if used properly, can 

meet standards of merit-based hiring. 

Earlier in this report, we outlined 10 standards for merit-based hiring. 

Agencies believed—and we agree—that automated hiring systems can 

be part of a process that meets these standards. Specifically, automated 

systems can do much to make the hiring process: 

• Not unduly burdensome. Automated hiring systems cannot, at 

present, make it possible to apply for a Federal job without effort. 

Federal agencies simply require too much information for that to 

be possible. But technology can make the job search much more 

efficient and the application process less paper-intensive and timeconsuming. 

• Transparent. Automated hiring systems do not prevent agencies 

from providing clear, helpful descriptions of job requirements, 

desired skills, and the assessment process. In fact, as shown in 

demonstrations of planned upgrades to USAJOBS, effective use of 

technology can make this information much easier to find.55 But 

neither does technology force agencies to make their assessment 

criteria transparent. For example, although applicants using 

question-driven systems will usually be able to deduce what 

attributes (e.g., experience, skills, and credentials) the agency is 

seeking, they will not know what weight (if any) the agency will 

give those attributes unless the agency provides that information. 

Similarly, applicants using an application-driven system will not 

know what attributes the agency is seeking, let alone what weight 

will be given to those attributes, unless the agency chooses to 

disclose them.56 

• Thorough, fair, and able to consistently distinguish among 

applicants to identify the most promising. Automated hiring 

systems cannot meet these standards unaided. In their current 

form, they are not realistically capable of more than a rough and 

imperfect sorting of applicants. These systems’ initial evaluations 

of training and experience must be supplemented by human 

judgment—and subsequent assessment(s), manual or 

automated—if the process is to be fair and thorough, as opposed 

to impartially cursory. 
• Compliant with legal requirements and professional 

standards. Automated hiring systems can apply decision rules 

and mathematical formulas quickly and accurately. Yet agency 

compliance with legal requirements cannot be left entirely to 

automated systems. People must still decide which laws and 

regulations apply and whether to obey those laws and regulations. 

For example, automated hiring systems can do much to assure 

that veterans’ preference is applied when required—but they 

cannot force selecting officials to observe veterans’ preference. 

The same is true of compliance with professional standards (such 

as the Uniform Guidelines) and merit system principles. 

Automated hiring systems can contain databases and decision 

guides to help managers and HR professionals prepare job 

descriptions and choose effective, job-related selection criteria. 

But the systems can neither assure that the “virtual” job 

description accurately reflects the actual job, nor prevent 

managers from applying ineffective or inappropriate selection 

criteria, either online or “off line.”57 

• Able to withstand scrutiny. As noted above, automated hiring 

systems can support assessment practices and hiring decisions 

that meet legal and professional standards. And automated hiring 

systems can collect data on the hiring process, helping agencies 

better understand how that process is working and, when needed, 

demonstrate the soundness and legality of their hiring decisions. 

Agency experience suggests that initial ratings 

of training and experience—whether automated 

or manual—are better used to sort applicants 

than to select them. 

Agencies with effective “content” found that their automated hiring 

systems could make useful distinctions among applicants. For example, 

staff at the U.S. Geological Survey, which uses a question-driven 

automated hiring system, stated that a carefully chosen set of questions 

would usually produce a wide distribution of scores, with relatively few 

applicants earning very high scores. Similarly, staff from NASA, which 
uses an application-driven system, believed that their system produced 

good results when supported by thorough job analysis, a well-written 

vacancy announcement, and a sound rating procedure. These users 

also indicated that these distinctions were—with appropriate quality 

control measures in place—at least as clear and accurate as the 

distinctions previously made by people reading applications and scoring 

applicants. 

Nevertheless, our discussions with these and other agencies make it 

very clear that initial automated assessments of training and experience 

should not be used to make fine distinctions among applicants. This is 

not a criticism of automated hiring systems. Instead, it is a 

fundamental limitation of the underlying method of assessment and its 

context, whether automated or manual. 

As we have noted, ratings of training and experience provide only 

limited (albeit useful) insight into an applicant’s ability. That insight is 

further limited by practical constraints on the quantity and quality of 

information available to agencies during initial assessment and 

constraints on how much time and effort agencies can devote to 

evaluating that information. Moreover, regardless of automation, 

evaluation of training and experience is not a particularly effective tool 

for measuring many attributes—such as writing ability, problem 

solving, or interpersonal skills—that are important in many Federal 

jobs. 

For these reasons, while initial assessments of training and experience 

are useful for identifying an “A group” of promising candidates, they 

are much less effective at identifying a small, predetermined number of 

candidates to be considered to the exclusion of all other applicants.58 

Yet the latter approach is precisely how many agencies have used these 

assessments. This approach may have been necessitated by the realities 

of paper-based examining (many applications and few staff and limited 

time to process those applications). However, it has never been sound 

assessment practice or a good way to hire the best qualified people. 

Automation does not change this fact. 
Automated hiring systems, used properly, have 

great potential to improve how jobs are filled. 

As we have indicated above, agencies believe that their automated 

hiring systems are fully adequate for performing many hiring tasks. 

But automated systems can be more than “good enough.” Automated 

systems have potential to improve hiring processes and hiring results. 

In particular, automated hiring systems can help agencies meet higher 

standards of: 

• Timeliness. Agency experience generally confirms what logic 

suggests: that automation can significantly improve timeliness.59 

For example, HR staff at the Department of the Interior’s U.S. 

Geological Survey report that they can deliver a list of candidates 

to the hiring manager within 7 days of the closing of the vacancy 

announcement, in contrast to the 30 to 60 days that was typical 

before automation. Such speed was implausible (or prohibitively 

costly) when examining was paper-based and, as the quote below 

suggests, it surprises applicants who expect a slow-moving 

process: 

Interesting thing! I had a declination from one of my 

applicants. Okay, why is that interesting? She said 

she applied for my position figuring she would not 

hear from us for several MONTHS by which time she 

would know whether she and spouse would be moving 

to the area. She was astounded that I had a certificate 

two business days after the announcement closed. She 

declined consideration because she was not prepared to 

move for several months * * *.60 

But agencies also indicated that improvement is not guaranteed. 

Several conditions must be met to improve timeliness and to ensure 

that greater speed is not an empty achievement. These include an 

assessment process that consistently produces good (not just fast) 

results and hiring policies and practices adapted to automation. 

• Selection Quality. Automated hiring systems can improve the 

quality of new employees in two ways. First, these systems can 

improve the pool of applicants by making it easier to apply for 

jobs. Agencies noted that the improvement is not uniform; 

automating the application process tended to increase the number 

of unqualified or marginally qualified applicants as well as the 
number of good applicants. But agencies also believed that 

automation had improved their ability to reach and attract 

“passive applicants”61—prospects who were usually not willing 

to enter the “essay contest” typical of Federal paper-based hiring. 

Second, automated hiring systems can improve selection quality by 

helping managers identify the best candidates. Automated hiring 

systems give agencies several options for improving applicant 

assessment. Table 5 lists and briefly describes them. 

Table 5. Options for Improving Assessment Using Automated Hiring 

Systems 

Option Discussion 

Improve evaluation of 

training and 

experience. 

Compared to paper-based approaches, automated 

hiring systems can: 

• Assess applicants on more competencies; 

• Collect more specific information on an 

applicant’s competencies; and 

• Apply more complex decision rules when 

sorting applicants. 

Multiple hurdles— 

follow initial 

assessment with one or 

more structured 

assessments. 

Automated systems can: 

• Provide guidance on the content and conduct 

of secondary assessments. For example, an 

automated system could “suggest” interview 

questions and provide a scale for rating 

candidates’ responses; 

• Help administer the assessment; and 

• Assist in the recordkeeping and 

administration associated with the use of 

multiple assessments. 

Use alternatives to 

evaluation based on 

training and 

experience. 

Most of the automated hiring systems used by 

Federal agencies can or should soon be able to 

support objective tests (such as job knowledge 

and cognitive ability tests). The Federal 

Government has traditionally used such tests for 

both screening and ranking, but this is not 

mandatory. Such tests could be used either for 

the initial screening or to make distinctions 

within a group of promising applicants after the 

initial screening. 

Technology can also support other methods of 

assessment, such as work samples or assessment 
centers. 

Use category rating. 

62 

In 

multiple hurdles. 

Category rating is not a method of assessment; it 

is a method of sorting and selection. Under 

category rating, an agency sorts candidates into 

quality groups (e.g., “C,” “B,” and “A”) and selects 

from the highest group, instead of assigning 

numerical scores, ranking candidates (that is, 

referring candidates in order of score), and 

following the “rule of three” in selection.

We list this option here because it supports the 

assessment options described above. 

particular, it is an ideal complement to the use of 

As discussed above, there are limits to the precision and practical uses 

of the initial rating of an applicant’s training and experience. 

Nevertheless, automated hiring systems can help make these ratings 

more useful, by improving: 

• The quantity of information that can be collected from 

applicants. For example, few applicants would tolerate a request 

to provide specific, written descriptions of training, experience, 

and accomplishments on 10 distinct competencies. Compared to 

paper-based hiring, automated hiring systems can collect more 

information with much less work on the applicant’s part. 

• The quality of information that can be collected from 

applicants. Under paper-based examining, it is rarely feasible to 

request that applicants provide precise—much less 

standardized—descriptions of their experience and 

accomplishments. Automated hiring systems make this possible, 

within limits; and 

• The sophistication of applicant sorting. Under paper-based 

examining, raters rarely have the time or information needed to 

evaluate applicants on more than a few KSA’s or competencies. 

Consequently, the method used to sort and refer applicants is 

usually quite crude: the more points, the better. As noted above, 

automated systems may have more information to work with— 

and if they do, these systems can apply decision rules that reflect a 
more subtle understanding of jobs and applicants’ 

qualifications.63 

Agencies that made systematic improvements in 

the hiring process had better results. 

Few agencies we talked with had made extensive use of the options 

described in Table 5.64 However, agency experience suggests that 

they should: agencies that used automation to improve the hiring 

process, instead of simply replicating the previous paper-based process, 

achieved better results. For example, the U.S. Geological Survey 

devoted substantial time and effort to improving the initial assessment 

of applicants, yielding a process that was much better able to 

distinguish good from mediocre applicants. The National Aeronautics 

and Space Administration (NASA) used examining flexibilities to allow 

managers to consider more highly qualified applicants than they could 

under standard methods of referral and selection.65 

The options described above may be used singly or in combination. 

However, we believe that the category rating option should not be used 

alone, but, at the very least, should be used with the multiple hurdles 

option.66 This is because the process that was used to sort candidates 

into categories probably evaluated only a limited number of 

competencies using limited information. If so, then further rigorous 

assessment tools (i.e., multiple hurdles) should be used before a 

selection is made. Not to do so is to treat as equals candidates who may 

differ in important and measurable ways, or to rely on superficial 

assessment (or nonassessments such as random numbers67) to make 

specious distinctions among candidates, instead of valid distinctions. 

Automated hiring systems raise policy issues. 

Agencies indicated that automated hiring systems are a mixed blessing. 

Automation can expand the applicant pool (i.e., increase the number of 

applications)—but the agency must then process those applications. 

Automation can be used to create a paperless application process—but 

the agency must then do without paper or find an adequate substitute. 

Automation greatly reduces human involvement in initial screening 

and sorting of applicants—reducing workload and the potential for 
errors in judgment and subtle or overt discrimination—but also 

introduces the possibility that applicants’ inadvertent or deliberate 

misrepresentation of their qualifications will go undetected and 

uncorrected. Consequently, agencies have found that they need ways 

to: (1) balance applicant burden and agency risk; (2) manage volume 

(that is, a large number of applications); and (3) assure the integrity of 

the assessment and referral process. 

Balancing Applicant Burden and Agency Risk. This issue is one of 

documentation: what to require and when to require it. In the past, 

most agencies required applicants to submit documentation of their 

qualifications and eligibility (such as college transcripts, veterans’ 

preference claims forms and documentation, and lists of awards and 

training received) at the time of application. This made sense when the 

process of examining relied on paper (and the postal system to deliver 

that paper). Most agencies also took a “no risk” approach to rating, 

usually rating candidates ineligible unless they proved their eligibility. 

That caution reflects, in part, the Federal system’s lack of tolerance for 

error. But caution also cost little in a paper-based system: the 

documents needed to verify eligibility and the human examiners needed 

to review and analyze those documents were both readily available. 

The introduction of an automated hiring system raises the cost of 

caution. First, requiring applicants to provide transcripts, service 

records, and the like along with their resume or questionnaire may 

deter people from applying—especially highly accomplished applicants 

who can attract competing offers with little effort. Second, such a 

requirement simply creates two parallel application processes, one 

paperless and one paper-based, forfeiting many of the possible benefits 

of automation. 

Thus, it is problematic for agencies to require applicants to document 

each claim and credential “up front.” Yet neither can agencies afford to 

go paperless and accept all applicant claims at face value. At some 

point, agencies still need paper (or its electronic equivalent) to 

document applicants’ citizenship, education, licensures, military service, 

and the like. 

Managing Volume. This issue is not new; agencies have long 

struggled with how to process large numbers of applications. Yet 
automation makes the issue more pressing. Compared to paper-based 

job announcement and application processes, well-designed automated 

processes will make it easy for applicants to find job information and 

apply for jobs. The likely consequence is that agencies will receive 

unprecedentedly large numbers of applications. Agencies must find 

ways to handle these applications that are both fair and expeditious. 

Otherwise, agencies will (1) be buried by applications, (2) seek to 

artificially limit the number of applications received, possibly by 

establishing new barriers to application, or (3) resort to cursory 

assessment of applicants, driven by considerations of cost and time. 

Assuring the Integrity of the Assessment and Referral Process. 

“Assuring the integrity of the assessment and referral process” means 

more than following laws and regulations and treating candidates evenhandedly. It means assuring that the process is fulfilling its primary 

purpose: making consistent, accurate, and merit-based distinctions 

among applicants. Automated hiring systems can help achieve this 

purpose. They are less likely than people to lose or misfile an 

application; they do not tire; they bring neither prejudices nor feelings 

to the assessment process; they do not make mathematical errors. 

However, automated systems’ relative freedom from these human 

failings does not mean that their results can be accepted at face value. 

As we discuss below, research and agencies’ experience clearly show 

that human judgment and intervention are still needed to maintain the 

integrity of the hiring process. 

Relying solely on applicant-provided information and applicant selfratings to make merit-based distinctions is ill-advised. Even applicants 

who try to be honest (who, we hope, constitute the vast majority of 

applicants for Federal jobs) may systematically overestimate or 

underestimate their abilities. Research suggests that individuals may 

tend to self-enhancement (i.e., an unduly positive view of one’s abilities 

and accomplishments) or self-diminishment (i.e., an unduly modest selfimage).68 Table 6 summarizes the risks and consequences associated 

with both overstatement and understatement. 
Table 6. Consequences and Risks of Self-Rating Errors 

Error Consequences Level of Risk 

Overstatement 

(Self-Enhancement) 

Less qualified applicants 

may be “screened in” and 

referred to the hiring 

manager. Depending on the 

referral and selection rules 

in effect, this may exclude 

other, better qualified 

candidates from 

consideration. 

If this error occurs, it can be 

detected and corrected, 

especially if the assessment 

is conducted through 

“multiple hurdles.” 

Applicants have a clear 

incentive to overstate their 

qualifications.69 

Overstatement may be 

more likely among 

candidates who have 

unrealistically positive selfimages. 

Understatement 

(SelfDiminishment) 

Highly qualified applicants 

may be excluded from 

consideration. If undetected, 

the error is fatal to these 

applicants’ employment 

chances. 

Detection is unlikely, even 

under a multiple hurdles 

approach, unless the agency 

takes steps to review and 

correct self-ratings. 

Applicants have no 

incentive to knowingly 

understate their 

qualifications. However, 

understatement may occur 

for reasons such as 

modesty or an imperfect fit 

between applicants’ 

experiences and the 

questions they are asked to 

answer during a self-rating 

exercise. 

The Recommendations section of this report provides some 

suggestions on how agencies can manage these risks. 

Automation’s long-term effects on efficiency are 

not yet known. 

Many agencies hoped that automated hiring systems would allow them 

to make the hiring process more efficient, not just faster. Some 

agencies anticipated that greater efficiency would enable them to 
reduce the level of HR staff without compromising hiring timeliness or 

quality. 

Yet the agencies we talked to, including those that had achieved 

significant improvements in timeliness, did not tell us that they had 

significantly reduced their level of HR staff. Moreover, Federal 

employment data provide little evidence of such reductions.70 Possible 

explanations for this include: 

• Increased volume of applications. Efficiency gains may be 

offset by an increased number of applications; 

• The “learning curve.” Automated hiring systems can require 

new skills, from the mechanical (“which buttons to push”) to the 

technical (e.g., developing rating procedures in the format 

required by the automated hiring system) to the professional (e.g., 

working with managers as a “consultant” instead of “processor”). 

Until HR staff develop these skills and become fully productive, 

staff reductions make little sense; 

• Transitional work. Automated systems use content in formats 

that differ from that used in paper-based hiring. For example, 

existing KSA’s and crediting plans must be rewritten as questions 

and response scales or as keywords and grammar. Moreover, 

many agencies found that it was not sufficient to simply convert 

existing “content” into the format used by the automated hiring 

system, because that content was itself dated or flawed. 

Consequently, many agencies had to develop new content or 

spend much time “fixing” or reissuing referral lists; 

• Rising expectations. Federal agencies are paying greater 

attention to human capital issues at both the strategic level (e.g., 

linking staffing levels and recruiting efforts to organizational 

goals) and the operational level (e.g., writing better vacancy 

announcements). This attention may have (1) limited any shortterm gains in efficiency by preventing automation of “business as 

usual” or (2) diverted any gains in efficiency in the hiring process 

achieved through automation;71 and 
• Aging of “content.” Jobs, work methods, and terminology are 

not static. They change, as do agencies’ missions and workforce 

needs. Applicants also change. Over time, applicants can discern 

which questionnaire responses (or words and phrases) produce 

higher ratings, and unscrupulous applicants may provide the 

“right” answers regardless of their actual qualifications and 

accomplishments. For these reasons, agencies cannot develop 

content once and reuse it indefinitely, but must continually 

maintain and refine their content—which requires work. 

Thus, it appears that automation’s promised gains in efficiency (1) have 

yet to be fully realized; (2) are less than some agencies hoped; or (3) will 

not necessarily be reflected in HR staff levels. It is simply too soon to 

tell. 

Competent HR professionals are critical to 

effective use of automated hiring systems. 

Automated systems may eventually perform much work previously or 

currently performed by HR staff.72 Yet these systems do not eliminate 

the need for competent HR professionals. On the contrary. Designing 

and administering an effective hiring process requires knowledge of 

many areas, including the organization’s mission and workforce needs, 

employment laws and regulations, negotiated agreements, and 

assessment principles and methods. HR professionals are more likely 

than any other group (including line managers, attorneys, contractors, 

and personnel psychologists) to have knowledge of all of these areas. 

In fact, agencies found that the introduction of an automated hiring 

system increases the need for competent HR professionals. First, it is 

much more difficult to design new tools and processes than to 

administer existing ones. For example, an HR assistant can apply a 

predeveloped crediting plan with reasonable consistency and accuracy. 

An HR assistant will be much harder pressed to translate a written 

crediting plan into questions, keywords, and decision rules. That task 

requires both an understanding of the fundamental properties of an 

automated hiring system and experience in developing assessment 

tools. 
Second, it appears that the need for new and greater skills is not 

transitory. Instead, automated hiring systems “raise the bar” for HR 

professionals. These systems can minimize or eliminate tasks such as 

screening applicants, calculating ratings, and compiling referral lists, 

but they do not eliminate high-level functions such as developing 

assessment strategies and rating procedures. Moreover, it appears that 

these high-level functions are becoming more demanding, not less. For 

example, the agencies that had greater success with automated hiring 

systems used selection criteria and rating procedures based on 

extensive data collection and analysis. The sophistication of these 

criteria and procedures far exceeded that of most “traditional” KSA’s 

and crediting plans. 

Finally, automated hiring systems do not merely require that HR 

professionals know more. They also require that HR professionals do 

more. HR professionals cannot be passive operators or observers of 

automated hiring systems. HR professionals must identify and choose 

effective selection criteria; they must monitor the operation of the 

automated hiring system to ensure that it is working properly; and they 

must ensure that the content (and not merely the format) of referral 

lists is satisfactory. These tasks require judgment, decisiveness, and 

acceptance of accountability.73 

Agencies reported that not all HR professionals were prepared to 

perform at this higher level.74 Causes varied. In some cases, staff had 

never received appropriate training or acquired the necessary skills. In 

other cases, important skills and behaviors had simply fallen into 

disuse. Whatever the cause, many agencies found that they needed to 

provide coaching or training so that HR staff could perform new 

functions and assume new roles. 
Summary and Conclusions

Hiring Decisions and Assessment Tools 

• Hiring decisions have long-term consequences for an 

organization’s productivity and performance. Therefore, 

quality—not speed—should be the primary measure of the 

success of hiring decisions and the underlying hiring process. 

• Good hiring decisions require valid selection criteria and effective 

assessment tools. Managers’ good intentions and unaided 

judgment will not suffice. 

• Assessment tools are not all created equal. Evaluation of training 

and experience—the Federal Government’s most commonly-used 

assessment tool—can be useful, but only if that evaluation gives 

credit only for training and experience that is clearly relevant to 

the job’s roles and duties. Mechanical evaluations of training and 

experience, such as those that give “automatic” credit for years of 

experience or education, have little value. 

• Qualification standards based on training and experience, the 

Federal Government’s most common type, may be a barrier to 

efficient, effective hiring. Their validity is uncertain and they may 

exclude qualified candidates who have followed atypical career 

paths.75 The primary argument for use of such standards— 

administrative convenience—has become less compelling with the 

emergence of automated hiring systems and competency-based 

qualifications. 

The Potential of Automated Hiring Systems 

• Automated hiring systems offer speed, consistency, and 

economies of scale. However, these benefits do not guarantee 

good results. Results depend on how agencies use and support 

their automated hiring systems. 

• Automated hiring systems can help agencies distribute job 

information quickly and widely and reduce applicant burden. 
However, automated hiring systems are not a substitute for 

recruitment programs. They are merely tools to support 

recruitment. 

• Agency experience indicates that both the question-driven and 

application-driven approaches to assessment can produce usable 

results. However, the approaches are very different. Agencies 

need to understand the capabilities and limitations of their chosen 

approach and their chosen automated hiring system(s). 

• Automated hiring systems can and should be used to improve the 

initial sorting of applicants. However, these systems do not 

transcend the limitations of any initial assessment of applicants. 

Thus, automated hiring systems are better suited to making 

broad distinctions among applicants (e.g., not qualified, qualified, 

highly qualified) than to making fine distinctions (e.g., selecting 

among highly qualified applicants). 

• Automated hiring systems are not restricted to evaluating 

training and experience. These systems can be used to administer 

or support other types of assessment, such as objective tests and 

structured interviews. 

• Automated hiring systems can be used to analyze and understand 

the hiring process—not just administer it. Federal agencies have 

only begun to make use of this ability. 

• Automated hiring systems may provide long-term gains in 

efficiency. However, the introduction of an automated hiring 

system requires much transitional work, requires new skills, and 

changes the roles of managers and HR professionals. For these 

reasons, it may be unrealistic to expect immediate gains in 

efficiency, especially ones that translate into HR staff reductions. 

Designing a Hiring Process and Using Automated Hiring Systems 

• Agencies must balance competing objectives when designing a 

hiring process. Agencies should take care to ensure that 

mandates—such as legal compliance and fair treatment of 

applicants—and immediate concerns—such as reducing time to 
hire—do not overwhelm the most important objective: 

systematic identification of the best person(s) for the job. 

• Agencies should use automated hiring systems to do more than 

simply “pave the cowpath.” Agencies that materially changed and 

improved the hiring process, instead of merely replicating the 

preexisting paper-based process, generally fared better than 

agencies that sought to faithfully replicate the paper-based hiring 

process. The agencies that achieved the best results changed 

their assessment and referral processes, not just the application 

process. 

• The quality of selections depends on “content”—the criteria and 

methods used to distinguish among applicants. Agencies using 

automated hiring systems report that good content requires an 

investment of time and effort, just as it does in manual systems. 

Use of Automated Hiring Systems 

• Agencies should evaluate their paper-based content before 

translating it into the format used by their automated hiring 

system. Experience suggests the relevance and soundness of 

paper-based content cannot be assumed. 

• Automated hiring systems can sharply increase the volume 

(number) of applications received. Agencies need to develop 

strategies to handle this volume; they cannot rely solely on 

automated hiring systems to do this.

• Automated hiring systems rely heavily on self-reported 

information. Agencies have found that undue reliance on selfreported information can compromise the quality of referrals and 

the integrity of the hiring process. Consequently, agencies found 

that quality control—a procedure for verifying and documenting 

applicant’s self-ratings, and modifying ratings, referrals, or both 

when appropriate—is critical to consistent referral quality and the 

integrity of the hiring process. 
Recommendations

We offer the following recommendations to maximize the benefits of 

using automated hiring systems and to improve the Federal hiring 

process. For convenience, Appendix F lists these recommendations on 

a single page and links them to factors critical to the successful use of 

an automated hiring system. 

Federal departments and agencies should— 

A. With respect to leadership: 

1. Manage hiring as a critical business process, not an administrative 

function. 

The obvious purpose of the hiring process is to hire the best 

possible employees. Unaided, even the most intelligent and 

well-intentioned managers cannot consistently make good 

hiring decisions. They need valid selection criteria and 

assessment procedures. Agency leaders must understand the 

importance and nature of the hiring process if they are to 

invest the resources, attention, and leadership needed to 

develop and implement a successful automated hiring process. 

2. Manage the introduction of an automated hiring system as an 

organizational change initiative, not as an information technology or 

an “HR office” initiative. 

The success of the hiring process depends on valid selection 

criteria and assessment methods, but it also depends on people. 

Introducing an automated hiring system changes almost every 

significant aspect of the hiring process, including the roles of 

managers, HR staff, and applicants. For this reason, agencies 

should take an equally comprehensive approach to the design 

and implementation of an automated hiring system. 

3. Invest appropriate resources in the design, implementation, and 

operation of the hiring system. 

“Resources” include dollars, staff, and training. The purchase, 

lease, or in-house development of an automated hiring system 

is only a part of the total investment cost. Evaluating and 

improving selection criteria, developing effective questions or 
grammar,76 training HR staffs and line managers—these 

measures come at a cost but they provide a return. If not done, 

or not done well, then the money spent to obtain the system 

becomes merely spent money with nothing to show for it. 

B. With respect to designing the hiring process: 

4. Use automated hiring systems to support recruitment programs, not to 

replace them. 

Automated hiring systems can make it easier for job seekers to 

learn about an agency’s job openings and to apply for available 

jobs, but they are not substitutes for recruiting programs. 

Agencies should avoid burdening their automated hiring 

systems with such expectations. 

5. Emphasize selection quality over less-important outcomes such as cost 

and efficiency. 

When introducing an automated hiring system, agencies 

should not seek gains in timeliness at the expense of referral 

quality. Such gains will be illusory or come at the price of 

lowered organizational performance. Instead, agencies should 

keep their focus on the quality of selections and use automated 

hiring systems in the way most likely to produce high-quality 

referrals and selections. 

For example, we encourage agencies to consider using 

automated hiring systems to sort applicants (instead of making 

fine distinctions based on an initial assessment of applicants), 

and then to use one or more subsequent assessment tools 

(which may be manual or supported by an automated system) 

to make further qualitative distinctions between applicants. 

This “multiple hurdles” approach requires more time, initially, 

than a single assessment—but it should produce better 

referrals and selections and, in so doing, save both time and 

money in the long run. 

6. Evaluate the soundness and usefulness of existing assessment tools 

before “converting” them for use by their automated hiring systems. 

This step is necessary for at least two reasons. First, existing 

assessment tools may not have aged well—they may be so 

outdated that their usefulness as manual assessment tools has 
substantially deteriorated. Second, existing tools may not 

translate easily or at all to the methodology used by the 

automated hiring system, essentially rendering the system’s 

results useless unless this step is first accomplished. 

C. With respect to implementing an automated hiring system: 

7. Invest in “content”—selection criteria, methods for assessing 

applicants against those criteria, and methods for translating 

assessment results into ratings and referrals. 

An automated hiring system’s value depends on the quality of 

the information and decision rules it uses to process 

applications. Resources spent to develop good content will pay 

substantial dividends; failure to invest in content makes it 

likely that the automated assessment process will produce poor 

results. 

8. Communicate roles and expectations to line managers, HR 

professionals, and applicants. 

All of these groups will find their roles changed as a result of 

automation—and all will find that they have roles to play. 

Agency leaders need to clearly define these roles and 

communicate their expectations. Otherwise, individuals may 

misunderstand and fail to perform their assigned role(s). Line 

managers, in particular, need to understand that automated 

hiring systems do not relieve them of their responsibilities, 

including active participation in job analysis and the 

assessment of applicants. As one agency HR manager noted, 

“Managers would like for us to provide them the ideal 

candidate with no work on their part. We can’t.” 

9. Ensure the competence of HR professionals. 

Many HR professionals who performed adequately in a paperbased environment are less prepared for an environment that 

demands proficiency with technology and a higher level of 

professional knowledge and judgment. Important skills may 

have atrophied or be lacking. Since HR professionals play 

major roles in making automated hiring systems work, 

ensuring their competence is essential. 
10. Adopt documentation requirements that balance applicant burden and 

risk. 

Historically, many agencies took a “no risk” approach to 

documentation and ratings: they required applicants to 

provide complete documentation of their credentials and 

entitlements with their initial application, and they determined 

that a standard (such as an experience or education 

requirement) was unmet unless documentation had been 

provided. That approach (captured in the axiom “when in 

doubt, rate them out”) was understandable when the hiring 

process was paper-based—documentation was generally 

available, and the time an agency needed to obtain missing 

documentation was prohibitive—but it makes less sense when 

automated hiring systems are used. Yet a totally paperless 

process remains out of reach; documentation of education, 

veterans’ preference, citizenship, and other important matters 

remains essential. 

We suggest that agencies take an approach between these two 

extremes: postpone requesting documentation when (1) the 

risk of error is low and (2) the cost of error is not excessive. 

For example, an agency recruiting high-level research 

scientists probably does not need to require college transcripts 

at the time of application: any serious candidate for the job, 

who will have an extensive record of professional 

accomplishment, is also likely to have the required 

education.77 Moreover, requesting documentation later (for 

example, at the time of interview) should permit the agency to 

“recover” from any errors. 

We do not provide detailed guidelines on this matter. The 

costs of requesting documentation and the risks of forgoing 

documentation depend on many factors, including the agency’s 

hiring process, the agency’s use of technology, applicable laws 

and regulations, the applicant pool, and the type of job being 

filled. For this reason, each agency must decide for itself what 

documentation to require and when to require it. 
11. Adopt a “triage” strategy to manage increased volume. 

The introduction of an automated hiring system can sharply 

increase the volume of applications received. Unfortunately, 

the introduction of an automated hiring system is not, by itself, 

an answer to the challenges posed by increased volume. 

Automated hiring systems do not relieve agencies of the 

responsibility to operate a fair, open, equitable, and rigorous 

hiring process. And, as we have discussed in this report, 

automated hiring systems cannot perform all the tasks required 

by such a process. Yet managers and HR professionals cannot 

spend an equal amount of time and attention on each applicant, 

unless that amount is minimal—far too little for thorough 

assessment, let alone “selling” the job and the organization. 

Thus, automated hiring systems demand that HR professionals 

and managers use their time wisely. We suggest triage: deal 

with all applicants fairly, promptly, and courteously—but 

devote limited time and attention to unqualified or average 

applicants; devote more time to distinguishing the “A group” 

from the “B group”; and devote considerable time to assessing 

the “A group.” Our suggested approach to quality control, 

discussed below, reflects this approach. 

We reiterate that a well-designed hiring process—one that is 

transparent, reasonable, rigorous, and uses technology 

effectively—is essential. An agency with a poorly designed or 

poorly administered hiring process, whether automated or 

manual, will surely be inundated with questions from confused 

applicants, inquiries from candidates left “in the dark,” and 

complaints from dissatisfied applicants. Effective triage is not 

possible when every applicant needs personal assistance. 

12. Implement quality control measures. 

Automated systems rely heavily on information provided by 

job applicants. That reliance is efficient and often appropriate. 

However, agencies must not permit considerations of efficiency 

and equitable treatment to compromise the quality of referrals 

or the integrity of the hiring process. Agencies cannot expect 

applicants and automated hiring system algorithms to do a 

flawless job of screening out unqualified applicants and 

identifying the best applicants; the judgments of HR 
professionals and subject matter experts continue to have an 

important place in the hiring process. Quality control 

measures, including review and verification of selected 

applicants’ self-ratings and actual accomplishments and 

credentials, are still needed to ensure that the right candidates 

are referred. 

We believe it is neither necessary nor practicable to manually 

review the qualifications and self-ratings of each applicant. In 

most cases, it is reasonable to accept the self-ratings of 

applicants who are unqualified or well below the referral 

threshold. Agencies should generally reserve manual review 

and verification prior to referral for those situations where 

comparatively minor self-rating errors may have significant 

consequences, such as those described in Table 7. 

Table 7. Situations That May Warrant Manual Review of SelfRatings 

Situation Rationale for Review 

The applicant is rated 

at or above the referral 

threshold. 

If the number of candidates referred is fixed (e.g., 

when the “rule of three” applies), overstatement 

or misrating may prevent consideration of a more 

highly qualified candidate. Also, referral of a 

candidate who has overstated his or her 

qualifications can create negative perceptions of 

the hiring process. 

The applicant is rated 

just under the referral 

threshold. 

Understatement or misrating may eliminate an 

excellent candidate from consideration. 

The applicant is 

entitled to special 

selection 

consideration, but 

does not appear to 

meet one or more 

requirements. 

The selection decision may be compromised if the 

applicant actually does meet requirements. 

13. Systematically evaluate and improve the hiring process. 

Even a well-designed automated hiring process may not 

produce the desired results at first. Adjustments and “mid-
course corrections” will almost certainly be needed. Systematic 

collection and analysis of data on the operation of the hiring 

process and its outcomes can help agencies identify and make 

these adjustments and corrections. 

Systematic evaluation should go beyond collecting routine 

measures of satisfaction and efficiency. Automated hiring 

systems can collect data, ranging from referral source to 

applicant demographics to the assessment criteria used, with 

much greater success and at much lower cost than is possible 

with manual systems. Analysis of such data can improve hiring 

effectiveness (for example, by identifying a need for targeted 

recruiting or by identifying assessment strategies that are 

particularly good or poor at making distinctions among 

applicants) and help agencies comply with merit principles, 

public policy goals, and other requirements. 

The Director and staff of the Office of Personnel Management 

should, in cooperation with Federal agencies— 

1. Establish a core application document and Governmentwide 

“applicant service standards.” 

All agencies require applicants to provide basic information 

such as dates of employment, job titles and major duties, and 

education. Yet an applicant who wishes to apply for jobs in 

several different agencies cannot simply provide this 

information once. Instead, that applicant may have to provide 

this information several times, in several different formats. A 

standardized electronic “core application document” that 

contains basic, universally-required information, such as name, 

telephone number, jobs held, dates of employment could reduce 

the need for repetitive data entry and submission. Such a core 

document would not be a complete employment application— 

applicants would still need to provide agency-and job-specific 

information—but it would simplify the application process. 

All agencies should provide applicants with information about 

matters such as the agency’s receipt of an application, the 

agency’s overall assessment of the applicant’s qualifications, 

and the status of the job(s) being filled. Most provide some 

information—but not all do. Applicants deserve better. A 
Governmentwide “applicant service standard” would give 

applicants a clearer sense of what information will be given 

them, and commit agencies to provide that information. 

OPM currently plans to do both as part of its Recruitment 

One-Stop e-government initiative. We support these plans 

because they can reduce applicant burden and improve 

communication with applicants—areas where the Federal 

Government needs to improve. OPM’s plans, as described to 

us, will not prevent agencies from (1) selecting a particular 

automated hiring system; (2) obtaining additional information 

from applicants beyond the minimum specified by OPM; or (3) 

selecting and using the assessment methods they deem most 

appropriate. Specifically, although agencies would be required 

to accept this core document, they would not be required to 

make it the sole (or even primary) basis of their assessment 

process. 

2. Move toward competency-based qualification standards. 

OPM has already developed and pilot-tested competency-based 

qualification standards (called job profiles) for two occupations. 

Such standards should, in the long term, provide a more valid 

and more usable basis for evaluating applicants than existing 

training- and experience-based qualification standards. 

(Appendix E, “An Argument for Competency-Based 

Qualification Standards,” provides our rationale for this 

assertion.) Accordingly, we recommend that OPM proceed 

from experimentation to active development and publication of 

job profiles. 

Agencies will require assistance to use job profiles effectively. First, 

job profiles generally do not contain the clear, if often arbitrary, “bright 

lines” found in existing qualification standards (e.g., “1 year of 

experience” or “24 credit hours”). Second, job profiles and their 

underlying competency models contain “soft” competencies such as 

flexibility and continual learning that are not particularly easy to 

measure. Nonetheless, it is important that agencies develop the ability 

to measure such competencies. The potential benefits of job profiles 

will not be realized if agencies avoid assessing applicants on such 

competencies, or if they measure those competencies poorly. OPM 
leadership—in the form of education, training, and development and 

distribution of assessment tools and guidelines—is needed to help 

agencies develop the required measurement capability.78 We note 

that the development of this capacity is a joint venture between OPM 

and the agencies. Agencies must also, as recommended above, invest in 

their own HR staff and in assessment tools; they cannot expect OPM to 

provide all necessary resources and expertise. 
Appendix A—An Overview of 

the Federal Hiring Process 

The information in this appendix is adapted from the OPM’s 2003 

Delegated Examining Operations Handbook.79 The sequence of steps 

in the table is illustrative; not all are performed each time a job is filled, 

and some steps may be combined or reordered. 

Steps in the Hiring Process— 

The table below outlines the major steps in the hiring process. The 

text following the table explains these steps in more detail. 

Major Steps in the Federal Hiring Process 

Step Action Who Performs 

1 Define the job and conduct a job analysis 

to identify job requirements. 

HR staff, working 

with the selecting 

official, subject 

matter experts, or 

functional experts 

(such as staff from 

an OPM service 

center or outside 

consultants). 

Selecting official. 

2 Select assessment tools. 

3 Develop a rating procedure—a method for 

distinguishing among qualified applicants. 

4 Recruit, publicize the job, and receive 

applications. 

5 Review applications for legal requirements. 

6 Assess minimum qualifications—determine 

whether an applicants meet minimum 

requirements. 

7 Assess relative qualifications—make 

distinctions among candidates (qualified 

applicants). 

8 Issue a list of candidates to the selecting 

official. 

9 Assess candidates. 

10 Select a candidate (or candidates). 
Step 1—Define and classify the job and identify job requirements. 

Job requirements are identified through a process known as job 

analysis. OPM defines job analysis as “a systematic procedure for 

gathering, documenting, and analyzing information about the content, 

context, and requirements of the job.” Job analysis produces (1) a list of 

important job roles or tasks and (2) a list of attributes needed to 

perform the tasks successfully. Generally, these attributes fall into one 

of three categories (knowledge, skill, or ability—referred to collectively 

as KSA’s or competencies). 

Step 2—Select assessment tools. OPM-issued qualification 

standards may specify how agencies will assess minimum qualifications, 

but agencies generally can determine how they will assess relative 

qualifications. (The concepts of minimum qualifications and relative 

qualifications are discussed below.) The most frequently used 

assessment tool is a rating of training and experience. Other 

assessment tools that agencies may use include objective tests, work 

sample tests, assessment centers, and unstructured and structured 

interviews. 

Step 3—Develop a rating procedure. A rating procedure is a plan 

for making job-related distinctions between candidates.80 

Components of a rating procedure include (1) the competencies to be 

evaluated and the assessment tool(s) that will be used to evaluate those 

competencies; (2) benchmarks for evaluating applicants, such as a 

crediting plan for rating training and experience or a scoring key for an 

objective test; and (3) decision rules for translating the results into 

numerical scores or qualitative groupings. 

Step 4—Recruit, publicize the job, and receive applications. 

Generally, an agency seeking to hire from outside its own workforce 

must issue a vacancy announcement and post that announcement on 

USAJOBS, the Federal Government’s web-based job information site. 

The extent of recruitment varies; often, recruitment is limited to 

posting. 

Step 5—Review applications for legal requirements. This step 

determines whether an applicant can be considered for appointment 

and how the applicant will be considered. Factors may include age, 

citizenship, previous Federal service, entitlement to veterans’ 
preference, and meeting criteria for special employment programs such 

as the Interagency Career Transition Assistance Program (ICTAP). 

Step 6—Assess minimum qualifications. The agency determines 

whether an applicant meets the minimum requirements for the job. 

These requirements are usually outlined in OPM-issued qualification 

standards. The standards generally stipulate that applicants must 

possess a certain quantity and quality of experience; for some 

occupations, the standards may also include a positive educational 

requirement or a written test requirement. Applicants who do not 

meet minimum qualification requirements are not further considered. 

Step 7—Assess relative qualifications. Assessment of relative 

qualifications determines how well a qualified applicant (now a 

“candidate”) is likely to perform. Candidates are assessed using the 

methods, benchmarks, and scoring procedures specified in the rating 

procedure. Candidates are grouped or rank-ordered based on the 

results.81 

Step 8—Refer candidates to the selecting official. The number of 

candidates referred depends on several factors, including eligibility, 

assessment results, the number of positions to be filled, and the 

appointing authority. Depending on the appointing authority and the 

terms used by the hiring agency, this list may be called a “referral list,” 

“best qualified list,” or a “certificate.” 

Step 9—Assess referred candidates. The selecting official may use 

formal or informal assessments, such as interviews and reference 

checks, to further reduce the number of candidates and make a 

selection. Typically, these assessments are relatively unstructured and 

are not scored. 

Step 10—Select a candidate (or candidates). The selecting official’s 

discretion in selection depends on the appointing authority and the 

procedures in use. In some cases, the selecting official may be able to 

select any referred candidate. (This is usually true in merit promotion.) 

In competitive examining, the selecting official may be able to select 

only from among the top three candidates—and in some cases, the 

selecting official’s choice is effectively limited to only a single 

candidate.82 
Appendix B—A Brief 

Discussion of Selection Tool 

Validity 

What is Validity? 

In the context of employee selection, “validity” typically refers to the 

relationship between performance on an assessment tool (e.g., the score 

on a written test) and a measure of job performance (e.g., the 

employee’s performance appraisal).83 Validity is expressed as a 

number between 1.0 and -1.0. A value of 1.0 means that there is a 

perfect positive relationship between the score received on the 

assessment tool and performance on the job. A value of 0 means that 

there is no relationship—in practical terms, that the assessment tool 

has no ability to predict job performance. A negative value indicates an 

inverse relationship: the better the performance on the assessment 

tool, the worse the expected on-the-job performance.84 

Validity measures are used to estimate how much of the variability in 

an employee’s performance can be predicted by the assessment tool. 

The estimate is calculated by squaring the validity measure, as 

illustrated below. 

Examples—Estimating the Predictive Value of an Assessment Tool85 

do on the job. 

Example 2—Reference Checks 

1. 

2. 0.26 x 0.26 = 0.0676, or approximately 6.8 percent. 

3. 

Example 1—General Mental Ability (GMA) Tests 

1. GMA tests have a validity of 0.51. 

2. 0.51 x 0.51 = 0.2601, or approximately 26 percent. 

3. GMA tests predict 26 percent of the variability in how well people will 

Reference checks have a validity of 0.26. 

Reference checks predict 6.8 percent of the variability in how well 

people will do on the job. 
No instrument has achieved a validity measure of 1.0. The best 

commonly used assessment tools in the Federal sector include the work 

sample tests (0.54), structured interviews (0.51), and general mental 

ability tests (0.51). Ratings of training and experience—the most 

common assessment—range from 0.11 to 0.45, depending on the rating 

method used. 

What is incremental validity? 

Incremental validity is the increase in validity resulting from the use of 

an assessment tool in addition to the tool(s) already in use. For 

example, as shown above, general mental ability tests have a validity of 

0.51. The combination of mental ability tests and structured interviews 

has a validity of 0.63.86 The gain in validity is 0.12. 

When initial validity is high, even small gains in validity have real 

value. In the example above, a mental ability test, alone, predicts 

approximately 26 percent of the variability in on-the-job performance. 

But the combination of that same test and a good structured interview 

predicts almost 40 percent of the variability in on-the-job performance. 

The converse also holds true: when initial validity is low, seemingly 

impressive increases mean little. An organization that increases an 

assessment tool’s validity from 0.05 to 0.30—a step in the right 

direction—nevertheless knows very little about how candidates 

assessed with that tool might perform. 

Research does not provide estimates for every possible combination of 

assessment tools. However, research does offer some guidelines for 

combining assessment tools: 

• Each individual assessment tool must be job-related and have 

some validity in its own right. Poor assessment tools do not 

become valuable by virtue of association with good assessment 

tools. For example, graphology (handwriting analysis) has a 

validity of 0.02—in practical terms, useless—and the validity of a 

GMA test and graphology combined is 0.51, the same as a GMA 

test alone.87 
• Incremental validity is greater when an assessment tool does not 

duplicate any assessment tool already in use. Restated, an 

assessment tool should not merely tell you what you already 

know. To the extent possible, different assessment tools should 

measure different job-related criteria. When used to complement 

good “baseline” assessment, even assessment tools that are not 

among the best—such as reference checks and integrity tests— 

can make a meaningful contribution to the selection process. 

We offer two cautions on this latter point. First, although research 

discourages “overlapping” assessments, the choice between collecting 

new information and verifying existing information is less clear in 

practice. Most managers have, at one time or another, reviewed an 

inflated resume or received a glowing recommendation for a mediocre 

candidate. When information and assessment tools are not perfectly 

reliable, there is a strong case—which we emphasize in this report—for 

using an assessment tool (such as reference checks) to confirm or 

disconfirm the results of a previous assessment (such as a rating of 

training and experience). Second, it is essential that all assessment 

tools be carefully designed and administered. Casual inquiries about an 

applicant’s work habits do not qualify as thorough “reference checks,” 

nor do snap judgments about an applicant’s honesty qualify as an 

“integrity test.” 
Appendix C—How 

Automated Hiring Systems 

Perform Assessments 

This appendix provides brief descriptions of the Federal Government’s 

two distinct approaches to using automation in the assessment process. 

We provide these descriptions to give the reader a basic understanding 

of how the systems assess applicants;88 we do not discuss matters such 

as the information technology infrastructure, operating expenses, 

licensing and contracting, data management, or ease of use. 

These descriptions are composites that represent “typical” or common 

practice; they are not highly detailed, nor do they reflect the policy or 

practice of any single system or agency. However, we do describe 

system-or agency-specific practices when a “generic” description is 

insufficient or nonexistent. Such descriptions are neither endorsements 

nor criticisms of the system or practice described. 

In this discussion, we frequently refer to “the system.” This usage is 

convenient and focuses attention on the properties and capabilities of 

the automated tools. However, the reader should keep in mind that 

“the system” is much more than a collection of computer hardware and 

software. Instead, it is a complex and constantly changing product of 

people, policies, hardware, and software. 

Application-Driven Systems 

Who uses this type of system? 

Major users are the National Aeronautics and Space Administration 

and the Department of Defense (including the Department of the Air 

Force, the Department of the Army, the Department of the Navy, and 

DOD components such as the Defense Logistics Agency). 
Who provides this type of system? 

The core component of this type of system is a suite of software 

components collectively known as Resumix. However, Federal 

agencies do not use Resumix by itself. Resumix is not “Federalized”— 

that is, it does not incorporate the procedures, rules, or formulas 

associated with the Federal Government’s competitive examining 

process. For this reason, Federal users “augment” Resumix with 

customized programs that apply decision rules (e.g., assign candidate 

scores) and produce staffing documents (e.g., referral lists). Thus, 

current application-driven systems are not a single product provided by 

a single source; instead, each system is a collection of interrelated 

programs, which may come from a variety of sources. 

What does an employment application include? 

An application typically includes: 

• A resume providing basic applicant information (such as name 

and telephone numbers) and describing the applicant’s work 

history (duties and accomplishments), education and training, and 

relevant awards and credentials. The agency may ask the 

applicant to describe specific skills (such as typing) or types of 

experience (such as contracting). The resume must adhere to 

agency formatting rules governing the placement and sequence of 

information, document length, typeface, and the like. 

• A profile or data sheet containing other types of information, such 

as supervisors’ names and contact information, veterans’ 

preference, and career status; and 

• A formal expression of interest in a specific job vacancy.89 

What do these systems do? 

These systems attempt to describe applicants’ competencies, 

credentials, and types of experience using a common vocabulary.90 To 

do this, Resumix uses two software components—the KnowledgeBase 

(a set of business rules, linguistic algorithms, and employment-related 

terms) and the AutoMatch Search function.91 The components enable 
the system user to “read” an application (resume),92 “extract” the 

candidate’s skills and experience, and produce a list of terms (“key 

words”) that characterize the applicant’s skills and experience. This list 

is frequently referred to as a “skills bucket.” The extraction process 

differs from manual KSA-based examining (evaluating knowledges, 

skills, and abilities) in some important ways: 

• First, it makes no level distinctions. For example, when Resumix 

extracts “proofreading” from a resume, it makes no judgment 

about the candidate’s level of proofreading experience or 

proficiency; and 

• Second, the process extracts key words without regard to their 

relevance to a particular job or occupation. (Put differently, 

Resumix does not require the user to specify desired competencies 

and credentials before it receives applications.) In contrast, 

question-driven systems (like their manual predecessors) require 

the user to determine the questions to be asked—and thus, the 

experience, education, and competencies to be measured—before 

receiving an application. 

How does the system assess minimum 

qualifications? 

Agencies may use the automated system to make a preliminary 

qualifications determination. (At the time of our study, some did; 

others did not.) To do this, the user identifies competencies essential to 

successful job performance, identifies a set (or sets) of key words that 

reflect those competencies, and (with computer assistance) “looks in” 

the skills bucket for the designated key words. However, agencies do 

not rely solely on the automated hiring system: they also conduct a 

manual review of minimum qualifications.93 Manual review is 

necessary for two reasons: accurate minimum qualifications 

determinations are critical to the integrity of the hiring process, and 

the structure of existing qualification standards makes it difficult to 

fully automate the minimum qualifications determination. Typical 

standards require a specific quantity of experience (usually 1 year) at a 

specific quality level, or a specific quantity and type of education (often 

in a particular curriculum), or both. Evaluating applicants against such 
standards requires a level of detail beyond that of a typical resume and 

an interpretive ability beyond the practical reach of existing systems. 

How does the system distinguish among 

qualified applicants? 

All agencies made the distinction by comparing candidates’ skills 

buckets. However, each agency we visited took a different approach to 

making the comparison. Below, we briefly describe the processes of 

three agencies. These descriptions are intended to illustrate each 

agency’s definition of “quality” and how it implements that definition. 

Agency 1 

At the time of our interview, Agency 1 used its automated system for 

merit promotion only. The agency’s merit promotion plan does not 

require ranking or numerical scores; all that is needed is to distinguish 

a group of best qualified candidates. The user: 

1. Identifies a set of desired competencies (typically no more than 

eight). 

2. Searches candidates’ skills buckets for the desired competencies. 

The system returns the number of competencies that each 

candidate possesses. Candidates possessing a higher number of 

competencies are considered better qualified. (In effect, each 

competency carries equal weight.) 

3. Refers the candidates with the most competencies, consistent with 

merit promotion plan provisions governing candidate referral. If 

the search yields too few (or no) candidates, the user may revise 

the search criteria to make them less restrictive. 

Agency 2 

Agency 2 uses its automated system for delegated examining.94 The 

user: 

1. Conducts a job analysis to identify major job requirements and 

job-related competencies. 
2. Develops a crediting plan (a plan for evaluating applicant 

qualifications). The crediting plan contains three quality levels: 

basic experience (experience that meets minimum qualifications), 

good experience, and outstanding experience. 

3. Establishes key-word-based criteria for each level. The result is a 

set of decision rules that will be used to assign each candidate to a 

quality level. The table below provides a simplified sample 

crediting plan. As shown, there is usually more than one way to 

meet any given level. 

4. Runs a program to apply the crediting plan (that is, search 

applicants’ skills buckets for the identified sets of key words) and 

assign each candidate to a quality level. 

Sample Crediting Plan—Agency 2 

In this example, the user has identified eight job-related key words (labeled 

“A” through “H” for simplicity). The table lists the three quality levels and 

the key word or combinations of key words needed to meet that quality level. 

Level Level Criteria (Required Key Words) 

70 (meets minimum 

qualifications) 

A or B 

80 (good experience) (B and C) or (C and D) or (D and E) 

90 (outstanding experience) (B and F) or (B and G) or (F and H) 

Ineligible Meets none of the above 

Note: Each candidate is assigned to the highest level met. In this example, 

a candidate whose skills bucket contains “F” and “H” will be assigned to the 

90 level, even without “A” or “B”. 

Agency 3 

Agency 3, like Agency 2, uses its automated system for delegated 

examining and sorts candidates into three quality groups. The 

examiner: 

1. Conducts a job analysis to identify major job requirements and 

job-related skills. The examiner then identifies a set of key words 
(typically 8 to 12) associated with those skills. The skills that are 

the most important to the job are designated “primary” skills. 

2. The user runs a program to search candidates’ skills buckets and 

apply a standard decision rule, shown below, to assign each 

candidate to a quality level. 

Decision Rule for Assigning Candidates to Quality Levels— 

Agency 3 

Level Key Words 

70 One or more 

80 50% of primary skills and 50% of all skills 

90 75% of primary skills and 65% of all skills 

Question-Driven Systems 

Who uses this type of system? 

Most civilian departments and agencies that use automated systems, 

with the exception of NASA and DOD, use a question-driven system. 

Who provides this type of system? 

Major providers of question-driven automated systems used by Federal 

agencies include (in alphabetical order) Avue Digital Systems, the U.S. 

Department of Commerce (Commerce Opportunities On-Line, referred 

to as COOL), Monster Government Solutions (QuickHire), and the 

U.S. Office of Personnel Management (USA Staffing). 

What does an employment application include? 

1. A multiple-choice questionnaire that asks applicants about their 

experience, education, and competencies. Agencies may also ask 

applicants to provide a narrative statement to support their 

response; and 

2. A written employment application that provides information 

(such as dates, academic credits and degrees, and duties and 

accomplishments) on relevant education and experience. 
Agencies also ask applicants for information such as address, level of 

education, veterans’ preference, geographic preference, and eligibility 

for special appointing authorities. This information may be collected 

through the questionnaire, the written application, or an on-line 

applicant profile (an electronic record, unique to each applicant, that 

contains personal information such as name, address, telephone 

number, and job interests). 

How does the system assess minimum 

qualifications? 

The most common approach is to assess minimum qualifications 

through experience- and education -based questions. These questions 

ask applicants whether they meet minimum education and experience 

requirements. An applicant who gives a negative response to any such 

question is eliminated from consideration. Agencies may also ask 

questions to determine whether a candidate has essential competencies. 

Most OPM qualification standards state that qualifying experience 

must provide the candidate with essential competencies, in addition to 

meeting a level of difficulty and complexity. The table below provides 

some examples of questions used to assess minimum qualifications. 
Sample Questions Used to Assess Minimum Qualifications 

The questions below appeared in vacancy announcements advertised on 

USAJOBS in February 2003. We have edited or reformatted some of the 

questions for readability. 

Question and Source Response Options 

Choose the one answer 

that best describes 

your education and 

experience. 

(From an 

announcement for a 

grade GS-7 wildlife 

biologist position.) 

1. I completed 4 years of college study leading to a 

bachelor’s degree in biological science 

including: 9 semester hours in wildlife subjects; 

12 semester hours in zoology; and 9 semester 

hours in botany or plant sciences. 

2. I have completed courses in an accredited 

college or university equivalent to a major in 

biological science (30 semester hours) with at 

least 9 semester hours in wildlife subjects, 12 

semester hours in zoology, and 9 semester 

hours in botany or related plant science, plus 

appropriate experience and/or additional 

education comparable to the completion of a 4-

year course of study. 

3. I am currently employed in this series in the 

federal government. 

4. I do not meet any of the requirements as 

described above. 

Choose the one answer 

that best describes 

your education and 

experience. 

(From an 

announcement for a 

grade GS-12 

administrative officer 

position.) 

1. I possess 1 year of specialized experience 

equivalent to the GS-11. Examples of 

experience may include: formulating & 

executing annual operating budgets; analyzing 

budget/fiscal reports; communicating with a 

variety of personnel at all levels in 

administrative management and support; 

working with staff in interpreting human 

resources policies and directives; reviewing and 

approving procurement activities; 

communicating and providing guidance to staff 

on issues. 

2. I do not meet any of the requirements as 

described above. 

Do you have 

knowledge of 

generally accepted 

accounting principles 

1. Yes. 

2. No. 
promulgated by the 

Financial Accounting 

(FASB) or the Federal 

Accounting Standards 

Advisory Board? 

announcement for a 

grade GS-11 

40 words per minute. 

announcement for a 

grade GS-6 office 

automation assistant 

position.) 

Yes/No—The applicant checks a box to respond 

(GAAP) and standards 

Standards Board 

(From an 

accountant position.) 

I certify that I can type 

(From an 

“Yes” or leaves it unchecked to respond “No.” 

However, as noted in the preceding discussion of application-based 

systems, the minimum qualifications determination is both informationand judgment-intensive. For this reason—and because accurate 

minimum qualifications determinations are important—agencies do not 

rely solely on the automated system to make final minimum 

qualifications determinations.95 If nothing else, agencies will 

manually verify a selectee’s minimum qualifications. 

How does the system assess relative 

qualifications? 

Question-based systems assess relative qualifications by asking 

applicants to assess themselves on one or more job-related 

competencies (e.g., oral communication) or tasks related to a 

competency (e.g., giving briefings on a technical subject to a nontechnical audience).96 The applicants rate their training, experience, or 

proficiency using the provided response scale. Responses typically 

cover a wide range of ability and experience, from no experience or 

training to a high level of experience or proficiency. The response scale 

may be generic or may be specific to the competency or task described 
in the question. The table below provides some sample questions and 

response options. 

Sample Questions Used to Assess Relative Qualifications 

These questions appeared in vacancy announcements advertised through 

USAJOBS in February 2003. We have edited or reformatted some 

questions and responses for readability. 

Question and Source Response Options 

Provide advice and 

assistance on staffing and 

recruitment. 

(From an announcement 

for a grade GS-11 human 

resources specialist 

position.) 

1. I have not had experience, education, or 

training in performing this task. 

2. I have completed formal education or 

training in performing this task, but have 

not yet performed this task on the job. 

3. I have performed this task on the job under 

close supervision by a supervisor, manager 

or senior employee to ensure compliance 

with correct procedures. 

4. I have performed this task as a regular part 

of a job, independently and usually without 

review by a supervisor, manager or senior 

employee. 

5. This task has been a central or major part of 

my work. I have performed it myself 

routinely, and I have trained others in 

performance of this task, and (or) others 

have consulted me as an expert for 

assistance in performing this task. 

Applied administrative 

procedures such as 

grammar, spelling, 

syntax, and other 

required formats 

sufficient to produce 

correspondence, reports, 

and briefings in an office 

setting. 

(From an announcement 

for a grade GS-6 office 

automation assistant 

position.) 

1. Prepared draft documents and routine 

correspondence. 

2. Prepared grammatical corrections of own 

work in preparation of final documents. 

3. Reviewed the work of others for 

grammatical, spelling, and punctuation 

accuracy. Proofread formal documents and 

composed material to be read by others. 

4. Experience, education, or training less than 

the levels described above. 

5. None of the above. 
The system scores candidates based on their responses, using a preestablished scoring key. The keys vary by job and by agency, but they 

all follow the same general pattern: candidates who claim higher levels 

of proficiency on a greater number of skills receive higher scores. Most 

agencies that use question-based systems use the full 70- to 100-point 

range when assigning scores, instead of the 70/80/90 pattern discussed 

earlier in the section on application-driven systems.97 
Appendix D—An Overview 

of “Multiple Hurdles” 

The Multiple Hurdles Concept98 

Even the best assessment tool merely predicts future performance; it 

cannot guarantee it. Consequently—although we wish it were 

otherwise—the hiring decision remains, at best, an educated guess. 

One strategy that can increase the probability of making a good 

selection is to use multiple assessment tools in succession. This is often 

called a “multiple hurdles” strategy, using an analogy from sports. In 

track and field, a runner who does not clear a hurdle is out of the race. 

Similarly, when multiple hurdles are used, applicants who perform 

poorly on an assessment are eliminated from further consideration for 

the job. Applicants who perform sufficiently well move on to another 

assessment. The process is repeated until a selection is made. (This 

strategy is also sometimes called “phased assessment.”) 

A recent study confirms that two assessments can be significantly 

better than one. The study examined the validity of mental ability 

tests, used alone, and the validity of such tests combined with a second 

assessment tool.99 The authors estimated that mental ability tests, 

alone, have a validity of 0.51. Adding a structured interview raised 

validity to 0.63; adding reference checks instead raised validity to 

0.57.100 Although these improvements may appear minor, they are 

not: they represent a substantial improvement in ability to predict 

actual job performance. (Appendix B, “A Brief Discussion of Selection 

Tool Validity,” discusses the concept of validity in more detail.) 

Using Multiple Hurdles 

Most managers already intuitively understand the rationale for 

multiple hurdles. For example, few managers wish to consider a 

candidate whose resume contains no applicable experience or education. 

But neither are most managers willing to hire, sight unseen, a 

candidate on the basis of excellent paper qualifications alone. 

Typically, hiring managers will (at a minimum) interview the most 

promising candidates to further narrow the pool. In other words, they 
use the written application as the first hurdle and the interview as the 

second hurdle. The challenge, then, is to use multiple hurdles 

effectively. Keys to effective use include ensuring that: 

• Each hurdle is carefully developed and administered, so that it 

produces usable, job-related information. It does little good—and 

may do substantial harm—to follow a valid assessment such as a 

work sample test with an invalid assessment such as graphology 

(handwriting analysis); 

• Each hurdle adds significantly to the depth, scope, or reliability of 

the organization’s knowledge of the candidates. An assessment 

tool should not duplicate (in mathematical terms, be highly 

correlated with) any previous assessment(s). Instead, any 

additional assessment should provide information on a different 

set of competencies, or provide a different perspective (and, 

therefore, perhaps a different result) on competencies that have 

previously been assessed; and 

• The value of the additional information outweighs its cost. 

How many hurdles are enough? Research confirms the value of a 

second assessment, but is silent on the value of a third (or fourth). We 

note that a cost-benefit analysis could well favor using a third 

assessment, such as reference checks or a work sample exercise, because 

the benefits can be substantial. Consider the value of avoiding a poor 

selection. A 2002 study conducted by the Employment Management 

Association reports that the average cost of hiring a professional 

employee is nearly $7,000. And this $7,000 is only a small portion of 

the actual costs of a poor selection, because it does not include items 

such as wasted salary dollars and the consequences of poor 

performance, which can be truly staggering. Viewed this way, a few 

hours devoted to conducting reference checks or reviewing work 

samples to reduce the likelihood of an expensive mistake is time well 

spent. 

So is more assessment always better? No. First, assessment must be 

timely as well as thorough, because good candidates will not wait 

forever for a hiring decision. Any agency that pursues the illusion of 

certainty in selecting employees will have a shallow and unpromising 
candidate pool indeed. Second, at some point, additional assessment 

can yield minimal or even negative returns. If a selecting official 

believes that he or she has a full and accurate understanding of 

candidates’ abilities, then further assessment will do little to inform the 

selection decision. It will also detract from the perceived fairness of the 

hiring process. As the Board found in its recent study of the Federal 

merit promotion program, continuing to assess candidates after a 

decision has been made merely contributes to applicant cynicism. 

In conclusion, there is no easy answer to the question of how many 

assessments to use. Research and experience suggest that agencies do 

well not to rely on a single assessment when they make selection 

decisions. However, assessment cannot go on indefinitely. Hiring 

decisions must be made, and they should be made in a timely manner 

and at reasonable cost. Ultimately, agencies and managers must use 

their judgment. 
Appendix E—An Argument 

for Competency-Based 

Qualification Standards 

In this report, we recommend that OPM replace experience- and 

education-based qualification standards (which we refer to as timebased qualification standards101) with competency-based qualification 

standards. We base this recommendation on two arguments. 

First, we believe that competency-based standards can provide a more 

sound basis for assessment and selection than time-based standards. 

The competency-based standards (“job profiles”) that OPM has 

tested102 were professionally developed, using research and input from 

agencies and subject matter experts. We are not aware of any 

comparable basis for existing time-based standards. 

Also, competency-based standards provide a reasonable balance 

between specificity and flexibility to accommodate differences among 

jobs, advances in knowledge and technology, and changing work and 

skills needs. In contrast, current time-based standards are flexible 

(responding to agency concerns about dated and inflexible standards), 

but have little specificity. Most time-based standards provide only 

general guidance, such as defining specialized experience as “experience 

that equipped the applicant with the particular knowledge, skills, and 

abilities to perform successfully the duties of the position, and that is 

typically in or related to the work of the position to be filled.” The 

standards are mostly silent on what those KSA’s might be. Recent 

OPM guidance, OPM oversight reviews, and the Board’s 2002 study of 

vacancy announcements all report that agencies have difficulty “filling 

in the blanks” in these standards. With considerable frequency, 

agencies either do too little—giving applicants little, if any, idea of 

whether they qualify—or do too much, establishing restrictive 

requirements that can deter or eliminate from consideration applicants 

who might actually be highly qualified.103 
Second, it appears that time-based qualification standards are becoming 

an impediment to efficient staffing operations, instead of an 

administrative convenience. In an era of paper-based staffing, 

examiners needed a consistent, quick-to-apply criterion for 

distinguishing (likely) qualified applicants from (likely) unqualified 

applicants. Time-based standards fulfilled this need. But time-based 

standards are not an administrative convenience when automated 

systems are used. If anything, they impede efficiency: it has proven 

rather difficult to automate minimum qualifications determinations 

grounded in time-based standards. Consequently, in many cases, it 

appears that agencies are devoting greater human effort to evaluating 

minimum qualifications than to evaluating relative qualifications. If so, 

more time is spent answering the question “Who could perform this 

job?” than is spent answering the question “Who should perform this 

job?” That would be unfortunate, because the latter question is far 

more important. 

Competency-based standards would make it possible to use a single 

criterion, competency, to assess both minimum and relative 

qualifications. This could reduce nonproductive work and help HR 

professionals and managers focus their attention on identifying the best 

qualified candidates—which is where their attention belongs. 
Appendix F—List of 

Recommendations Linked to 

Automation Success 

All of the recommendations in this report are aimed at helping agencies 

select high-performing employees. The table below links our 

recommendations to specific “success factors” needed to achieve that 

objective. 
Success Factor 

Effective Adequate Competent Line High Sound Appropriate Effective Hiring 

Leadership Resources HR Staff Mgmt. Quality Assess- Selection Assess- Process 

Support Applicant ment Criteria ment Integrity 

Pool Strategy Tools 

Recommendation 

Manage hiring X X 

as a critical 

business 

process. 

Manage the X X 

introduction of 

an automated 

hiring system 

as an 

organizational 

change 

initiative. 

Invest X 

appropriate 

resources in 

the design, 

implementation, and 

operation of 

the hiring 

process. 

Use automated X 

hiring systems 

to support 

recruitment 

programs, not 

to replace 

them. 

Design a X X X X 

hiring process 

that 

emphasizes 

selection 

quality over 

less important 

outcomes such 

as cost and 

efficiency. 

Evaluate X X X 

existing 

assessment 

tools before 

automating 

them. 

Invest in X X 

“content.” 

Communicate X X X X X X X 

roles and 

expectations 

to line 

managers, HR 

professionals, 

and applicants. 

Ensure the X X X X X X 

competence of 

HR 

professionals. 

Adopt X X 

documentation 

requirements 

that balance 

applicant 

burden and 

risk. 

Adopt a triage X X X 
strategy to 

manage 

volume. 

Adopt quality 

control 

measures. 

X X X 

Systematically 

evaluate and 

improve the 

hiring process. 

X X X X 

Establish a 

core document 

that applicants 

can use to 

apply for jobs 

in different 

agencies and 

Governmentwide 

“applicant 

service 

standards.” 

X 

Move toward 

competencybased 

qualification 

standards. 

X X X X 

1 Testimony of William H. Donaldson, Chairman of the Securities and 

Exchange Commission, before the Subcommittee on Commerce, Justice, 

State, and the Judiciary, Committee on Appropriations, United States 

House of Representatives, Mar. 31, 2004. 

2 Our use of the term “automation” is not meant to imply that the 

technology is inflexible or that it operates without human instruction 

and control. 

3 The Office of Personnel Management (OPM) has used automation 

for years to establish and maintain registers (inventories of candidates 

that were used to fill vacancies in multiple agencies and multiple 

locations). OPM also used computer programs for case examining. 

4 The General Accounting Office also has found that hiring 

automation is widespread in Federal agencies. In a May 2003 report, it 

reported that “Nineteen of the 24 agency HR directors we met with 

said they had automated or planned to automate at least a portion of 

their hiring processes.” United States General Accounting Office, 
“Human Capital: Opportunities to Improve Executive Agencies’ Hiring 

Processes,” Washington, DC, May 2003 (GAO-03-450), p. 23. 

5 Competitive examination is the “default” method of filling jobs in the 

competitive civil service. 

Elements of competitive examining include: public notice of the job, 

receipt of applications from 

the public, formal review of applicants’ qualifications, referral and 

selection in accordance with legal requirements, and (typically) a 

probationary period following appointment. When an agency conducts 

a competitive examination under a delegation of authority from OPM, 

competitive examining is usually called “delegated examining.” 

6 The Federal civil service comprises the competitive service, the 

excepted service, and the Senior Executive Service. The competitive 

service includes employees appointed through a competitive 

examination or an equivalent means. The excepted service includes (1) 

positions which generally are not filled through competitive examining 

and (2) employees who were appointed through noncompetitive means. 

7 Title 5, United States Code, Chapter 33, Subchapter I establishes 

general rules for competitive examining. OPM’s Delegated Examining 

Operations Handbook, available at www.opm.gov/deu, provides 

specific guidelines and procedures for agencies conducting competitive 

examinations. 

8 Robert M. Guion, “Assessment, Measurement, and Prediction for 

Personnel Decisions,” Lawrence 

Erlbaum Associates, Inc., Mahwah, NJ, 1998, p. 4. 

9 Much of this section is adapted from previous MSPB reports. See 

U.S. Merit Systems Protection Board, “The Federal Selection 

Interview: Unrealized Potential,” Washington, DC, February 2003, 

and U.S. Merit Systems Protection Board, “Assessing Federal Job 

Seekers in a Delegated Examining Environment,” Washington, DC, 

December 2001. 

10 Survey report from “The Human Capital Index: Linking Human 

Capital and Shareholder Value,” Watson Wyatt Worldwide, 1999, p. 5. 
11 Although it is simpler to terminate an employee during a 

probationary or trial period than it is to remove an employee for poor 

performance, termination still involves some costs. 

12 U.S. Merit Systems Protection Board, “Federal Supervisors and 

Poor Performers,” Washington, DC, July 1999, p. 15. 

13 For example, in a competitive examination, a veteran with a 

compensable service-connected disability rated at 10 percent or more 

who meets minimum qualification requirements must, for most 

positions, be placed at the top of the certificate (list of candidates). 

Federal agencies also are required to grant job preference to wellqualified Federal employees adversely affected by reductions in force 

(staff reductions). 

14 In a competitive examination, an agency may rank candidates (list 

them in order of numerical score) or place them in quality groups (a 

method called “category rating”). The ranking or grouping of 

candidates determines which candidates the hiring manager can select. 

If candidates are ranked, the manager must follow the “rule of three” 

and select from among the top three available candidates; if the 

candidates are placed in quality groups, the manager must select from 

the top group. (The manager must also observe veterans’ preference.) 

Our point is that accuracy is critical: the manager may not substitute 

his or her judgments for the “judgments” of the examining process. 

15 Validity coefficients range from 0 to 1. A coefficient of 0 would 

indicate that there is no relationship between the assessment and job 

performance. A coefficient of 1 would indicate that there is a perfect 

relationship—i.e., that higher scores on the assessment are always 

associated with better job performance. 

16 Source of validity coefficients: Frank L. Schmidt and John E. 

Hunter, “The Validity and Utility of Selection Methods in Personnel 

Psychology: Practical and Theoretical Implications of 85 Years of 

Research Findings,” Psychological Bulletin, The American 

Psychological Association, Inc., vol. 24, No. 2, September 1998, p. 265. 

17 MSPB’s 2000 Merit Principles Survey listed several sources of 

information about applicants and asked Federal supervisors to indicate 
the extent of their use of each source. Prior work experience was used 

to a “great extent” or “moderate extent” by 96 percent of respondents; 

level of education was used to a “great extent” or “moderate extent” by 

82 percent of respondents. 

18 We refer to these parties as “participants” instead of “stakeholders” 

because “stakeholders” is too broad and passive. The parties listed here 

do not merely have interests in the hiring process; they also have 

responsibilities. 

19 Here, “line managers” includes supervisors (i.e., first-level 

managers) and individuals they designate to carry out their roles (e.g., 

interview candidates) or represent their interests (e.g., participate in a 

job analysis). These individuals have direct knowledge of the job(s) 

being filled and are known as “subject matter experts.” 

20 Individual tasks may be performed outside the HR office, by line 

managers, administrative support staff, or contractors. Nevertheless, 

HR professionals usually remain wholly or partially accountable for the 

timeliness and quality of this work. 

21 Line managers may also initiate or perform many of these functions. 

22 These include regulations in 5 CFR Chapters 300, 302, 332, 333, 

335, 337, 338, and 339 and the Delegated Examining Operations 

Handbook. 

23 OPM's regulatory authority generally does not extend to 

employment under laws other than title 5, such as title 38 (covering 

Department of Veterans Affairs medical professionals) and title 10 

(covering selected defense and national security agencies and 

occupations). 

24 For example, if a qualification standard states that an applicant 

must have one year of specialized experience, an agency is effectively 

required to evaluate applicants’ training and experience. We note that 

OPM has eliminated the written test requirement from many 

qualification standards, making the use of a written test optional for 

those occupations unless it is otherwise required. 
25 Information about these and other OPM-led e-government 

initiatives is available at http://www.opm.gov/egov/. 

26 The term “black box” refers to an object or process whose internal 

logic and operations are unknown and difficult or impossible to discern. 

27 5 USC 2301(b)(1), the first merit system principle, states that 

“selection and advancement should be determined solely on the basis of 

relative ability, knowledge, and skills, after fair and open competition * 

* *.” 5 CFR 300.103, which governs competitive service employment 

practices, requires Federal agencies to base their employment practices 

(which include assessment criteria and methods) on job analysis and to 

establish a rational relationship between employment practices and job 

performance. 

28 Stephen W. Gilliland, “Effects of Procedural and Distributive 

Justice on Reactions to a Selection System,” Journal of Applied 

Psychology, vol. 79, No. 5, American Psychological Association, Inc., 

1994, p. 696. 

29 Excerpt from a letter from Colleen M. Kelley, National President 

of the National Treasury Employees Union to Steve Nelson, Director 

of Policy and Evaluation, U.S. Merit Systems Protection Board, dated 

Aug. 13, 2002. 

30 Nevertheless, managerial satisfaction does not constitute evidence 

of validity. It will not, for example, help an agency justify an 

employment practice that has adverse impact. 

31 USAJOBS (www.usajobs.opm.gov) is the Federal Government’s 

web-based job posting system. 

32 This list is illustrative, not exhaustive. 

33 Section 60-3, Uniform Guidelines on Employee Selection 

Procedures, 1978; 43 FR 38295, Aug. 25, 1978. We discuss Federal 

agency responsibilities, but the Uniform Guidelines also apply to 

private sector employers. 

34 Ibid., Sec. 16.9. 
35 Todd Raphael, “Cost per hire: don’t even bother”, Workforce, June 

2002, p. 112. 

36 We acknowledge that agencies may face situations, such as scarcity 

of qualified applicants or the need to fill a job immediately, where 

thorough pre-employment assessment is not practical. In such 

situations, post-employment assessment (including rigorous use of a 

probationary or trial period, if one applies) is particularly important. 

37 “James: building a new culture for federal work” (interview of Kay 

Coles James, Director, U.S. Office of Personnel Management), 

Government Computer News, vol. 21, No. 33, Post Newsweek Tech 

Media, Nov. 18, 2002, p. 112. 

38 See, for example, “From Red Tape to Results: Creating a 

Government that Works Better & Costs Less,” Washington, DC, Sep. 

7, 1993. This report of the National Performance Review 

recommended that agencies reduce the proportion of managers and 

supervisors in their workforce, and reduce the number of employees in 

“overhead” occupations such as human resources. 

39 For example, under paper-based staffing, a job application (which is 

typically received by mail) must be received, checked for timeliness and 

completeness, and filed. This task is not complicated, but the workload 

becomes substantial when multiplied by dozens or hundreds of 

vacancies and dozens or hundreds of applications per vacancy. 

40 The elimination of many technological barriers to automation has 

made it easier for small agencies to automate their hiring. However, 

there are still important differences between small and large agencies. 

Small agencies typically have fewer distinct jobs to fill in addition to 

smaller numbers of employees, making their business case for 

automating hiring different from that of larger agencies. Small 

agencies also typically have fewer resources to allocate to an 

automation initiative, but their smaller size also can simplify issues 

such as internal communications and access to subject matter experts. 

In the long run, the overall savings in staff time and the consistency 

gained in applicant assessment—outcomes also important to large 
agencies—seem to make small agencies’ initial investment in 

automation worthwhile. 

41 One system included in our study is not fully “federalized.” 

Agencies that use this system meet Federal hiring requirements by 

combining this system with other programs. 

42 Career transition assistance programs are intended to help surplus 

and displaced Federal employees find other Federal jobs. These 

programs are described at 5 CFR 330, subparts F and G. 

43 See U.S. Merit Systems Protection Board, “Assessing Federal Job 

Seekers in a Delegated Examining Environment,” Washington, DC, 

December 2001, and U.S. Merit Systems Protection Board, “Help 

Wanted: A Review of Federal Vacancy Announcements,” Washington, 

DC, April 2003. 

44 Cognitive ability is usually measured through objective tests. 

Because cognitive ability tests tend to be expensive to develop and may 

have a disparate impact on one or more classes of job applicants, only a 

few agencies routinely use them, and then only for a few occupations. 

However, agency experiences with these tests are very favorable. 

45 Question-based systems can also administer objective tests 

(including cognitive ability tests) although this is not often done. 

46 U.S. Merit Systems Protection Board, “Assessing Federal Job 

Seekers in a Delegated Examining Environment,” Washington, DC, 

December 2001, pp. viii, 28-29. 

47 Many agencies use non-T&E assessments such as interviews, 

reference checks, and writing samples when hiring employees. 

However, such assessments are rarely used as part of the formal rating 

process, and thus are used primarily to select among referred 

candidates; they rarely influence which candidates are referred to 

managers. 

48 We reiterate that automation does not dictate this approach, 

although the automated hiring systems used by Federal agencies are 

designed around evaluation of training and experience. OPM staff 
observed that automation and assessment methods can, and should be, 

considered separately. 

49 As of April 2004, the full text of the “Pledge to Applicants” was 

available at www.opm.gov/hrmc/2002/msg-087b.asp and in OPM’s 

Delegated Examining Operations Handbook. 

50 In most cases, it is not difficult to determine whether an applicant 

meets experience and education requirements. However, “borderline” 

cases—which may require close evaluation of job duties, time worked, 

and courses taken and credit hours earned—are not readily resolved 

using automated tools alone. 

51 We have no objective measure of applicants’ views of the various 

application processes that agencies have created to support their 

automated hiring systems. We have, however, observed many of these 

processes by reviewing vacancy announcements and applicant 

instructions. We have also received many comments from agency 

managers, HR professionals, and applicants. Those comments range 

from highly positive to highly negative. Clearly, not all automated 

application processes are created equal. 

52 One definition of change management, from Executive 

Transformations Consulting (www.changeready.com), is “a conscious, 

deliberative, collaborative effort to increase the performance 

of a human system.” Another, less formal characterization of change 

management is “an engineer’s approach to improving business 

performance and a psychologist’s approach to managing the human 

side of change.” See Jeff Hiatt and Tim Creasey, “The Definition and 

History of Change Management,” at www.prosci.com/tutorial-changemanagement-history.htm. 

53 Whatever process is chosen, agencies are expected to adhere to 

general professional standards for the development of employment 

practices. One mandatory step is job analysis. See, for example, 5 CFR 

300.103. 

54 Such problems are often caused by a deficient job analysis or by 

failure to act upon the results of a job analysis. 
55 The planned upgrade would divide a vacancy announcement into 

logical sections (e.g., job duties, job requirements, how to apply), 

instead of presenting the vacancy announcement as a single (and often 

very long) document, as is done now. 

56 Disclosure of desired attributes (i.e., selection criteria) may be 

explicit or implicit. The traditional and more common approach is 

explicit: listing the attribute (such as writing ability or a driver’s 

license) or asking a candidate a question directly related to the 

attribute. With implicit disclosure, an agency could describe 

representative job tasks—such as writing regulations and preparing 

Congressional testimony—and state that qualified candidates must be 

able to perform those tasks with minimal training and orientation. 

That would clearly indicate that writing ability is important. 

57 We note that any automated system that made error or abuse 

impossible would also be likely to prevent its users from modifying 

system inputs or outputs when appropriate. 

58 This is what the “rule of three” effectively requires agencies to do. 

59 We cannot quantify how automated hiring systems have affected 

timeliness: there are no broad, reliable measures of time to hire, and 

time to hire is affected by factors other than automation, such as 

vacancy announcement open periods, the agency decision-making 

process, and investigation and clearance requirements. 

60 Quote from a manager with the U.S. Geological Survey, shared in a 

meeting with MSPB staff during the research phase of our study. 

61 “Passive applicants” are individuals who are not actively seeking 

work. Often, they are individuals who are content with their current 

jobs and valued by their employer, yet willing to consider other 

opportunities, if those opportunities are appealing and can be pursued 

without too much initial effort. From the recruiter’s perspective, 

“passive applicants” are often excellent prospects—but not easy “sales.” 

62 This description of category rating is simplified for clarity. If the 

agency is hiring from outside the Federal service, it must also apply 

and observe veterans’ preference. 
63 For example, an automated system could assign higher scores to 

“balanced” candidates (that is, candidates with good ratings on all 

important competencies) than to “unbalanced” candidates (candidates 

with a mix of high and low ratings on important competencies), or vice 

versa. Or candidates could be rated “best qualified” if they have high 

scores in selected sets of competencies—recognizing that excellent 

candidates may have distinct strengths. We emphasize that these are 

illustrations, not suggestions. As always, selection criteria and related 

decision rules must be based on job analysis. 

64 One option—category rating—was not available to most agencies 

when we conducted our research for this report. 

65 Our interview with NASA predated Governmentwide availability of 

category rating. NASA combined two examining flexibilities (an 

alternative rating procedure and name requests of outstanding 

candidates) to create a process similar to category rating. The 1999 

edition of OPM’s Delegated Examining Handbook describes these 

flexibilities in detail. 

66 One possible exception is when all the candidates in the top 

category (the “A group”) are known to be highly qualified and to differ 

significantly only in individual accomplishments, strengths, and 

weaknesses, not in overall level of ability. We note that this situation 

will only occur if the initial assessment was extremely comprehensive 

and thorough—which is rarely practical or possible. Another exception 

is when there are very few qualified applicants. 

67 OPM’s Delegated Examining Handbook discusses the use of 

random numbers to break ties. We note that the Handbook also 

presents the option of using a job-related assessment to break ties. 

68 For example, one study that compared the ratings of participants 

in a simulated management task found striking differences in how 

participants’ self-ratings of performance compared with ratings 

provided by neutral, trained observers. Nearly one-third of the 

participants significantly overestimated their performance and nearly 

one-sixth significantly underestimated their performance. See Shaul 

Fox, Tamir Caspy, and Avner Reisler, “Variables affecting leniency, 
halo, and validity of self appraisal,” Journal of Occupational and 

Organizational Psychology, The British Psychological Society, 67, 

Leicester, UK, 1994, p. 53. 

69 Cultural factors may also create a tendency toward overstatement 

or understatement. Research on the role of culture in self-assessment 

accuracy is limited and inconclusive. 

70 We base this statement on an analysis of data from OPM’s Central 

Personnel Data File. For selected Cabinet-level agencies, we calculated 

a “servicing ratio” by dividing the number of permanent employees in 

non-HR occupations by the number of permanent employees in HR 

occupations. In most agencies, the servicing ratio changed little 

between 1998 and 2003. This analysis shows only that agencies’ 

employment of HR staff has not significantly decreased; it provides no 

insight into changes in operational efficiency or HR staff utilization. 

71 For example, staff may have taken on new responsibilities, such as 

human capital planning. Also, even comparatively minor departures 

from “business as usual”—such as improving the content and tone of 

vacancy announcements—can greatly reduce efficiency in the short 

term. 

72 “HR staff” refers to the people who design and administer HR 

programs, including HR specialists, assistants, and clerks. “HR 

professionals” refers to HR staff members who perform nonprocedural 

work. We use the term “professional” to indicate that these employees’ 

jobs require them to master an extensive body of specialized 

knowledge. 

73 These are distinct characteristics, but they are related. One agency 

observed that a reluctance to make decisions based on judgment 

(instead of clear-cut rules) was often the symptom of a deficiency in 

basic skills. For example, HR professionals who had difficulty 

analyzing and interpreting applicants’ work histories were often 

reluctant to rerate applicants when appropriate, or to accept 

responsibility for such rerating. 
74 This is not a criticism of all HR professionals. Many were fully 

prepared for these new demands; many have also revealed previously 

unrecognized and underused skills. 

75 The standards should, if applied properly, be content valid (that is, 

they should be rationally related to the duties and tasks of the job being 

filled). What is uncertain is the standards’ predictive validity; that is, 

their effectiveness at distinguishing applicants who can perform the job 

satisfactorily from applicants who cannot. 

76 “Grammar” refers to the terms and decision rules that an 

application-based automated hiring system uses to identify an 

applicant’s skills, experience, or other attributes. 

77 We emphasize that we are discussing when documentation should 

be required, not whether it should be required. As we have noted, 

documentation is necessary: falsification or misrepresentation of 

education and other credentials is a serious matter and is not 

infrequent. 

78 OPM has indicated that future competency-based standards will be 

modified based on agency feedback from the pilot tests. One possible 

modification is the inclusion of required competencies and required 

proficiency levels, which could make new standards more specific and 

easier to apply than the pilot standards. 

79 The full title of this document is “Delegated Examining Operations 

Handbook: A Guide for Federal Agency Examining Offices.” As of 

March 2003, this handbook could be obtained at 

http://www.opm.gov/deu/. 

80 This definition is much broader than the definition in OPM’s 

Delegated Examining Operations Handbook. OPM defines a rating 

procedure as the qualitative distinctions that result from the initial 

evaluation of applicants. 

81 This step corresponds to the application of quality ranking factors 

and the application of a rating procedure in OPM’s Delegated 

Examining Operations Handbook. We use the term “relative 

qualifications” to emphasize the fact that relative judgments are being 
made (as opposed to assessment against an absolute standard) and to 

emphasize that this step applies to every form of hiring, not just 

competitive examining. 

82 This situation occurs when the first-ranked candidate is entitled to 

veteran’s preference and the second- and third-ranked candidates are 

not eligible for such preference. The selecting official may reject the 

first candidate, but may not “pass over” that candidate to select one of 

the lower-ranked candidates unless he or she can successfully “object” 

to the first candidate. To do so, the selecting official must demonstrate 

that the candidate is unqualified or unsuitable for the position. 

“Unsuitable” does not mean “less than ideal for the position”; it means 

unfit by virtue of past misdeeds or some other factor. 

83 Wayne F. Cascio, “Applied Psychology in Human Resource 

Management,” fifth ed., Prentice-Hall, Inc., Upper Saddle River, NJ, 

1998, p. 103. 

84 Although assessments with a high negative validity coefficient 

could conceivably be quite useful, few—if any—are used. 

85 Source of validity coefficients: Frank L. Schmidt and John E. 

Hunter, “The Validity and Utility of Selection Methods in Personnel 

Psychology: Practical and Theoretical Implications of 85 Years of 

Research Findings,” Psychological Bulletin, The American 

Psychological Association, Inc., vol. 24, No. 2, September 1998, p. 265. 

86 Ibid. The validity of two assessment tools used together cannot be 

calculated based on the individual tools’ “stand alone” validity; the 

calculation must be based on actual experiments or complex statistical 

analysis. 

87 Ibid. 

88 Our discussion of “assessment” does not include eligibility—matters 

such as United States citizenship, civil service status, meeting age 

requirements, or time-in-grade requirements. 

89 Increasingly, Federal agencies allow any interested person to create 

and store a profile or resume on-line. However, storing a profile or 
resume is usually not sufficient for a person to be considered for 

employment. Generally, agencies do not regard a person as an 

“applicant” until he or she formally expresses interest in a particular 

job. That expression can take a number of forms, such as a cover letter 

containing a vacancy announcement number and job title or a “self 

nomination” (an electronic message containing an announcement 

number and a personal identifier). 

90 Competencies and types of experience are not always distinct. For 

example, the term “accounting” could indicate that a candidate has the 

ability to perform such work (i.e., the competency) or that a candidate 

has performed accounting work (the experience). However, other 

terms that Resumix might identify—such as “foreign governments”— 

can only be understood as a description of an applicant's background or 

work experience. 

91 This software is proprietary. Therefore, we do not know and 

cannot describe precisely how it works. 

92 Technology of this type, commonly called “natural language 

processing,” is much more ambitious and sophisticated than the “word 

search” function in common office software applications. The idea is to 

combine vocabulary and algorithms to provide interpretive power 

comparable to that of a human reader. Readers wishing a fuller 

description of this approach may contact Resumix through its web site 

at www.resumix.yahoo.com. 

93 For internal candidates, it may be possible to eliminate or greatly 

reduce the need for manual verification. For example, most current 

employees can provide occupational series and grade information (a 

commonly used “proxy” measure of type and quality of experience), 

which can be readily “read” and “assessed” by an automated system. 

Agencies may also evaluate education and credentials by proxy. For 

example, an agency filling a physician vacancy might assume that the 

professional credentials of a Federally-employed physician have already 

been reviewed and verified by the current employing agency. However, 

these “proxies” have their limitations, and are not available for most 

external candidates. 
94 Agency 2 and Agency 3, below, also use their systems for merit 

promotion. Here, we discuss only delegated examining, which requires 

finer distinctions among candidates. 

95 The extent of manual review depends on the importance of the 

qualifications determination. Agencies are most likely to review 

determinations that materially affect the integrity or outcome of the 

hiring process, such as those for candidates referred for selection 

consideration and (under delegated examining) candidates eligible for 

veterans’ preference. 

96 A question may be used for assessing both minimum and relative 

qualifications, but agencies generally use separate sets of questions for 

these purposes. 

97 Category rating is not mentioned here because category rating was 

not an option for most agencies at the time we interviewed them. 

98 The material presented in this appendix originally appeared in the 

September 2002 and July 2003 issues of Issues of Merit, the newsletter 

of MSPB’s Office of Policy and Evaluation. 

99 Frank L. Schmidt and John E. Hunter, op. cit., p. 265. 

100 Ibid. The authors used general mental ability (GMA) tests as the 

primary assessment because of their high validity and their applicability 

to a wide range of jobs. The authors did not publish estimates for any 

combination of assessments that did not include GMA tests. 

101 The term “time-based” reflects the fact that most OPM 

qualification standards define qualifying experience or education in 

terms of time. The time element may be explicit (such as “3 years of 

general experience”) or implicit (such as a baccalaureate degree, which 

is typically attained following 4 years of full-time college attendance). 

102 OPM has developed and tested two competency-based 

qualification standards (“job profiles”) for two occupations: accountant 

and information technology specialist. 
103 In a memorandum to the Human Resources Management Council 

(#MSG-022a, dated Mar. 25, 2002), OPM Director Kay Coles James 

states that “An increasing number of job announcements describe 

qualifications too narrowly. Although they must be job-related, 

qualifications should not be so agency-specific that only applicants who 

have worked for your agency qualify for consideration.” See also U.S. 

Merit Systems Protection Board, “Help Wanted: A Study of Federal 

Vacancy Announcements,” Government Printing Office, Washington, 

DC, April 2003, p. 24. 
U.S. Merit Systems Protection Board PRSRT STD 

1615 M Street NW U.S. POSTAGE PAID 

Washington, DC 20419 WASHINGTON DC 

PERMIT NO. G-113 

OFFICIAL BUSINESS 

PENALTY FOR PRIVATE USE $300 

Identifying Talent through Technology


Automated Hiring Systems in Federal Agencies