**DOE JOBS ONLINE**

**Competency Question Development**

**U. S. Department of Energy**

![](media/image1.wmf)

**Robert London**

**Human Resources Consulting**

**April 2001**

**[DOE JOBS ONLINE COMPETENCY QUESTION DEVELOPMENT
CONTENTS]{.underline}**

**[PART I (Page 1)]{.underline}**

-   What can we say about any automated rating and ranking system?

```{=html}
<!-- -->
```
-   What can we say about how people feel about automated rating and
    ranking systems?

```{=html}
<!-- -->
```
-   What are some of the perceptions people have regarding automated
    rating and ranking systems?

-   Let's look at these perceptions from some different points-of-view
    and see what some people have said and might say:

-   What can we say about the HR specialist's role in the implementation
    of an automated rating and ranking system?

**[PART II (Page 6)]{.underline}**

-   Why MUST we evaluate applicants?

-   Why SHOULD we evaluate applicants?

-   What is the key to this evaluation?

-   What do we have to do?

-   Why do we have to document the positive relationship (linkage)
    between the duties and tasks performed and the knowledge, skills,
    and abilities or competencies needed to perform the tasks?

-   What are some of the technical standards for content validity?

-   When should we systematically study the job?

-   When do we usually systematically study the job?

-   Whom do we have to involve?

-   What are our goals?

**[PART III (Page 12)]{.underline}**

-   What is a DOE Jobs ONLINE competency question?

-   You know you've got a good DOE Jobs ONLINE competency question if
    it's:

-   You know you've got a good set of DOE Jobs ONLINE competency
    questions if it's:

-   You know you've got a great set of DOE Jobs ONLINE competency
    question if it's:

-   What widely used applicant assessment method is most like a good set
    of competency questions?

-   What widely used applicant assessment method is useful in following
    up on responses to competency questions?

-   How many questions do you need to ask?

-   What about some of the internal DOE guidance on writing competency
    questions?

##### 

##### 

##### PART IV (Page 18)

-   How do you decide what you want to measure (12-step long version)?

##### PART V (Page 32)

-   How do you decide what you want to measure (10-step short version)?

##### PART VI (Page 40)

-   How do you decide what you want to measure (7-step Readers Digest
    condensed version)?

**[PART VII (Page 46)]{.underline}**

-   How about an overly simplified hypothetical demonstration that
    compares and contrasts manual and DOE Jobs ONLINE rating and ranking
    procedures?

**[PART VIII (Page 53)]{.underline}**

-   How about a strategy for working with selecting officials to develop
    DOE Jobs ONLINE competency questions?

# 

# **[PART IX (Page 56)]{.underline}**

-   What's the challenge for HR Specialists?

-   What's the challenge for selecting officials?

-   How do we get there?

-   What are those "easy ways out" we want to avoid?

-   How about some final thoughts on all of this?

-   What's the bottom line?

**[DOE JOBS ONLINE COMPETENCY QUESTION DEVELOPMENT - PART
I]{.underline}**

**[What can we say about any automated rating and ranking
system]{.underline}?**

-   There are advantages in every automated rating and ranking system

-   There are disadvantages in every automated rating and ranking system

**[What can we say about how people feel about automated rating and
ranking systems]{.underline}?**

-   Some managers like automated rating and ranking systems

-   Some managers don't like automated rating and ranking systems

```{=html}
<!-- -->
```
-   Some managers don't care

```{=html}
<!-- -->
```
-   Some applicants like automated rating and ranking systems

-   Some applicants don't like automated rating and ranking systems

```{=html}
<!-- -->
```
-   Some applicants don't care

```{=html}
<!-- -->
```
-   Some HR specialists like automated rating and ranking systems

-   Some HR specialists don't like automated rating and ranking systems

```{=html}
<!-- -->
```
-   Some HR specialists don't care

```{=html}
<!-- -->
```
-   Some people perceive that there is only "good" in an automated
    rating and ranking system

-   Some people perceive that there is only "not so good" in an
    automated rating and ranking system

```{=html}
<!-- -->
```
-   Some people perceive there is nothing "good" or "not so good" in an
    automated rating and ranking system

```{=html}
<!-- -->
```
-   Some people perceive an automated rating and ranking system as a
    solution to all of their problems

```{=html}
<!-- -->
```
-   Some people perceive an automated rating and ranking system as a
    bigger problem than the problems it's supposed to solve

-   Some people don't perceive an automated rating and ranking system as
    either a problem or a solution

**[What are some of the perceptions people have regarding automated
rating and ranking systems]{.underline}?**

-   They're fast

```{=html}
<!-- -->
```
-   They're completely objective

```{=html}
<!-- -->
```
-   They eliminate paper

-   They eliminate the need for rating panels

```{=html}
<!-- -->
```
-   They allow for a considerable amount of standardization

```{=html}
<!-- -->
```
-   They allow for a greater range of numerical ratings

```{=html}
<!-- -->
```
-   They don't allow for very much to be changed after the announcement
    has been issued

**[Let's look at these perceptions from some different points-of-view
and see what some people have said and might say]{.underline}:**

-   **They're fast**

```{=html}
<!-- -->
```
-   You mean to tell me that once the system is set up and applicants
    start using it, the system assigns numerical ratings instantaneously
    and I can get my certificates almost as soon as the announcement
    closes?

```{=html}
<!-- -->
```
-   You mean to tell me HR specialists are still going to have to
    perform a manual review of qualifications to ensure that applicants
    being referred at least meet the minimum qualification requirements
    for the announcement? Why? How much time is that going to take? You
    said I'd get my certificates the day after the announcement closed.
    Now you're saying I won't? I'm going to talk to Claudia about this.

```{=html}
<!-- -->
```
-   You mean to tell me that in order to take advantage of the system's
    speed, I'm going to have to spend a considerable amount of my
    valuable time working with my HR specialist just to get my
    announcement up?

```{=html}
<!-- -->
```
-   My AO takes care of all that. Talk to her.

```{=html}
<!-- -->
```
-   I don't care. As long as I get the person I want to hire quicker,
    I'm in favor of it.

```{=html}
<!-- -->
```
-   **They're completely objective**

```{=html}
<!-- -->
```
-   You mean to tell me I won't have to read through pages and pages of
    an applicant addressing KSAs just to try and find the one place
    where they say the one thing that gave them the points they got? And
    I'll automatically get the best-qualified ones, too? Where do I sign
    up?

```{=html}
<!-- -->
```
-   I like to read through a complete application and decide for myself
    whether or not applicants should get credit based on what they say.
    I know I sometimes have to interpret what they write, but that gives
    me a better feel for their qualifications. Besides, ability to
    communicate in writing is a very important part of these jobs and I
    feel people who can write well are better qualified than people who
    can't write well.

```{=html}
<!-- -->
```
-   But we're telling applicants exactly how they'll be rated. That
    compromises the whole process.

```{=html}
<!-- -->
```
-   I don't care how objective or subjective anything is. As long as I
    get some real good people to choose from, I'm in favor of it.

```{=html}
<!-- -->
```
-   I don't care. As long as I get the person I want to hire, I'm in
    favor of it.

```{=html}
<!-- -->
```
-   **They eliminate paper**

```{=html}
<!-- -->
```
-   You mean to tell me I can look at their resumes on-line? That's
    great!

```{=html}
<!-- -->
```
-   But I'm still going to have to get a DD-214, proof of 10-point
    veterans' preference, transcripts, etc.

```{=html}
<!-- -->
```
-   But I'm still going to have to set up and maintain a paper case file
    for everything?

```{=html}
<!-- -->
```
-   I'm not going to bother with this apply online stuff. I'm just going
    to send my CV and publications list over to that program.

```{=html}
<!-- -->
```
-   How will we know if they're lying or not? At least with supplemental
    statements we could see through the BS, and I don't mean Bachelor of
    Science.

```{=html}
<!-- -->
```
-   I don't care. As long as I get the person I want to hire, I'm in
    favor of it.

```{=html}
<!-- -->
```
-   **They eliminate the need for rating panels**

```{=html}
<!-- -->
```
-   Boy! Will I ever be glad when I don't have to set up rating panels
    any more! Do you know how hard it is to get two or three big wigs
    together at the same time to do this? They're always on travel and I
    have to make copies of everything for them.

```{=html}
<!-- -->
```
-   They should have just done away with rating panels altogether and
    let HR specialists do the rating themselves. Just doing that would
    have saved a lot of time.

```{=html}
<!-- -->
```
-   They should have just done away with rating panels altogether and
    let managers do the rating themselves. Just doing that would have
    saved a lot of time.

```{=html}
<!-- -->
```
-   What do you mean do away with rating panels? Subject-matter-experts
    can rate applicants better than any machine. They know the field,
    they know the job, and they can see right through any smoke screens.

```{=html}
<!-- -->
```
-   What do you mean do away with rating panels? Rating panels are the
    only part of the system that keeps managers in line.

```{=html}
<!-- -->
```
-   Do away with rating panels? Great! They were always just an
    extension of management and did whatever the supervisor wanted them
    to do.

```{=html}
<!-- -->
```
-   I don't care. As long as I get the person I want to hire, I'm in
    favor of it.

```{=html}
<!-- -->
```
-   **They allow for a considerable amount of standardization**

```{=html}
<!-- -->
```
-   Is this one of those cookie-cutter things?

```{=html}
<!-- -->
```
-   All of my positions are unique and this won't work with them.

```{=html}
<!-- -->
```
-   We hire scientists and no machine can evaluate a scientist.

```{=html}
<!-- -->
```
-   Why should I waste my time developing a whole series of "Yes/No" and
    "Choose the response that best describes your experience" questions.
    The kind of people I'm looking for are not going to waste their time
    with that.

```{=html}
<!-- -->
```
-   Choose the response that best describes your experience, my eye. You
    know every applicant is going to overrate himself or herself. What's
    that going to get us?

```{=html}
<!-- -->
```
-   How will we know if they're lying or not?

```{=html}
<!-- -->
```
-   I don't care. As long as I get the person I want to hire, I'm in
    favor of it.

```{=html}
<!-- -->
```
-   **They allow for a greater range of numerical ratings**

```{=html}
<!-- -->
```
-   You mean to tell me there's going to be as many as 2,998 possible
    ratings? 3,997 counting veterans' preference? Boy, that's pretty
    impressive.

```{=html}
<!-- -->
```
-   What do you mean I can't hire the person I want because he missed
    out by one one-hundredth of a point? I'm going to call Claudia about
    this garbage.

```{=html}
<!-- -->
```
-   I don't care. As long as I get the person I want to hire, I'm in
    favor of it.

```{=html}
<!-- -->
```
-   **They don't allow for very much to be changed after the
    announcement has been issued**

```{=html}
<!-- -->
```
-   We should never ever under any circumstances change anything in the
    crediting plan after the announcement has been issued. If we do,
    someone can come back and get us for playing with the system to
    ensure a name request was selectable. We'd be better off if we just
    cancel this announcement, tweak it a little bit, and then
    re-announce the vacancy.

```{=html}
<!-- -->
```
-   You know that crediting plan we're using for this announcement?
    Well, everyone's coming out with the same score. There are some
    folks who really look to be better qualified but our benchmarks
    aren't giving them credit. How do you feel about taking a look at
    the benchmarks and seeing if we can come up with some additional
    examples of experience that might help break up the logjam?

```{=html}
<!-- -->
```
-   I can go back in, change the weights we assigned to some of the
    responses, have the system re-score everyone, and see what happens.
    But I can't go back in and change anything else.

```{=html}
<!-- -->
```
-   I don't care. As long as I get the person I want to hire, I'm in
    favor of it.

**[What can we say about the HR specialist's role in the implementation
of an automated rating and ranking system]{.underline}?**

-   HR specialists need to be aware of the system's advantages and
    disadvantages

```{=html}
<!-- -->
```
-   HR specialists need to be aware of customers' perceptions

**[DOE JOBS ONLINE COMPETENCY QUESTION DEVELOPMENT - PART
II]{.underline}**

**[Why MUST we evaluate applicants]{.underline}?**

-   To be in compliance with applicable laws, rules, and regulations

```{=html}
<!-- -->
```
-   Merit System Principles (5 U.S.C. 2301)

```{=html}
<!-- -->
```
-   Prohibited Personnel Practices (5 U.S.C. 2302)

```{=html}
<!-- -->
```
-   Uniform Guidelines on Employee Selection Procedures (29 U.S.C. 1607)

```{=html}
<!-- -->
```
-   Veterans\' Preference Act of 1944, as amended (5 U.S.C. 2108)

**[Why SHOULD we evaluate applicants]{.underline}?**

-   So that the best qualified applicants are referred for selection
    consideration

**[What is the key to this evaluation]{.underline}?**

-   A well-constructed, valid, job analysis-based assessment tool that
    measures

-   The important work behaviors required for successful performance

-   The tasks and duties associated with them

```{=html}
<!-- -->
```
-   Their relative importance

**[What do we have to do]{.underline}?**

-   Systematically study the job to identify

```{=html}
<!-- -->
```
-   The basic duties and tasks that the employee will perform

```{=html}
<!-- -->
```
-   The relative importance of the duties and tasks that the employee
    will perform

```{=html}
<!-- -->
```
-   The level of responsibility the employee will have in performing the
    duties and tasks

-   The knowledges, skills, and abilities or competencies the employee
    will need to perform the duties and tasks

```{=html}
<!-- -->
```
-   Document the positive relationship (linkage) between the duties and
    tasks performed and the knowledge, skills, and abilities or
    competencies needed to perform the tasks

```{=html}
<!-- -->
```
-   What did we do (measure)?

```{=html}
<!-- -->
```
-   Why did we do (measure) it?

**[Why do we have to document the positive relationship (linkage)
between the duties and tasks performed and the knowledge, skills, and
abilities or competencies needed to perform the tasks]{.underline}?**

-   To be in compliance with the Uniform Guidelines on Employee
    Selection Procedures

```{=html}
<!-- -->
```
-   Content validity

```{=html}
<!-- -->
```
-   Degree to which past accomplishments and achievements are
    representative of important aspects of on-the-job performance and
    significantly correlated with successful performance

```{=html}
<!-- -->
```
-   Essential documentation

```{=html}
<!-- -->
```
-   Description of the method used to analyze the job

```{=html}
<!-- -->
```
-   Work behavior(s), associated tasks, and, if behavior results in a
    work product, complete description of work products

```{=html}
<!-- -->
```
-   Measures of criticality and/or importance of work behavior(s) and
    method of determining these measures

```{=html}
<!-- -->
```
-   Where job analysis also identified the knowledges, skills, and
    abilities used in work behavior(s), an operational definition for
    each knowledge in terms of a body of learned information and for
    each skill and ability in terms of observable behaviors and
    outcomes, and the relationship between each knowledge, skill, or
    ability and each work behavior, as well as the method used to
    determine this relationship

```{=html}
<!-- -->
```
-   To demonstrate compliance with Merit Principle No. 1

```{=html}
<!-- -->
```
-   Recruitment should be from qualified individuals from appropriate
    sources in an endeavor to achieve a work force from all segments of
    society, and selection and advancement should be determined solely
    on the basis of relative ability, knowledge and skills, after fair
    and open competition which assures that all receive equal
    opportunity.

**[What are some of the technical standards for content
validity]{.underline}?**

-   **Appropriateness of content validity**

```{=html}
<!-- -->
```
-   A selection procedure can be supported by a content validity
    strategy to the extent that it is a representative sample of the
    content of the job. Selection procedures which purport to measure
    knowledges, skills, or abilities may in certain circumstances be
    justified by content validity, although they may not be
    representative samples, if the knowledge, skill, or ability measured
    by the selection procedure can be operationally defined (see
    "Standards for demonstrating content validity" section below), and
    if that knowledge, skill, or ability is a necessary prerequisite to
    successful job performance.

```{=html}
<!-- -->
```
-   A selection procedure based upon inferences about mental processes
    cannot be supported solely or primarily on the basis of content
    validity. Thus, a content strategy is not appropriate for
    demonstrating the validity of selection procedures which purport to
    measure traits or constructs, such as intelligence, aptitude,
    personality, commonsense, judgment, leadership, and spatial ability.
    Content validity is also not an appropriate strategy when the
    selection procedure involves knowledges, skills, or abilities which
    an employee will be expected to learn on the job.

```{=html}
<!-- -->
```
-   **Job analysis for content validity**

```{=html}
<!-- -->
```
-   There should be a job analysis which includes an analysis of the
    important work behavior(s) required for successful performance and
    their relative importance and, if the behavior results in work
    product(s), an analysis of the work product(s). Any job analysis
    should focus on the work behavior(s) and the tasks associated with
    them. If work behavior(s) are not observable, the job analysis
    should identify and analyze those aspects of the behavior(s) that
    can be observed and the observed work products. The work behavior(s)
    selected for measurement should be critical work behavior(s) and/or
    important work behavior(s) constituting most of the job.

```{=html}
<!-- -->
```
-   **Development of selection procedures**

```{=html}
<!-- -->
```
-   A selection procedure designed to measure the work behavior may be
    developed specifically from the job and job analysis in question, or
    may have been previously developed by the user, or by other users.

```{=html}
<!-- -->
```
-   **Standards for demonstrating content validity**

```{=html}
<!-- -->
```
-   To demonstrate the content validity of a selection procedure, a user
    should show that the behavior(s) demonstrated in the selection
    procedure are a representative sample of the behavior(s) of the job
    in question or that the selection procedure provides a
    representative sample of the work product of the job. In the case of
    a selection procedure measuring a knowledge, skill, or ability, the
    knowledge, skill, or ability being measured should be operationally
    defined. In the case of a selection procedure measuring a knowledge,
    the knowledge being measured should be operationally defined as that
    body of learned information which is used in and is a necessary
    prerequisite for observable aspects of work behavior of the job. In
    the case of skills or abilities, the skill or ability being measured
    should be operationally defined in terms of observable aspects of
    work behavior of the job. For any selection procedure measuring a
    knowledge, skill, or ability the user should show that (a) the
    selection procedure measures and is a representative sample of that
    knowledge, skill, or ability; and (b) that knowledge, skill, or
    ability is used in and is a necessary prerequisite to performance of
    critical or important work behavior(s). In addition, to be content
    valid, a selection procedure measuring a skill or ability should
    either closely approximate an observable work behavior, or its
    product should closely approximate an observable work product. The
    closer the content and the context of the selection procedure are to
    work samples or work behaviors, the stronger is the basis for
    showing content validity. As the content of the selection procedure
    less resembles a work behavior, the less likely the selection
    procedure is to be content valid, and the greater the need for other
    evidence of validity.

```{=html}
<!-- -->
```
-   **Reliability**

```{=html}
<!-- -->
```
-   The reliability of selection procedures justified on the basis of
    content validity should be a matter of concern to the user. Whenever
    it is feasible, appropriate statistical estimates should be made of
    the reliability of the selection procedure.

```{=html}
<!-- -->
```
-   **Prior training or experience**

```{=html}
<!-- -->
```
-   A requirement for or evaluation of specific prior training or
    experience based on content validity, including a specification of
    level or amount of training or experience, should be justified on
    the basis of the relationship between the content of the training or
    experience and the content of the job for which the training or
    experience is to be required or evaluated. The critical
    consideration is the resemblance between the specific behaviors,
    products, knowledges, skills, or abilities in the experience or
    training and the specific behaviors, products, knowledges, skills,
    or abilities required on the job, whether or not there is close
    resemblance between the experience or training as a whole and the
    job as a whole.

```{=html}
<!-- -->
```
-   **Content validity of training success**

```{=html}
<!-- -->
```
-   Where a measure of success in a training program is used as a
    selection procedure and the content of a training program is
    justified on the basis of content validity, the use should be
    justified on the relationship between the content of the training
    program and the content of the job.

```{=html}
<!-- -->
```
-   **Operational use**

```{=html}
<!-- -->
```
-   A selection procedure which is supported on the basis of content
    validity may be used for a job if it represents a critical work
    behavior (i.e., a behavior which is necessary for performance of the
    job) or work behaviors which constitute most of the important parts
    of the job.

```{=html}
<!-- -->
```
-   **Ranking based on content validity**

```{=html}
<!-- -->
```
-   If a user can show, by a job analysis or otherwise, that a higher
    score on a content valid selection procedure is likely to result in
    better job performance, the results may be used to rank persons who
    score above minimum levels. Where a selection procedure supported
    solely or primarily by content validity is used to rank job
    candidates, the selection procedure should measure those aspects of
    performance which differentiate among levels of job performance.

**[When should we systematically study the job]{.underline}?**

-   Ideally, as the first step of basic position description development

**[When do we usually systematically study the job]{.underline}?**

-   Realistically, most of the time, it's not done after the position
    description has been developed and submitted to the HR specialist
    for classification and announcing

**[Whom do we have to involve]{.underline}?**

-   Person(s) with bona fide expert knowledge about what the job does
    and what it takes to do the job (i.e., supervisor, someone who
    previously held the job, someone who currently holds the job, etc.)

**[What are our goals]{.underline}?**

-   To fairly measure applicants' qualifications against important job
    related criteria

```{=html}
<!-- -->
```
-   To determine who are the best qualified applicants relative to
    important job related criteria

-   To determine who are the best qualified applicants relative to each
    other

```{=html}
<!-- -->
```
-   To determine who are the best qualified applicants without regard to
    any non-merit factor

```{=html}
<!-- -->
```
-   To make selections from among the best qualified applicants

**[DOE JOBS ONLINE COMPETENCY QUESTION DEVELOPMENT - PART
III]{.underline}**

**[What is a DOE Jobs ONLINE competency question]{.underline}?**

-   Task-based question you ask the applicant which is designed to
    measure

```{=html}
<!-- -->
```
-   Whether or not the applicant has education, training, and/or
    experience in performing a specific task or duty (typically Yes/No,
    True/False, Multiple Choice, etc.)

```{=html}
<!-- -->
```
-   From the response to this type of question, we may logically presume
    whether or not the applicant has the requisite knowledge required to
    perform the specific task or duty (i.e., if a certain knowledge is
    required to perform a certain task and an applicant indicates he/she
    has experience performing that task, we may logically presume the
    applicant also has that certain knowledge)

```{=html}
<!-- -->
```
-   From the response to this type of question, we cannot presume
    whether or not the applicant can perform the specific task or duty
    at a particular level of expertise unless we build that concept into
    the question

```{=html}
<!-- -->
```
-   Whether or not the applicant has education, training, and/or
    experience in performing a specific task or duty at a particular
    level (typically Applicant Assessment, but can also be built in to
    Yes/No, Multiple Choice, etc.)

-   From the response to this type of question, we may presume not only
    whether or not the applicant has the requisite knowledge required to
    perform the specific task or duty, but we may also presume whether
    or not the applicant can perform the specific task or duty well

```{=html}
<!-- -->
```
-   A competency question is very similar in concept to the benchmarks
    used in traditional crediting plans

```{=html}
<!-- -->
```
-   Applicants are asked to address the knowledges, skills, and
    abilities listed in the announcement as ranking factors

```{=html}
<!-- -->
```
-   Applicants' responses are compared against pre-determined examples
    of education, training, and/or experience performing tasks or duties
    that provide a basis for assigning points.

**[You know you've got a good DOE Jobs ONLINE competency question if
it's]{.underline}:**

-   Behaviorally consistent (i.e., it asks about past experience,
    accomplishments, and achievements which are representative of
    important aspects of on-the-job performance)

```{=html}
<!-- -->
```
-   Distinguishing (i.e., it differentiates between minimally qualified
    and highly qualified applicants solely on the basis of relative
    job-related qualifications)

```{=html}
<!-- -->
```
-   Good sounding (i.e., it sounds right when you ask it and answer it
    out loud)

```{=html}
<!-- -->
```
-   Important (i.e., it asks about a task or duty important for
    successful job performance)

```{=html}
<!-- -->
```
-   Job-related (i.e., it asks about a task or duty that's going to be
    performed on-the-job)

```{=html}
<!-- -->
```
-   Objective (i.e., it's asked in such a way as to eliminate, or
    minimize to the greatest extent possible, subjective interpretation
    on the part of the rater, who happens to be the applicant)

```{=html}
<!-- -->
```
-   Practical (i.e., it asks about something that is appropriate for the
    occupation/grade, labor market, and expected applicant pool)

```{=html}
<!-- -->
```
-   Purposeful (i.e., it asks for some piece of information which will
    be useful)

```{=html}
<!-- -->
```
-   Ratable (i.e., it asks about something that can be measured)

```{=html}
<!-- -->
```
-   Reasonable (i.e., it asks about something that's reasonable to
    expect applicants to have relative to both the job and the applicant
    population)

```{=html}
<!-- -->
```
-   Reliable in the short term (i.e., it's asked in such a way that
    applicants with similar experience will respond similarly)

```{=html}
<!-- -->
```
-   Simplistic (i.e., it asks about a single well-defined task, duty,
    responsibility, or concept, rather than a broad pattern of
    knowledges, abilities, skills, behaviors, and other characteristics)

```{=html}
<!-- -->
```
-   Valid in the short term (i.e., it's significantly and positively
    correlated with the performance of important job-related tasks and
    duties identified through the job analysis)

```{=html}
<!-- -->
```
-   Verifiable (i.e., it produces a response that can be verified at
    some point in the process through resume reviews, interviews,
    reference checks, submission of additional information, etc.)

**[You know you've got a good set of DOE Jobs ONLINE competency
questions if it's]{.underline}:**

-   All of the above

**[You know you've got a great set of DOE Jobs ONLINE competency
question if it's]{.underline}:**

-   Reliable in the long term (i.e., it's consistently able, over a
    significant period of time, to differentiate between minimally
    qualified applicants and highly qualified applicants)

```{=html}
<!-- -->
```
-   Valid in the long term (i.e., it's consistently able, over a
    significant period of time, to predict successful performance on the
    job)

**[What widely used applicant assessment method is most like a good set
of competency questions]{.underline}?**

-   Structured interview

```{=html}
<!-- -->
```
-   Questions are consistently used for all interviews for a particular
    job

```{=html}
<!-- -->
```
-   Questions are based on job-related knowledges, skills, abilities,
    competencies, and/or factors critical to successful job performance

-   Questions focus on duties and functions which are "unique" to a
    specific position

**[What widely used applicant assessment method is useful in following
up on responses to competency questions]{.underline}?**

-   Structured interview

```{=html}
<!-- -->
```
-   For "Yes/No" questions, applicants should be asked to elaborate on
    "Yes" responses, give some examples, provide some additional
    information, etc., which will allow the interviewer to assess
    whether or not their "Yes" response is accurate

```{=html}
<!-- -->
```
-   For "Applicant Assessment" questions, applicants should be asked to
    elaborate on their responses (except "I have no education.,
    experience,..." responses), give some examples, provide some
    additional information, etc., which will allow the interviewer to
    assess whether or not they have accurately assessed their level of
    education and/or experience

**[How many questions do you need to ask]{.underline}?**

-   However many it takes to determine who are the best qualified
    applicants relative to the important job related criteria and
    relative to each other

```{=html}
<!-- -->
```
-   No optimum number of questions

```{=html}
<!-- -->
```
-   Number of questions will vary from job to job

```{=html}
<!-- -->
```
-   Too few questions may produce a limited number of ratings which may
    not (or may) result in any meaningful distinctions between
    applicants

```{=html}
<!-- -->
```
-   Too many questions may produce a greater number of ratings which may
    not (or may) contribute substantially to the accuracy of the
    prediction of how well a person will perform

```{=html}
<!-- -->
```
-   Too many questions may (or may not) deter some highly qualified
    applicants from applying

**[What about some of the internal DOE guidance on writing competency
questions]{.underline}?**

-   Doe Jobs ONLINE User Guide (11/20/2000), Pages 79-80

```{=html}
<!-- -->
```
-   Explain in clear terms what you mean by terms commonly used in the
    position description. For example, "contracting for construction
    projects..." could mean a number of things

-   Clear terminology would include verbs like "negotiate," "develop,"
    "evaluate," "write," etc., to characterize the actual task being
    performed

```{=html}
<!-- -->
```
-   When writing questions, consider exactly what the applicant will do
    and the capability needed in those areas

-   Remember who your audience is **(your audience is your rater)**

```{=html}
<!-- -->
```
-   Focus on dimensions of the applicant's accomplishments that make him
    or her qualified for position

```{=html}
<!-- -->
```
-   "Knowledge of position classification, as indicated by..."

-   "Administers a procurement program"

```{=html}
<!-- -->
```
-   Begin statements with imperative verbs

# Instead of using "managing a program...," say "manage a program..."

-   Instead of using "assuring...," say "assure..."

-   Instead of using "maintaining...," say "maintain..."

```{=html}
<!-- -->
```
-   There should be consistent verb tense/structure throughout the
    document

```{=html}
<!-- -->
```
-   Present tense is preferred

```{=html}
<!-- -->
```
-   Use **complete** sentences as often as possible

-   Begin sentences with a command

-   "Select the answer that best describes you knowledge of..."

-   Don't use broad descriptive statements, such as "administered
    large-scale contracts"

-   Define what is meant by including details to avoid interpretations

-   Avoid using qualifying descriptive adjectives, such as
    "outstanding," "thorough," demonstrated," etc.

-   Avoid using acronyms

```{=html}
<!-- -->
```
-   External applicants may not understand their meaning

-   **Internal applicants may not understand their meaning**

```{=html}
<!-- -->
```
-   State questions without using the phrase "for example"

```{=html}
<!-- -->
```
-   A well-written question or statement should stand alone **(i.e., on
    its own merits)**

```{=html}
<!-- -->
```
-   Use caution in structuring long answer questions as the answers are
    not rated

```{=html}
<!-- -->
```
-   Long answer questions are needed ONLY if its important to gather
    information about something that you would not expect to appear in
    someone's resume

```{=html}
<!-- -->
```
-   Watch the length of questions

```{=html}
<!-- -->
```
-   There is a 500 character (including blank spaces) limitation

```{=html}
<!-- -->
```
-   This equates to about 7 ½ lines on a sheet of 8 ½ x 11 paper with 1"
    margins on each side.

```{=html}
<!-- -->
```
-   Applicant assessment questions should be worded in the first person

```{=html}
<!-- -->
```
-   "Evaluate and prepare estimates and schedules for design,
    procurement, and construction projects"

```{=html}
<!-- -->
```
-   Bee veri carful to spel chick your dokument befor loding it into the
    sistem

```{=html}
<!-- -->
```
-   Mispelings and gramatical erors will shows up on vakancy
    pronouncements thruout the sistem if not corected befor thay r
    loded.

**[DOE JOBS ONLINE COMPETENCY QUESTION DEVELOPMENT - PART
IV]{.underline}**

**[How do you decide what you want to measure (long
version)]{.underline}?**

## **STEP 1** - Identify and list all of the basic tasks or duties (i.e., analyze the job)

-   **STEP 2** - Group the basic tasks and duties identified in STEP 1
    into major duties or functions

```{=html}
<!-- -->
```
-   **STEP 3** - Develop a knowledge, skill, or ability statement for
    each major duty or function

```{=html}
<!-- -->
```
-   **STEP 4** - Review everything from the first three steps for
    completeness and understanding

```{=html}
<!-- -->
```
-   **STEP 5** - Take a break; you've earned it

```{=html}
<!-- -->
```
-   **STEP 6** - Decide which of the major duties or functions
    identified in STEP 2 are essential are which are desirable

-   **STEP 7** - Qualify the desirable major duties or functions
    remaining from STEP 6

```{=html}
<!-- -->
```
-   **STEP 8** - Determine the relative importance of each remaining
    major duty or function from STEP 7

```{=html}
<!-- -->
```
-   **STEP 9** - Qualify and determine the relative importance of the
    basic tasks and duties associated with the desirable major duties or
    functions remaining from STEP 8

```{=html}
<!-- -->
```
-   **STEP 10** - Decide what kind of information you're looking for

```{=html}
<!-- -->
```
-   **STEP 11** - Select existing DOE Jobs ONLINE competency questions

```{=html}
<!-- -->
```
-   **STEP 12** - Develop new DOE Jobs ONLINE competency questions

# 

-   **STEP 1 - Identify and list all of the basic tasks or duties (i.e.,
    analyze the job)**

```{=html}
<!-- -->
```
-   Position description is the primary source of information, the most
    important document in the process, and more often than not, the only
    document used in the process

```{=html}
<!-- -->
```
-   If position description accurately and adequately describes all of
    the basic tasks or duties, it should be the only document you need
    in the process

```{=html}
<!-- -->
```
-   If position description accurately and adequately describes all of
    the basic tasks or duties, the validity of the end result is more
    assured

```{=html}
<!-- -->
```
-   If position description does not accurately and adequately describe
    the basic tasks or duties, the validity of the end result is
    questionable

```{=html}
<!-- -->
```
-   Identifying and listing all of the basic tasks or duties has major
    benefits

```{=html}
<!-- -->
```
-   It's the first step toward meeting the documentation requirements

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clear
    understanding of every little thing the position does

```{=html}
<!-- -->
```
-   It may reveal some tasks and duties that are not really performed

-   It may reveal some tasks and duties that are really performed are
    missing

```{=html}
<!-- -->
```
-   Other potentially useful sources of information include
    qualification standard, classification standard, existing
    performance plan, and occupational literature

```{=html}
<!-- -->
```
-   Reference where each listed task or duty comes from

```{=html}
<!-- -->
```
-   Especially important if more than one source of information is used

```{=html}
<!-- -->
```
-   Not necessary if all tasks and duties come from position description

```{=html}
<!-- -->
```
-   **STEP 2 - Group the basic tasks and duties identified in STEP 1
    into major duties or functions**

```{=html}
<!-- -->
```
-   Major duties or functions are comprised of basic tasks, duties, and
    responsibilities that have a common denominator

```{=html}
<!-- -->
```
-   Grouping basic tasks and duties into major duties or functions has
    major benefits

# It's the second step toward meeting the documentation requirements

-   It will provide both you and the selecting official with a clear
    understanding of major functional areas for which the position is
    responsible

```{=html}
<!-- -->
```
-   It will make the original tasks and duties lists appear more
    manageable

```{=html}
<!-- -->
```
-   It may reveal some major functions are missing

-   It may reveal some major functions are really unimportant

```{=html}
<!-- -->
```
-   **STEP 3 - Develop a knowledge, skill, or ability statement for each
    major duty or function identified in STEP 2**

```{=html}
<!-- -->
```
-   Each KSA statement should embody one relatively simple, readily
    identifiable characteristic

```{=html}
<!-- -->
```
-   Each KSA must begin with "Knowledge of," "Skill in," or "Ability to"

```{=html}
<!-- -->
```
-   "Knowledge" is a body of information that is applied directly to the
    performance of a function

```{=html}
<!-- -->
```
-   "Skill" is a present observable competence to perform a learned
    psychomotor act

```{=html}
<!-- -->
```
-   "Ability" is a present competence to perform an observable behavior
    or a behavior that results in an observable product

```{=html}
<!-- -->
```
-   Use descriptive phrases (i.e., operational definitions) that
    indicate how KSAs will be used

```{=html}
<!-- -->
```
-   Especially important for knowledges (i.e., "Knowledge of X
    sufficient to perform Y" or "Knowledge of X in order to perform Y")
    since a knowledge, by itself, is not necessarily task-oriented

```{=html}
<!-- -->
```
-   Not as important for skills and abilities since they should already
    be task-oriented

```{=html}
<!-- -->
```
-   Don't begin any KSA with "Must have..."

-   Don't use qualifiers that are inexact or subject to different
    interpretations (i.e., thorough knowledge of, basic knowledge of,
    familiarity with, demonstrated ability to, etc.)

-   Don't mix "Knowledge," "Skill," and/or "Ability" in the same KSA

```{=html}
<!-- -->
```
-   The very rare and infamous KSA "Hat Trick"

```{=html}
<!-- -->
```
-   "KNOWLEDGE of and ABILITY to interpret and apply legislative,
    executive, regulatory, policy and procedural requirements that
    affect Federal mail and reproduction management programs, as
    evidenced by SKILL in independently directing the management and
    administration of mail and duplicating service operations." (Support
    Services Supervisor, GS-0342-12)

```{=html}
<!-- -->
```
-   Developing knowledge, skill, or ability statements has major
    benefits

```{=html}
<!-- -->
```
-   It's the third step toward meeting the documentation requirements

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clear
    understanding of what the position needs in terms of knowledges,
    skills, and abilities

```{=html}
<!-- -->
```
-   It may reveal some KSAs that are really needed are missing

```{=html}
<!-- -->
```
-   It may reveal some KSAs that are not really needed

```{=html}
<!-- -->
```
-   It may help you later on in developing your questions by providing
    you with a known reference point

```{=html}
<!-- -->
```
-   **STEP 4 - Review everything from the first three steps for
    completeness and understanding**

```{=html}
<!-- -->
```
-   At this point in the process, both you and the selecting official
    should be able to clearly articulate either one of the following:

```{=html}
<!-- -->
```
-   The perfect person for this job should have (insert appropriate KSA
    statement), so that he/she can perform the basic tasks and duties
    associated with (insert appropriate major task, duty, or function)

-   The perfect person for this job should be able to perform the basic
    tasks and duties associated with (insert appropriate major task,
    duty, or function), the performance of which requires (insert
    appropriate KSA statement)

```{=html}
<!-- -->
```
-   Clearly articulating either one of the above has a major benefit

```{=html}
<!-- -->
```
-   It's the fourth step toward meeting the documentation requirements

```{=html}
<!-- -->
```
-   It will ensure that both you and the selecting official have the
    same understanding

```{=html}
<!-- -->
```
-   **STEP 5 - Take a break; you've earned it**

# Taking a break has major benefits

-   It will break the tension

-   It will allow both you and the selecting official to breathe a sigh
    of relief

-   It will allow both you and the selecting official to "take a step
    back" and think about what you've done from a more detached
    perspective

-   It will allow both you and the selecting official to "psych"
    yourselves up for the remaining steps

```{=html}
<!-- -->
```
-   **STEP 6 - Decide which major duties or functions identified in STEP
    2 are essential and which are desirable**

```{=html}
<!-- -->
```
-   For each major duty or function identified in STEP 2, you and the
    selecting official need to answer the following:

```{=html}
<!-- -->
```
-   Is it **ESSENTIAL** for an applicant to have had education,
    training, and/or experience in this specific major duty or function
    prior to entering on duty in order to perform in a satisfactory way
    within a reasonable period of time (i.e., will an applicant who has
    not had any education, training, and/or experience in this specific
    major duty or function prior to entering on duty be ineligible
    regardless of any and all other education, training, and/or
    experience the applicant has)?

```{=html}
<!-- -->
```
-   If it is absolutely essential for an applicant to have had
    education, training, and/or experience in a specific major duty or
    function prior to entering on duty, it must be incorporated into the
    minimum qualification statement on the announcement

-   If it is not absolutely essential for an applicant to have had
    education, training, and/or experience in a specific major duty or
    function prior to entering on duty, it cannot be incorporated into
    the minimum qualification statement on the announcement

```{=html}
<!-- -->
```
-   Is it **DESIRABLE** for an applicant to have had education,
    training, and/or experience in this specific major duty or function
    prior to entering on duty in order to perform in a satisfactory way
    within a reasonable period of time (i.e., will an applicant who has
    not had any education, training, and/or experience in this specific
    major duty or function prior to entering on duty still be eligible)?

```{=html}
<!-- -->
```
-   If it is merely desirable for an applicant to have had education,
    training, and/or experience in a specific major duty or function
    prior to entering on duty, it cannot be incorporated into the
    minimum qualification statement on the announcement

```{=html}
<!-- -->
```
-   If it is neither absolutely essential nor merely desirable for an
    applicant to have had education, training, and/or experience in a
    specific major duty or function prior to entering on duty, there is
    no reason to evaluate it

```{=html}
<!-- -->
```
-   SPECIAL NOTE: If it is absolutely **ESSENTIAL** for an applicant to
    have had education, training, and/or experience in this specific
    major duty or function prior to entering on duty in order to perform
    in a satisfactory way within a reasonable period of time **AND** the
    basic tasks and duties associated with this specific major duty or
    function can be individually evaluated, it can also be considered
    desirable

```{=html}
<!-- -->
```
-   Deciding which major duties or functions identified in STEP 2 are
    essential and which are desirable has major benefits

```{=html}
<!-- -->
```
-   It's the fifth step toward meeting the documentation requirements

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clearer
    understanding of what kinds of education, training, and/or
    experience

-   Applicants for the position absolutely must have prior to entering
    on duty

-   Applicants for the position need not necessarily have prior to
    entering on duty

```{=html}
<!-- -->
```
-   It may result in a decision to eliminate some major duties or
    functions, and the basic tasks and duties associated with them, from
    the evaluation process

```{=html}
<!-- -->
```
-   It will get both you and the selecting official started on thinking
    about the types of questions you're going to ask

```{=html}
<!-- -->
```
-   **STEP 7 - Qualify the desirable major duties or functions remaining
    from STEP 6**

```{=html}
<!-- -->
```
-   For each desirable major duty or function remaining from STEP 6, you
    and the selecting official need to answer the following:

```{=html}
<!-- -->
```
-   Is it **REASONABLE** to expect applicants to have had education,
    training, and/or experience in this specific major duty or function
    prior to entering on duty, taking into account the applicant pool,
    labor market, grade level of the job, results of previous
    announcements, etc.?

```{=html}
<!-- -->
```
-   If it is not reasonable to expect applicants to have had education,
    training, and/or experience in a specific major duty or function
    prior to entering on duty, taking into account applicant pool, labor
    market, grade level of the job, results of previous announcements,
    etc., there is no reason to ask a question about it

```{=html}
<!-- -->
```
-   Is it **RATABLE** using DOE Jobs ONLINE (i.e., can I ask applicants
    a question or questions about this specific major duty or function
    that will give me useful information about that applicant)?

```{=html}
<!-- -->
```
-   If you can't ask applicants a question or questions about a specific
    major duty or function that will give you useful information using
    DOE Jobs ONLINE (or any other assessment tool for that matter),
    there is no reason to ask a question about it

```{=html}
<!-- -->
```
-   Is it **VERIFIABLE** at some point in the selection process (i.e.,
    can I assure myself through a resume review, interview, reference
    check, etc., that an applicant has responded to the question or
    questions on this specific major duty or function accurately)?

```{=html}
<!-- -->
```
-   If you can't verify applicants' responses at some point in the
    selection process using the instruments available, there is no
    reason to ask a question about it

```{=html}
<!-- -->
```
-   Does it **DISTINGUISH** between a minimally qualified applicant and
    a highly qualified applicant (i.e., would an applicant who has had
    education, training, and/or experience in this specific major duty
    or function prior to entering on duty be better qualified than one
    who has not)?

```{=html}
<!-- -->
```
-   If an applicant who has had education, training, and/or experience
    in a major duty or function prior to entering on duty would not
    necessarily be better qualified than an applicant who has not, there
    is no reason to ask a question about it

```{=html}
<!-- -->
```
-   Qualifying the desirable major duties or functions has major
    benefits

```{=html}
<!-- -->
```
-   It's the sixth step toward meeting the documentation requirements

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clearer
    understanding of what kinds of education, training, and/or
    experience

-   Will be used to differentiate between minimally qualified and highly
    qualified applicants

```{=html}
<!-- -->
```
-   It may result in a decision to eliminate some more major duties or
    functions from the evaluation process

```{=html}
<!-- -->
```
-   It will keep both you and the selecting official thinking about the
    types of questions you're going to ask

```{=html}
<!-- -->
```
-   **STEP 8 - Determine the relative importance of each remaining major
    duty or function from STEP 7**

```{=html}
<!-- -->
```
-   No hard and fast rule for doing this

```{=html}
<!-- -->
```
-   Relative importance may be determined based on

```{=html}
<!-- -->
```
-   Consequences of the major duty or function not being performed
    satisfactorily

```{=html}
<!-- -->
```
-   Benefits/payoff of the major duty or function being performed
    satisfactorily

```{=html}
<!-- -->
```
-   Frequency with which the major duty or function is performed
    absolutely (i.e., daily, weekly, monthly, semi-annually, annually,
    etc.)

```{=html}
<!-- -->
```
-   Frequency with which the major duty or function is performed
    compared to the other functions (i.e., more than, less than, same
    as, etc.)

```{=html}
<!-- -->
```
-   Grade level equivalent of the major duty or function compared to the
    grade level of the position

```{=html}
<!-- -->
```
-   Grade level equivalent of the major duty or function compared to the
    other major duties or functions

```{=html}
<!-- -->
```
-   Grade controlling tasks or duties should always have relatively high
    importance

```{=html}
<!-- -->
```
-   Assign a relative value to each major duty or function which
    corresponds to its importance (i.e.: 3 = Very Important, 2 =
    Important, 1 = Less Important; or VI = Very Important, I =
    Important, LI = Less Important)

```{=html}
<!-- -->
```
-   Determining the relative importance of each remaining major duty or
    function has major benefits

```{=html}
<!-- -->
```
-   It's the seventh step toward meeting the documentation requirements

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a beginning
    basis for weighting DOE Jobs ONLINE responses

```{=html}
<!-- -->
```
-   It may reveal some more previously identified major duties or
    functions are relatively unimportant

```{=html}
<!-- -->
```
-   It may result in a decision to eliminate relatively unimportant
    major duties or functions from the evaluation process

```{=html}
<!-- -->
```
-   **STEP 9 - Qualify and determine the relative importance of the
    basic tasks and duties associated with the desirable major duties or
    functions remaining from STEP 8**

```{=html}
<!-- -->
```
-   For each of the basic tasks and duties associated with the desirable
    major duties and functions remaining from STEP 8, you and the
    selecting official need to

```{=html}
<!-- -->
```
-   Qualify the task or duty in the same manner you qualified the
    desirable major duties or functions in STEP 7 (i.e., desirable,
    reasonable, ratable, etc.)

```{=html}
<!-- -->
```
-   Determine the relative importance of each basic task or duty in the
    same manner you determined the relative importance of the desirable
    major duties and functions in STEP 8 (i.e., consequences,
    benefits/payoff, frequency, etc.)

```{=html}
<!-- -->
```
-   Assign a relative value to each basic task or duty which corresponds
    to its importance (i.e.: 3 = Very Important, 2 = Important, 1 = Less
    Important; or VI = Very Important, I = Important, LI = Less
    Important; VI = Very Important, I = Important, NI = Not As
    Important)

```{=html}
<!-- -->
```
-   Qualifying and determining the relative importance of the basic
    tasks and duties

```{=html}
<!-- -->
```
-   It's the eighth step toward meeting the documentation requirements

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with what should
    be the final list of tasks and duties which will form the basis for
    your DOE Jobs ONLINE questions

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a further
    basis for weighting DOE Jobs ONLINE responses

```{=html}
<!-- -->
```
-   It may reveal some basic tasks or duties are relatively unimportant

```{=html}
<!-- -->
```
-   It may result in a decision to eliminate relatively unimportant
    basic tasks or duties from the evaluation process

```{=html}
<!-- -->
```
-   **STEP 10 - Decide what kind of information you're looking for**

```{=html}
<!-- -->
```
-   For each of the basic tasks and duties remaining from STEP 9, you
    and the selecting official need to decide what type of information
    you're looking for

```{=html}
<!-- -->
```
-   Absolute (i.e., has or has not performed the task or duty, no more
    and no less)

```{=html}
<!-- -->
```
-   Qualified absolute (i.e., has or has not performed the task or duty
    at a single specific level, no more and no less)

```{=html}
<!-- -->
```
-   Qualitative (i.e., progressively higher levels of expertise in
    performing the task or duty)

```{=html}
<!-- -->
```
-   Quantitative (i.e., has or has not performed a number of different
    tasks or duties)

```{=html}
<!-- -->
```
-   **STEP 11 - Select DOE Jobs ONLINE competency questions**

```{=html}
<!-- -->
```
-   For each of the basic tasks and duties remaining from STEP 9, you
    and the selecting official need to decide

```{=html}
<!-- -->
```
-   If there is an existing question in the DOE Jobs ONLINE KSOC
    database that

-   Addresses the specific task or duty

-   Provides the type of response you're looking for

```{=html}
<!-- -->
```
-   Weights the responses appropriately or allows you to change the
    weights appropriately

```{=html}
<!-- -->
```
-   Meets the characteristics of a good competency question (i.e.,
    behaviorally consistent, distinguishing, good sounding, etc.)

```{=html}
<!-- -->
```
-   **STEP 12 - Develop new DOE Jobs ONLINE competency questions**

```{=html}
<!-- -->
```
-   For each of the basic tasks and duties remaining from STEP 9 that
    you and the selecting official have decided that there is no
    appropriate question in the DOE Jobs ONLINE KSOC database, you need
    to

```{=html}
<!-- -->
```
-   Develop a new question to be added to the DOE Jobs ONLINE KSOC
    database that

```{=html}
<!-- -->
```
-   Addresses the specific task or duty

-   Provides the type of response you're looking for

```{=html}
<!-- -->
```
-   Weights the responses appropriately

```{=html}
<!-- -->
```
-   Meets the characteristics of a good competency question (i.e.,
    behaviorally consistent, distinguishing, good sounding, etc.)

```{=html}
<!-- -->
```
-   By the end of this step

# 

-   You will have a well documented content valid procedure for
    evaluating applicants for a job based on how closely their
    background of education, training, and experience matches valid,
    measurable, reasonable, and important job-related criteria

**[DOE JOBS ONLINE COMPETENCY QUESTION DEVELOPMENT - PART
V]{.underline}**

**[How do you decide what you want to measure (short
version)]{.underline}?**

## **STEP 1** - Identify and list all of the basic tasks or duties (i.e., analyze the job)

-   **STEP 2** - Think about how you would group the basic tasks and
    duties identified in STEP 1 into major duties or functions and which
    basic tasks or duties you would put in your groups

```{=html}
<!-- -->
```
-   **STEP 3** - Think about a knowledge, skill, or ability statement
    for each major duty or function

```{=html}
<!-- -->
```
-   **STEP 4** - Gather your thoughts from the first three steps for
    completeness and understanding

```{=html}
<!-- -->
```
-   **STEP 5** - Decide which of the basic tasks or duties identified in
    STEP 1 are essential and which are desirable

-   **STEP 6** - Qualify the desirable tasks or duties remaining from
    STEP 5

```{=html}
<!-- -->
```
-   **STEP 7** - Determine the relative importance of each remaining
    task or duty from STEP 6

```{=html}
<!-- -->
```
-   **STEP 8** - Decide what type of information you're looking for

```{=html}
<!-- -->
```
-   **STEP 9** - Select existing DOE Jobs ONLINE competency questions

```{=html}
<!-- -->
```
-   **STEP 10** - Develop new DOE Jobs ONLINE competency questions

## **STEP 1 - Identify and list all of the basic tasks or duties (i.e., analyze the job)**

-   Major benefits

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clear
    understanding of every little thing the position does

```{=html}
<!-- -->
```
-   It may reveal some tasks and duties in the position description that
    are not really performed

-   It may reveal some tasks and duties that are really performed are
    missing from the position description

```{=html}
<!-- -->
```
-   **STEP 2 - Think about how you would group basic tasks and duties
    identified in STEP 1 into major duties or functions and which basic
    tasks or duties you would put in your groups**

```{=html}
<!-- -->
```
-   Major benefits

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clear
    understanding of major functional areas for which the position is
    responsible

```{=html}
<!-- -->
```
-   It will make the original tasks and duties lists appear more
    manageable

```{=html}
<!-- -->
```
-   It may reveal some major duties or functions are missing

-   It may reveal some major duties or functions are really unimportant
    and therefore, you may decide later on that any basic tasks or
    duties associated with them need not be measured

```{=html}
<!-- -->
```
-   **STEP 3 - Think about a knowledge, skill, or ability statement for
    each major duty or function identified in STEP 2**

```{=html}
<!-- -->
```
-   Major benefits

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clear
    understanding of what the position needs in terms of knowledges,
    skills, and abilities

```{=html}
<!-- -->
```
-   It may reveal some KSAs that are really needed are missing

```{=html}
<!-- -->
```
-   It may reveal some KSAs that are not really needed

```{=html}
<!-- -->
```
-   It may help you later on in developing your questions by providing
    you with a known reference point

```{=html}
<!-- -->
```
-   **STEP 4 - Gather your thoughts from the first three steps for
    completeness and understanding**

```{=html}
<!-- -->
```
-   At this point in the process, both you and the selecting official
    should be able to clearly articulate in your minds either one of the
    following:

```{=html}
<!-- -->
```
-   The perfect person for this job should have (insert appropriate KSA
    statement), so that he/she can perform the basic tasks and duties
    associated with (insert appropriate major task, duty, or function)

-   The perfect person for this job should be able to perform the basic
    tasks and duties associated with (insert appropriate major task,
    duty, or function), which requires (insert appropriate KSA
    statement)

```{=html}
<!-- -->
```
-   Major benefit

```{=html}
<!-- -->
```
-   It will ensure that both you and the selecting official have the
    same understanding

```{=html}
<!-- -->
```
-   **STEP 5 - Decide which of the basic tasks or duties identified in
    STEP 2 are essential and which are desirable**

```{=html}
<!-- -->
```
-   Omit basic tasks or duties associated with major duties or functions
    that in STEP 2 you determined were unimportant

```{=html}
<!-- -->
```
-   For each remaining basic task or duty identified in STEP 1, decide

```{=html}
<!-- -->
```
-   If it is **ESSENTIAL** for an applicant to have had education,
    training, and/or experience in the basic task or duty prior to
    entering on duty in order to perform in a satisfactory way within a
    reasonable period of time (i.e., will an applicant who has not had
    any education, training, and/or experience in this specific task or
    duty prior to entering on duty be ineligible regardless of any and
    all other education, training, and/or experience the applicant has)

-   If it is **DESIRABLE** for an applicant to have had education,
    training, and/or experience in the basic task or duty prior to
    entering on duty in order to perform in a satisfactory way within a
    reasonable period of time (i.e., will an applicant who has not had
    any education, training, and/or experience in this specific task or
    duty prior to entering on duty still be eligible)

```{=html}
<!-- -->
```
-   Major benefits

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clearer
    understanding of what kinds of education, training, and/or
    experience

-   Applicants for the position must have prior to entering on duty

```{=html}
<!-- -->
```
-   Applicants need not necessarily have prior to entering on duty

```{=html}
<!-- -->
```
-   **STEP 6 - Qualify the desirable tasks or duties remaining from STEP
    5**

```{=html}
<!-- -->
```
-   For each desirable basic task or duty identified in STEP 5, decide

```{=html}
<!-- -->
```
-   If it is **REASONABLE** to expect applicants to have had education,
    training, and/or experience in the basic task or duty prior to
    entering on duty, taking into account applicant pool, labor market,
    grade level of the job, etc.

```{=html}
<!-- -->
```
-   If it is **RATABLE** using DOE Jobs ONLINE (i.e., can you ask a
    question or questions about it that will give you useable
    information)

-   If it is **VERIFIABLE** at some point in the selection process using
    the instruments available (i.e., resume, interview, reference check,
    etc.)

```{=html}
<!-- -->
```
-   If it **DISTINGUISHES** between a minimally qualified applicant and
    a highly qualified applicant (i.e., would an applicant who has had
    education, training, and/or experience in the basic task or duty
    prior to entering on duty be better qualified than one who has not)

```{=html}
<!-- -->
```
-   Major benefits

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clearer
    understanding of what kinds of education, training, and/or
    experience

-   That will be used to differentiate between minimally qualified and
    highly qualified applicants

```{=html}
<!-- -->
```
-   It may result in deciding to eliminate basic tasks or duties that
    are unreasonable, unratable, unverifiable, and/or undistinguishing

```{=html}
<!-- -->
```
-   **STEP 7 - Determine the relative importance of each remaining task
    or duty from STEP 6**

```{=html}
<!-- -->
```
-   No hard and fast rule for doing this

```{=html}
<!-- -->
```
-   Relative importance may be determined based on

```{=html}
<!-- -->
```
-   Consequences of the basic task or duty not being performed
    satisfactorily

```{=html}
<!-- -->
```
-   Benefits/payoff of the basic task or duty being performed
    satisfactorily

```{=html}
<!-- -->
```
-   Frequency with which the basic task or duty is performed absolutely
    (i.e., daily, weekly, monthly, semi-annually, annually, etc.)

```{=html}
<!-- -->
```
-   Frequency with which the basic task or duty is performed compared to
    other basic tasks or duties (i.e., more than, less than, same as,
    etc.)

```{=html}
<!-- -->
```
-   Grade level equivalent of the basic task or duty compared to the
    grade level of the position

```{=html}
<!-- -->
```
-   Grade level equivalent of the basic task or duty compared to the
    grade level equivalent of other basic tasks or duties

```{=html}
<!-- -->
```
-   Grade controlling tasks or duties should always have relatively high
    importance

```{=html}
<!-- -->
```
-   Assign a relative value to each basic task or duty which corresponds
    to its importance (i.e.: 3 = Very Important, 2 = Important, 1 = Less
    Important; or VI = Very Important, I = Important, LI = Less
    Important; VI = Very Important, I = Important, NI = Not As
    Important)

```{=html}
<!-- -->
```
-   Major benefits

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with what should
    be the final list of tasks and duties which will form the basis for
    your DOE Jobs ONLINE questions

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a basis for
    weighting DOE Jobs ONLINE responses

```{=html}
<!-- -->
```
-   It may reveal some additional tasks or duties are relatively
    unimportant

```{=html}
<!-- -->
```
-   It may result in eliminating additional relatively unimportant tasks
    or duties from the evaluation process

```{=html}
<!-- -->
```
-   **STEP 8 - Decide what type of information you're looking for**

```{=html}
<!-- -->
```
-   For each of the basic tasks and duties remaining from STEP 7, you
    and the selecting official need to decide what type of information
    you're looking for

```{=html}
<!-- -->
```
-   Absolute (i.e., has or has not performed the task or duty, no more
    and no less)

```{=html}
<!-- -->
```
-   Qualified absolute (i.e., has or has not performed the task or duty
    at a single specific level, no more and no less)

```{=html}
<!-- -->
```
-   Qualitative (i.e., progressively higher levels of expertise in
    performing the task or duty)

```{=html}
<!-- -->
```
-   Quantitative (i.e., has or has not performed a number of different
    tasks or duties)

```{=html}
<!-- -->
```
-   **STEP 9 - Select DOE Jobs ONLINE competency questions**

```{=html}
<!-- -->
```
-   For each of the basic tasks and duties remaining from STEP 7, you
    and the selecting need to decide

```{=html}
<!-- -->
```
-   If there is an existing question in the DOE Jobs ONLINE KSOC
    database that

-   Addresses the specific task or duty

-   Provides the type of response you're looking for

```{=html}
<!-- -->
```
-   Weights the responses appropriately or allows you to change the
    weights appropriately

```{=html}
<!-- -->
```
-   Meets the characteristics of a good competency question (i.e.,
    behaviorally consistent, distinguishing, good sounding, etc.)

```{=html}
<!-- -->
```
-   **STEP 10 - Develop new DOE Jobs ONLINE competency questions**

```{=html}
<!-- -->
```
-   For each of the basic tasks and duties remaining from STEP 8, for
    which you and the selecting official have decided that there is no
    appropriate question in the DOE Jobs ONLINE KSOC database, you need
    to

```{=html}
<!-- -->
```
-   Develop a new question to be added to the DOE Jobs ONLINE KSOC
    database that

```{=html}
<!-- -->
```
-   Addresses the specific task or duty

-   Provides the type of response you're looking for

```{=html}
<!-- -->
```
-   Weights the responses appropriately

```{=html}
<!-- -->
```
-   Meets the characteristics of a good competency question (i.e.,
    behaviorally consistent, distinguishing, good sounding, etc.)

**[DOE JOBS ONLINE COMPETENCY QUESTION DEVELOPMENT - PART
VI]{.underline}**

**[How do you decide what you want to measure (Readers Digest condensed
version)]{.underline}?**

## **STEP 1** - Identify and list all of the basic tasks or duties (i.e., analyze the job)

-   **STEP 2** - Decide which basic tasks or duties are essential and
    which are desirable

-   **STEP 3** - Qualify the desirable tasks or duties

```{=html}
<!-- -->
```
-   **STEP 4** - Determine the relative importance of each desirable
    task or duty

```{=html}
<!-- -->
```
-   **STEP 5** - Decide what type of information you're looking for

```{=html}
<!-- -->
```
-   **STEP 6** - Select existing DOE Jobs ONLINE competency questions

```{=html}
<!-- -->
```
-   **STEP 7** - Develop new DOE Jobs ONLINE competency questions

## **STEP 1 - Identify and list all of the basic tasks or duties (i.e., analyze the job)**

-   Major benefits

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clear
    understanding of every little thing the position does

```{=html}
<!-- -->
```
-   It may reveal some basic tasks and duties in the position
    description that are not really performed

-   It may reveal some basic tasks and duties that are really performed
    are missing from the position description

```{=html}
<!-- -->
```
-   It may reveal some tasks and duties you thought were "basic" are
    really "major" duties or functions

-   Major duties or functions are comprised of any number of "basic"
    tasks and duties which have a common denominator

```{=html}
<!-- -->
```
-   **STEP 2 - Decide which of the basic tasks or duties identified in
    STEP 1 are essential and which are desirable**

```{=html}
<!-- -->
```
-   For each remaining basic task or duty identified in Step 1, decide

```{=html}
<!-- -->
```
-   If it is **ESSENTIAL** for an applicant to have had education,
    training, and/or experience in the basic task or duty prior to
    entering on duty in order to perform in a satisfactory way within a
    reasonable period of time (i.e., will an applicant who has not had
    any education, training, and/or experience in this specific task or
    duty prior to entering on duty be ineligible regardless of any and
    all other education, training, and/or experience the applicant has)

-   If it is **DESIRABLE** for an applicant to have had education,
    training, and/or experience in the basic task or duty prior to
    entering on duty in order to perform in a satisfactory way within a
    reasonable period of time (i.e., will an applicant who has not had
    any education, training, and/or experience in this specific task
    duty or prior to entering on duty still be eligible)

```{=html}
<!-- -->
```
-   Major benefits

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clear
    understanding of what kinds of education, training, and/or
    experience

-   Applicants for the position must have prior to entering on duty

```{=html}
<!-- -->
```
-   Applicants need not necessarily have prior to entering on duty

```{=html}
<!-- -->
```
-   **STEP 3 - Qualify the desirable tasks or duties**

```{=html}
<!-- -->
```
-   For each desirable basic task or duty, decide

```{=html}
<!-- -->
```
-   If it is **REASONABLE** to expect applicants to have had education,
    training, and/or experience in the basic task or duty prior to
    entering on duty, taking into account applicant pool, labor market,
    grade level of the job, etc.

```{=html}
<!-- -->
```
-   If it is **RATABLE** using DOE Jobs ONLINE (i.e., can you ask a
    question or questions about it that will give you useable
    information)

-   If it is **VERIFIABLE** at some point in the selection process
    (i.e., through resume review, interview, reference check, etc.)

```{=html}
<!-- -->
```
-   If it **DISTINGUISHES** between a minimally qualified applicant and
    a highly qualified applicant (i.e., would an applicant who has had
    education, training, and/or experience in the basic task or duty
    prior to entering on duty be better qualified than one who has not)

```{=html}
<!-- -->
```
-   Major benefits

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a clearer
    understanding of what kinds of education, training, and/or
    experience

-   That will be used to differentiate between minimally qualified and
    highly qualified applicants

```{=html}
<!-- -->
```
-   It may result in deciding to eliminate basic tasks or duties that
    are unreasonable, unratable, unverifiable, and/or undistinguishing

```{=html}
<!-- -->
```
-   **STEP 4 - Determine the relative importance of each desirable task
    or duty**

```{=html}
<!-- -->
```
-   No hard and fast rule for doing this

```{=html}
<!-- -->
```
-   Relative importance may be determined based on

```{=html}
<!-- -->
```
-   Consequences of the basic task or duty not being performed
    satisfactorily

```{=html}
<!-- -->
```
-   Benefits/payoff of the basic task or duty being performed
    satisfactorily

```{=html}
<!-- -->
```
-   Frequency with which the basic task or duty is performed absolutely
    (i.e., daily, weekly, monthly, semi-annually, annually, etc.)

```{=html}
<!-- -->
```
-   Frequency with which the basic task or duty is performed compared to
    other basic tasks or duties (i.e., more than, less than, same as,
    etc.)

```{=html}
<!-- -->
```
-   Grade level equivalent of the basic task or duty compared to the
    grade level of the position

```{=html}
<!-- -->
```
-   Grade level equivalent of the basic task or duty compared to the
    grade level equivalent of other basic tasks or duties

```{=html}
<!-- -->
```
-   Grade controlling tasks or duties should always have relatively high
    importance

```{=html}
<!-- -->
```
-   Assign a relative value to each basic task or duty which corresponds
    to its importance (i.e.: 3 = Very Important, 2 = Important, 1 = Less
    Important; or VI = Very Important, I = Important, LI = Less
    Important; VI = Very Important, I = Important,

> NI = Not As Important)

-   Major benefits

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with what should
    be the final list of tasks and duties which will form the basis for
    your DOE Jobs ONLINE questions

```{=html}
<!-- -->
```
-   It will provide both you and the selecting official with a basis for
    weighting DOE Jobs ONLINE responses

```{=html}
<!-- -->
```
-   It may reveal some additional tasks or duties are relatively
    unimportant

```{=html}
<!-- -->
```
-   It may result in eliminating additional relatively unimportant tasks
    or duties from the evaluation process

```{=html}
<!-- -->
```
-   **STEP 5 - Decide what type of information you're looking for**

```{=html}
<!-- -->
```
-   For each of the basic tasks and duties remaining from STEP 4, you
    and the selecting official need to decide what type of information
    you're looking for

```{=html}
<!-- -->
```
-   Absolute (i.e., has or has not performed the task or duty, no more
    and no less)

```{=html}
<!-- -->
```
-   Qualified absolute (i.e., has or has not performed the task or duty
    at a single specific level, no more and no less)

```{=html}
<!-- -->
```
-   Qualitative (i.e., progressively higher levels of expertise in
    performing the task or duty)

```{=html}
<!-- -->
```
-   Quantitative (i.e., has or has not performed a number of different
    tasks or duties)

```{=html}
<!-- -->
```
-   **STEP 6 - Select DOE Jobs ONLINE competency questions**

```{=html}
<!-- -->
```
-   For each of the basic tasks and duties remaining from STEP 4, you
    and the selecting need to decide

```{=html}
<!-- -->
```
-   If there is an existing question in the DOE Jobs ONLINE KSOC
    database that

-   Addresses the specific task or duty

-   Provides the type of response you're looking for

```{=html}
<!-- -->
```
-   Weights the responses appropriately or allows you to change the
    weights appropriately

```{=html}
<!-- -->
```
-   Meets the characteristics of a good competency question (i.e.,
    behaviorally consistent, distinguishing, good sounding, etc.)

```{=html}
<!-- -->
```
-   **STEP 7 - Develop DOE Jobs ONLINE competency questions**

```{=html}
<!-- -->
```
-   For each of the basic tasks and duties remaining from STEP 4, for
    which you and the selecting official have decided that there is no
    appropriate question in the DOE Jobs ONLINE KSOC database, you need
    to

```{=html}
<!-- -->
```
-   Develop a new question to be added to the DOE Jobs ONLINE KSOC
    database that

```{=html}
<!-- -->
```
-   Addresses the specific task or duty

-   Provides the type of response you're looking for

```{=html}
<!-- -->
```
-   Weights the responses appropriately

```{=html}
<!-- -->
```
-   Meets the characteristics of a good competency question (i.e.,
    behaviorally consistent, distinguishing, good sounding, etc.)

**[DOE JOBS ONLINE COMPETENCY QUESTION DEVELOPMENT - PART
VII]{.underline}**

**[How about an overly simplified hypothetical demonstration that
compares and contrasts manual and DOE Jobs ONLINE rating and ranking
procedures]{.underline}?**

#### Sample paragraph from a real DOE position description

-   Assists project managers on engineering issues in the design and
    construction of industrial scale facilities for handling nuclear
    material. Facilities employ state of the art processes for plutonium
    disposition, blending of highly enriched uranium, irradiation of
    mixed-oxide fuel, and the immobilization of nuclear material. The
    incumbent coordinates input of recommendations on design
    alternatives, recommends the organization of design efforts, assists
    in the resolution of conflicts on competing processes or designs and
    assists in coordinating the recommendations of other experts
    addressing engineering problems such that overall project objectives
    are met. Performs an expert analysis of contractor prepared design
    documents and design details for facility construction and develops
    recommendations for the project manager on acceptance or rejection.
    As an expert in engineering design, assesses the difficulty of
    constructing proposed facility features.

```{=html}
<!-- -->
```
-   **STEP 1 - Identify and list all of the basic task and duties**

1.  Coordinates input of recommendations on design alternatives

2.  Recommends the organization of design efforts

3.  Assists in the resolution of conflicts on competing processes

4.  Assists in the resolution of conflicts on competing designs

5.  Assists in coordinating the recommendations of other experts
    addressing engineering problems such that overall project objectives
    are met

6.  Analyzes contractor prepared design documents and design details for
    facility construction

7.  Develops recommendations for the project manager on acceptance of
    contractor prepared design documents and design details for facility
    construction

8.  Develops recommendations for the project manager on rejection of
    contractor prepared design documents and design details for facility
    construction

9.  Assesses the difficulty of constructing proposed facility features

-   For purposes of this demonstration, let's assume that "assists
    project managers on engineering issues in the design and
    construction of industrial scale facilities for handling nuclear
    material" is the major function and everything else in that
    paragraph describes the basic tasks and duties the incumbent
    performs in order to "assist project managers on engineering issues
    in the design and construction of industrial scale facilities for
    handling nuclear material"

**[Manual rating/ranking]{.underline}:**

-   The KSA we might develop is "Ability to assist project managers on
    engineering issues in the design and construction of industrial
    scale facilities for handling nuclear material"

```{=html}
<!-- -->
```
-   We might then incorporate some or all of the basic tasks and duties,
    as well as the specific types of nuclear material specified in the
    paragraph, into our quality level definitions

```{=html}
<!-- -->
```
-   15 Points - Meets 10-Point level AND has experience independently
    analyzing contractor prepared design documents and design details
    for facility construction, developing recommendations on acceptance
    or rejection of contractor prepared design documents and design
    details for facility construction, and assessing the difficulty of
    constructing proposed facility features. In order to be creditable
    at this level, the experience must have been with facilities that
    employed state-of-the-art processes used in plutonium disposition
    and blending of highly enriched uranium.

```{=html}
<!-- -->
```
-   10 Points - Meets 5-Point level AND has experience resolving
    conflicts on competing processes or designs, and coordinating the
    recommendations of other experts addressing engineering problems
    such that overall project objectives are met. In order to be
    creditable at this level, the experience must have been with
    facilities that employed state-of-the-art processes used in
    irradiation of mixed-oxide fuel and the immobilization of nuclear
    material.

```{=html}
<!-- -->
```
-   5 Points - Experience recommending the organization of design
    efforts, and coordinating input of recommendations on design
    alternatives. In order to be creditable at this level, the
    experience must have been with facilities that employed
    state-of-the-art processes used in irradiation of mixed-oxide fuel
    or the immobilization of nuclear material.

```{=html}
<!-- -->
```
-   We might also come up with examples of other experience not
    specifically stated in the position description that would also be
    indicative of an applicant's quality level

-   We "encourage" an applicant to provide a narrative response
    addressing "Ability to assist project managers on engineering issues
    in the design and construction of industrial scale facilities for
    handling nuclear material"

-   We assign points based on whether or not an applicant's narrative
    response speaks to the kinds of experiences that meet the point
    level criteria.

**[DOE Jobs ONLINE]{.underline}:**

-   We would not ask a question that dealt with "assisting project
    managers on engineering issues in the design and construction of
    industrial scale facilities for handling nuclear material,"
    specifically

```{=html}
<!-- -->
```
-   "Assisting" is too nebulous

-   "Assisting" can mean different things to different people

-   "Assisting" can encompass any one or combination of any number of
    different tasks

```{=html}
<!-- -->
```
-   We would use the individual tasks and duties to develop questions
    that the applicant could respond to

# Absolute "Yes/No"

> 1\. "Have you coordinated input of recommendations on design
> alternatives for the design and construction of industrial scale
> facilities for handling nuclear material?"

2.  "Have you recommended the organization of design efforts for the
    design and construction of industrial scale facilities for handling
    nuclear material?"

3.  "Have you resolved conflicts on competing processes for the design
    and construction of industrial scale facilities for handling nuclear
    material?"

4.  "Have you resolved conflicts on competing designs for the design and
    construction of industrial scale facilities for handling nuclear
    material?"

5.  "Have you coordinated the recommendations of other experts
    addressing engineering problems such that overall design and
    construction of industrial scale facilities for handling nuclear
    material project objectives are met?"

6.  "Have you analyzed contractor prepared design documents and design
    details for the design and construction of industrial scale
    facilities for handling nuclear material?"

7.  "Have you developed recommendations for project managers on
    acceptance of contractor prepared design documents and design
    details for the design and construction of industrial scale
    facilities for handling nuclear material?"

8.  "Have you developed recommendations for project managers on
    rejection of contractor prepared design documents and design details
    for the design and construction of industrial scale facilities for
    handling nuclear material?"

9.  "Have you assessed the difficulty of constructing proposed facility
    features for industrial scale facilities for handling nuclear
    material?"

10. "Was any of your experience in Nos. 1 through 9 above involved with
    processes for plutonium disposition?"

11. "Was any of your experience in Nos. 1 through 9 above involved with
    processes for blending of highly enriched uranium?"

12. "Was any of your experience in Nos. 1 through 9 above involved with
    processes for irradiation of mixed-oxide fuel?"

13. "Was any of your experience in Nos. 1 through 9 above involved with
    processes for the immobilization of nuclear material?"

# Qualified "Yes/No"

1.  "Have you independently, and usually without review by your
    supervisor or a senior employee, coordinated input of
    recommendations on design alternatives for the design and
    construction of industrial scale facilities for handling nuclear
    material?"

2.  "Have you independently, and usually without review by your
    supervisor or a senior employee, recommended the organization of
    design efforts for the design and construction of industrial scale
    facilities for handling nuclear material?"

3.  "Have you independently, and usually without review by your
    supervisor or a senior employee, resolved conflicts on competing
    processes for the design and construction of industrial scale
    facilities for handling nuclear material?"

4.  "Have you independently, and usually without review by your
    supervisor or a senior employee, resolved conflicts on competing
    designs for the design and construction of industrial scale
    facilities for handling nuclear material?"

5.  "Have you independently, and usually without review by your
    supervisor or a senior employee, coordinated the recommendations of
    other experts addressing engineering problems such that overall
    design and construction of industrial scale facilities for handling
    nuclear material project objectives are met?"

6.  "Have you independently, and usually without review by your
    supervisor or a senior employee, analyzed contractor prepared design
    documents and design details for the design and construction of
    industrial scale facilities for handling nuclear material?"

7.  "Have you independently, and usually without review by your
    supervisor or a senior employee, developed recommendations for
    project managers on acceptance of contractor prepared design
    documents and design details for the design and construction of
    industrial scale facilities for handling nuclear material?"

8.  "Have you independently, and usually without review by your
    supervisor or a senior employee, developed recommendations for
    project managers on rejection of contractor prepared design
    documents and design details for the design and construction of
    industrial scale facilities for handling nuclear material?"

9.  "Have you independently, and usually without review by your
    supervisor or a senior employee, assessed the difficulty of
    constructing proposed facility features for industrial scale
    facilities for handling nuclear material?"

10. "Was any of your experience in Nos. 1 through 9 above involved with
    processes for plutonium disposition?"

11. "Was any of your experience in Nos. 1 through 9 above involved with
    processes for blending of highly enriched uranium?"

12. "Was any of your experience in Nos. 1 through 9 above involved with
    processes for irradiation of mixed-oxide fuel?"

13. "Was any of your experience in Nos. 1 through 9 above involved with
    processes for the immobilization of nuclear material?"

# Self-Assessment

1.  Please select the one response that best describes your education,
    training, and/or experience in coordinating input of recommendations
    on design alternatives for the design and construction of industrial
    scale facilities for handling nuclear material.

```{=html}
<!-- -->
```
1.  I have no education, training or experience performing this task.

2.  I have had education or training in this task but have not yet
    performed it on-the-job.

3.  I have performed this task on-the-job with close supervision from a
    supervisor or senior employee.

4.  I have performed this task as a regular part of a job and normally
    without review by a supervisor or senior employee.

5.  I am considered an expert in this task and am normally consulted by
    others for assistance in performing this task.

```{=html}
<!-- -->
```
2.  Please select the one response that best describes your education,
    training, and/or experience in recommending the organization of
    design efforts for the design and construction of industrial scale
    facilities for handling nuclear material.

3.  Please select the one response that best describes your education,
    training, and/or experience in resolving conflicts on competing
    processes for the design and construction of industrial scale
    facilities for handling nuclear material.

4.  Please select the one response that best describes your education,
    training, and/or experience in resolving conflicts on competing
    designs for the design and construction of industrial scale
    facilities for handling nuclear material.

5.  Please select the one response that best describes your education,
    training, and/or experience in coordinating the recommendations of
    other experts addressing engineering problems such that overall
    design and construction of industrial scale facilities for handling
    nuclear material project objectives are met.

6.  Please select the one response that best describes your education,
    training, and/or experience in analyzing contractor prepared design
    documents and design details for the design and construction of
    industrial scale facilities for handling nuclear material

7.  Please select the one response that best describes your education,
    training, and/or experience in developing recommendations for
    project managers on acceptance of contractor prepared design
    documents and design details for the design and construction of
    industrial scale facilities for handling nuclear material

8.  Please select the one response that best describes your education,
    training, and/or experience in developing recommendations for
    project managers on rejection of contractor prepared design
    documents and design details for the design and construction of
    industrial scale facilities for handling nuclear material

9.  Please select the one response that best describes your education,
    training, and/or experience in assessing the difficulty of
    constructing proposed facility features for industrial scale
    facilities for handling nuclear material

# 

# Multiple choice/Multiple answer

-   Please check the box next to each type of facility you have
    designed.

1.  Facility for plutonium disposition

2.  Facility for blending of highly enriched uranium

3.  Facility for irradiation of mixed-oxide fuel

4.  Facility for the immobilization of nuclear material

-   Please check the box next to each task you have performed
    independently, and usually without review by your supervisor or a
    senior employee, as a regular part of a job.

> 1\. "Coordinated input of recommendations on design alternatives for
> the design and construction of industrial scale facilities for
> handling nuclear material."

2.  "Recommended the organization of design efforts for the design and
    construction of industrial scale facilities for handling nuclear
    material."

3.  "Resolved conflicts on competing processes for the design and
    construction of industrial scale facilities for handling nuclear
    material."

4.  "Resolved conflicts on competing designs for the design and
    construction of industrial scale facilities for handling nuclear
    material."

5.  "Coordinated the recommendations of other experts addressing
    engineering problems such that overall design and construction of
    industrial scale facilities for handling nuclear material project
    objectives are met."

6.  "Analyzed contractor prepared design documents and design details
    for the design and construction of industrial scale facilities for
    handling nuclear material."

7.  "Developed recommendations for project managers on acceptance of
    contractor prepared design documents and design details for the
    design and construction of industrial scale facilities for handling
    nuclear material."

8.  "Developed recommendations for project managers on rejection of
    contractor prepared design documents and design details for the
    design and construction of industrial scale facilities for handling
    nuclear material."

9.  "Assessed the difficulty of constructing proposed facility features
    for industrial scale facilities for handling nuclear material."

[So what's the point]{.underline}?

-   Each procedure starts out the same

```{=html}
<!-- -->
```
-   Each procedure, either directly or indirectly, measures tasks and
    duties an applicant has performed

```{=html}
<!-- -->
```
-   The thought processes involved in developing each procedure are very
    comparable

```{=html}
<!-- -->
```
-   There are normally three to six questions (in the form of KSAs) in a
    manually rated announcement

```{=html}
<!-- -->
```
-   There will normally be far more than six questions in a DOE Jobs
    ONLINE rated announcement

```{=html}
<!-- -->
```
-   Responses to any KSA are in one single format (i.e., applicants are
    encouraged to submit a narrative addressing the KSAs)

```{=html}
<!-- -->
```
-   Responses to DOE Jobs ONLINE questions can be in a variety of
    different formats (i.e., Yes/No, Applicant Assessment, Multiple
    choice/Multiple Answer, True/False, etc.)

**[DOE JOBS ONLINE COMPETENCY QUESTION DEVELOPMENT - PART
VIII]{.underline}**

**[How about a strategy for working with selecting officials to develop
DOE Jobs ONLINE competency questions]{.underline}?**

[Play a game with the selecting official and here's your
script]{.underline}:

-   I have just given you identical resumes from 25 applicants for your
    vacancy. Like most resumes, they're pretty sketchy and don't say a
    whole heck of a lot. At least I was able to determine that they are
    all basically qualified for the position. And if you think these
    look pretty pitiful, you ought to see the ones that didn't even make
    it this far.

```{=html}
<!-- -->
```
-   You can interview all 25, but you can only make a selection from the
    three applicants you determine to be the best qualified. To do this,
    you'll be conducting the interviews in two stages. The first stage
    is where you'll actually assign numerical ratings to all 25
    applicants. The second stage is where you'll call the top three back
    in for a follow-up interview.

```{=html}
<!-- -->
```
-   For the first stage, we need to develop a set of job-related
    questions that you will ask all of the applicants. Here are the
    ground rules:

```{=html}
<!-- -->
```
-   We can ask as many questions as you like

```{=html}
<!-- -->
```
-   Each question must deal with a single basic task or duty from the
    position description that you decide is a relatively important part
    of the job

```{=html}
<!-- -->
```
-   You can base your decisions on relative importance on any one or
    combination of the following:

```{=html}
<!-- -->
```
-   Consequences of the task not being performed satisfactorily

```{=html}
<!-- -->
```
-   Benefits/payoff of the task being performed satisfactorily

```{=html}
<!-- -->
```
-   Frequency with which the task is performed absolutely (i.e., daily,
    weekly, monthly, semi-annually, annually, etc.)

```{=html}
<!-- -->
```
-   Frequency with which the task is performed compared to other tasks
    (i.e., more than, less than, same as, etc.)

```{=html}
<!-- -->
```
-   Grade level equivalent of the task compared to the grade level of
    the position

```{=html}
<!-- -->
```
-   Grade level equivalent of the task compared to the grade level
    equivalent of other tasks

```{=html}
<!-- -->
```
-   To keep things as simple as we can and to keep track of things, why
    don't we label each very important task "VI," each average important
    task "I," and each not as important task "NI." That way, you can
    decide if it's even going to be worth your while to ask questions
    about tasks that are not as important.

```{=html}
<!-- -->
```
-   We can ask "Yes/No," "Multiple choice/Multiple answer," "Applicant
    Assessment," or "Number" questions. We can also ask "True/False"
    questions, but personally, I like to stay away from them because
    they're a little more difficult to write up. Besides, most anything
    that can be answered "True" or "False" can also be answered "Yes" or
    "No."

```{=html}
<!-- -->
```
-   Let me give you some real simplistic examples of what I'm talking
    about

```{=html}
<!-- -->
```
-   If you just simply wanted to know whether or not an applicant for an
    Electrician position has performed a particular task, we might ask a
    Yes/No question: "Have you installed electric ceiling fans?"

```{=html}
<!-- -->
```
-   If you just wanted to know whether or not that applicant has
    performed a particular task at only one particular level of
    responsibility, we might again ask a slightly differently worded
    Yes/No question: "Have you installed electric ceiling fans as a
    regular part of a job, independently and usually without review by
    your supervisor or a senior employee?"

```{=html}
<!-- -->
```
-   If you wanted to know whether or not the applicant has either
    education, training, or experience performing a particular task, we
    might ask an Applicant Assessment question: "Please select the one
    response that best describes your education, training, and/or
    experience in installing electric ceiling fans."

1.  I have no education, training or experience performing this task.

2.  I have had education or training in this task but have not yet
    performed it on-the-job.

3.  I have performed this task on-the-job with close supervision from a
    supervisor or senior employee.

4.  I have performed this task as a regular part of a job and usually
    without review by a supervisor or senior employee.

5.  I am considered an expert in this task and am normally consulted by
    others for assistance in performing this task.

-   If you wanted to know whether or not an applicant has performed a
    number of related tasks, or a number of related tasks at only one
    particular level of responsibility, we might ask a Multiple
    choice/Multiple answer question: "Please check the box next to each
    task you have performed as a regular part of a job, independently
    and usually without review by a supervisor or senior employee."

1.  "Installed ceiling fans without lights."

2.  "Installed ceiling fans with integrated lights."

3.  "Installed ceiling fans with the ceiling fan manufacturer's separate
    light kit."

4.  "Installed ceiling fans with after-market or other ceiling fan
    manufacturer's separate light kits."

5.  "Installed ceiling fan remote control units."

6.  "Retrofitted installed ceiling fans with lights."

-   If you wanted to know if the applicant has worked on a specific type
    of electrical system expressed in number of volts, we might ask a
    Number question: "Please indicate in the box below the maximum
    electrical system voltage of any structure you've worked in."

```{=html}
<!-- -->
```
-   Alternatively, we could ask a Yes/No question: "Have you worked in
    structure with a 220 volt (or greater) electrical system?"

```{=html}
<!-- -->
```
-   After we've developed the questions or while we're developing, we'll
    need to weight the possible responses so that questions dealing with
    very important tasks are given more credit than questions dealing
    with average important tasks.

```{=html}
<!-- -->
```
-   The questions we develop for the first stage have to carefully
    selected and carefully worded because the numerical ratings are
    going to be based solely on the applicants' responses to the
    question. In the first stage, we can't ask applicants to elaborate
    on their responses, give us some examples, provide some additional
    information, etc., which will allow you to assess whether their
    responses are accurate.

```{=html}
<!-- -->
```
-   The only applicants you'll be able to ask to elaborate on their
    responses, give you some examples, provide some additional
    information, etc., which will allow you to assess whether their
    responses are accurate are the top three from the first stage who
    you'll call back in for a follow-up interview.

```{=html}
<!-- -->
```
-   I know this is going to be difficult at first, but I've already
    identified and listed all of the basic tasks and duties from your
    position description. Why don't we start by looking over the list
    and you can let me know if I've missed anything or if anything I've
    listed doesn't belong anymore.

[Some other things you may want to incorporate]{.underline}:

-   Use an example from the selecting official's position description

```{=html}
<!-- -->
```
-   Explain the difference between a "basic task or duty" and a "major
    duty or function"

```{=html}
<!-- -->
```
-   Explain that by asking questions about "basic tasks and duties"
    you're indirectly evaluating whether or not the applicant has
    performed the "major duty or function"

-   Explain that by asking questions about "basic tasks and duties"
    you're indirectly evaluating whether or not the applicant has the
    knowledges, skills, and/or abilities needed to perform the "basic
    tasks and duties"

# **[What if the selecting official doesn't like to play games]{.underline}?**

# Try doing it the "old fashioned" way

-   Develop KSAs

```{=html}
<!-- -->
```
-   Determine the relative importance of each KSA

```{=html}
<!-- -->
```
-   Assign a value to each KSA based on its relative importance

```{=html}
<!-- -->
```
-   Eliminate relatively unimportant KSAs

```{=html}
<!-- -->
```
-   Develop three benchmarks (i.e., Good, Better, Best) for the
    remaining KSAs

```{=html}
<!-- -->
```
-   Express benchmarks in terms of tasks or duties performed

```{=html}
<!-- -->
```
-   Select existing DOE Jobs ONLINE questions that address the specific
    tasks or duties performed (if appropriate)

```{=html}
<!-- -->
```
-   Develop new DOE Jobs ONLINE questions that address the specific
    tasks or duties performed (if necessary)

```{=html}
<!-- -->
```
-   Weight the responses accordingly

# **[DOE JOBS ONLINE COMPETENCY QUESTION DEVELOPMENT - PART IX]{.underline}**

**[What's the challenge for HR Specialists]{.underline}?**

-   To start thinking "task-based" instead of "KSA-based"

```{=html}
<!-- -->
```
-   To involve selecting officials in the process

```{=html}
<!-- -->
```
-   To put in the time and effort to make it work

```{=html}
<!-- -->
```
-   To train selecting officials in the process

-   To become partners with selecting officials

```{=html}
<!-- -->
```
-   To learn from selecting officials

```{=html}
<!-- -->
```
-   To massage the egos of selecting officials

```{=html}
<!-- -->
```
-   To help selecting officials by doing as much of the preparatory work
    as you can

```{=html}
<!-- -->
```
-   To help yourselves by stopping short of doing selecting officials'
    jobs

```{=html}
<!-- -->
```
-   To respond to questions, complaints, etc., without saying "The
    computer did it"

```{=html}
<!-- -->
```
-   To use existing resources if and when possible and appropriate

```{=html}
<!-- -->
```
-   To recognize when existing resources are of little use, no use, or
    inappropriate

```{=html}
<!-- -->
```
-   To recognize when something didn't work well and change it

```{=html}
<!-- -->
```
-   To learn from \"glitches\"

```{=html}
<!-- -->
```
-   To share what you've done, both good and bad, with your peers

```{=html}
<!-- -->
```
-   To maximize the "good"

-   To minimize the "not so good"

**[What's the challenge for selecting officials]{.underline}?**

-   To start thinking "task-based" instead of "KSA-based"

```{=html}
<!-- -->
```
-   To get involved in the process

```{=html}
<!-- -->
```
-   To put in the time and effort to make it work

```{=html}
<!-- -->
```
-   To train HR specialists in their jobs

-   To become partners with HR specialists

-   To massage the egos of HR specialists

```{=html}
<!-- -->
```
-   To learn from HR specialists

```{=html}
<!-- -->
```
-   To help HR specialists by doing as much of the preparatory work as
    they can

```{=html}
<!-- -->
```
-   To not be satisfied with "The computer did it" responses to
    questions, complaints, etc.

```{=html}
<!-- -->
```
-   To use existing resources if and when possible and appropriate

```{=html}
<!-- -->
```
-   To recognize when existing resources are of little use, no use, or
    inappropriate

```{=html}
<!-- -->
```
-   To recognize when something didn't work well and change it

```{=html}
<!-- -->
```
-   To learn from \"glitches\"

```{=html}
<!-- -->
```
-   To share what they've done, both good and bad, with their peers

```{=html}
<!-- -->
```
-   To maximize the "good"

```{=html}
<!-- -->
```
-   To minimize the "not so good"

**[How do we get there]{.underline}?**

-   There's no magic formula

```{=html}
<!-- -->
```
-   You don't need one anyway

```{=html}
<!-- -->
```
-   It's not difficult

```{=html}
<!-- -->
```
-   It's just tedious

```{=html}
<!-- -->
```
-   It becomes easier every time you do it

```{=html}
<!-- -->
```
-   Because you can come up with your own shortcuts

-   Because you can draw on past experience

```{=html}
<!-- -->
```
-   Think about it

```{=html}
<!-- -->
```
-   Mull it over in your mind before you start on it

```{=html}
<!-- -->
```
-   Think it through

```{=html}
<!-- -->
```
-   Think about the goal and how you want to achieve it

```{=html}
<!-- -->
```
-   Do it one step at a time

```{=html}
<!-- -->
```
-   A big problem is nothing more than a bunch of little problems strung
    together

```{=html}
<!-- -->
```
-   Each time you solve a little part of a big problem, you move that
    much closer to solving the big problem

```{=html}
<!-- -->
```
-   A big project is nothing more than a bunch of little projects strung
    together

```{=html}
<!-- -->
```
-   Each time you complete a little part of a big project, you move that
    much closer to completing the big project

```{=html}
<!-- -->
```
-   Think about it two steps ahead

```{=html}
<!-- -->
```
-   Each successive step in the process depends on the preceding step
    and each step lays the foundation for the next

```{=html}
<!-- -->
```
-   Put in the time and the effort

```{=html}
<!-- -->
```
-   Don't yield to the temptation of taking the easy way out

**[What are those "easy ways out" we want to avoid]{.underline}?**

-   Measuring only those basic tasks and duties for which questions
    already exist without first determining whether or not those tasks
    and duties are important for successful performance

```{=html}
<!-- -->
```
-   Measuring a major duty or function instead of the basic tasks and
    duties that are associated with it

```{=html}
<!-- -->
```
-   Measuring a knowledge instead of the basic tasks and duties that
    require possession of the knowledge in order to perform them

```{=html}
<!-- -->
```
-   Using the same questions and/or weights for subsequent announcements
    for the same position without first making sure nothing has changed
    from the previous announcement

```{=html}
<!-- -->
```
-   Using the same questions from one announcement for another
    announcement without first making sure everything is exactly the
    same

```{=html}
<!-- -->
```
-   Not taking into account changes in the job due to changing
    technology

-   Not taking account changes in the job due to a change in the
    supervisor

-   Not taking into account changes in the job due to a change in agency
    mission or priorities

```{=html}
<!-- -->
```
-   Using compound questions

```{=html}
<!-- -->
```
-   Using global, generic, or non-specific questions

-   Using global, generic, or non-specific terms in questions

-   Using subjective or inexact adjectives in questions

**[How about some final thoughts on all of this]{.underline}?**

-   You and the selecting official are a team and are jointly
    responsible

```{=html}
<!-- -->
```
-   Positions are dynamic

```{=html}
<!-- -->
```
-   People are dynamic

```{=html}
<!-- -->
```
-   Organizations are dynamic

```{=html}
<!-- -->
```
-   Systems are dynamic

```{=html}
<!-- -->
```
-   All applicant evaluation systems, automated or manual, are
    intrinsically worthless

-   All applicant evaluation systems, automated or manual, derive their
    value solely from the people who use them

```{=html}
<!-- -->
```
-   Successful implementation and use of any applicant evaluation
    system, automated or manual, is directly proportional to the time
    and effort we put in to make it successful

```{=html}
<!-- -->
```
-   You can please some of the people all of the time

```{=html}
<!-- -->
```
-   You can please all of the people some of the time

```{=html}
<!-- -->
```
-   You can't please all of the people all of the time

**[What can we conclude from all of this]{.underline}?**

-   We MUST validly evaluate applicants because the law requires it

```{=html}
<!-- -->
```
-   We SHOULD validly evaluate applicants because it makes good business
    sense

```{=html}
<!-- -->
```
-   The techniques involved in developing valid task-based DOE Jobs
    ONLINE competency questions are very similar to the techniques
    involved in developing valid KSA-based applicant assessment tools