# O*NET® Analyst Occupational Skill Ratings: Procedures

Suzanne Tsacoumis
Shannon Willison

Prepared for:

National Center for O*NET Development
North Carolina O*NET Center
700 Wade Avenue
Raleigh, NC 27605

March 2010
# O*NET ANALYST OCCUPATIONAL SKILL RATINGS: PROCEDURES

## Executive Summary

The Occupational Information Network (O*NET®) is a comprehensive system developed by the U.S. Department of Labor that provides information for 965 occupations within the U.S. economy. This information is maintained in a comprehensive database, developed to replace the Dictionary of Occupational Titles (DOT) (U.S. Department of Labor, 1991). In order to keep the database current, the National Center for O*NET Development is involved in a continual data collection process aimed at identifying and maintaining up-to-date information on the characteristics of workers and jobs. In order to ensure a controlled data collection and management process, occupational data is being collected in groups or “cycles” of approximately 100 occupations.

Most of the occupational information is collected from job incumbents, including: occupational tasks, generalized work activities, knowledge, education and training, work styles, and work context areas. Occupational analysts provide the importance and level information regarding the abilities and skills associated with these occupations. This report describes the procedures used to collect the occupational analyst ratings on the 35 skill constructs, which mirror the procedures established for collecting ability ratings as described in *O*\*NET Analyst Occupational Abilities Ratings: Procedures* (Donsbach, Tsacoumis, Sager, & Updegraff, 2003).

To facilitate the skill ratings, relevant occupational information was developed from recent data collected from job incumbents. This information was provided to occupational analysts to help them make skill level and importance ratings. Specifically, occupational analysts received the:

* Title and definition of the occupation

* Job zone of the occupation (i.e., level of vocational preparation needed)

* Mean importance of core and supplementary tasks for the targeted occupation

* Mean importance of knowledge domains that have a mean importance $\ge$ 3.0

* Mean importance of Generalized Work Activities (GWAs) that (1) have a mean importance for the occupation $\ge$ 3.0, and (2) require the targeted skill to be performed

* Mean rating of Work Context (WC) statements that (1) have a mean ratings for the targeted occupation $\ge$ 3.0, and (2) require the targeted skill to work in that context

Following the development of the occupational information, 16 occupational analysts were selected as raters based on criteria related to education and work experience. These occupational analysts were trained to interpret the occupational data and make importance and level ratings of the skills. Following standardized procedures to review the occupational information, the trained occupational analysts completed the rating process. Occupational analysts' performance and ratings were monitored and evaluated throughout the project. If necessary, remedial training and guidance were provided. Final importance and level ratings for each occupation were delivered to the National Center for O*NET Development.
# O*NET ANALYST OCCUPATIONAL SKILL RATINGS: PROCEDURES

## Table of Contents

<table><tr><td>Executive Summary</td><td>i</td></tr><tr><td>Introduction</td><td>1</td></tr><tr><td>Skill Rating Process</td><td>3</td></tr><tr><td>Tasks</td><td>4</td></tr><tr><td>Knowledge</td><td>4</td></tr><tr><td>Generalized Work Activities (GWAs)</td><td>5</td></tr><tr><td>Work Context Descriptors</td><td>6</td></tr><tr><td>Skill/GWA and WC Linkages</td><td>6</td></tr><tr><td>Skill Rating Procedures</td><td>7</td></tr><tr><td>Stimulus Material Development</td><td>8</td></tr><tr><td>Occupation Title, Definition, and Job Zone</td><td>8</td></tr><tr><td>Tasks</td><td>8</td></tr><tr><td>Knowledge</td><td>8</td></tr><tr><td>Generalized Work Activities</td><td>9</td></tr><tr><td>Work Context Statements</td><td>9</td></tr><tr><td>Occupational Analysts</td><td>9</td></tr><tr><td>Occupational Analyst Criteria</td><td>10</td></tr><tr><td>Occupational Analyst Training</td><td>10</td></tr><tr><td>Experienced Occupational Analyst Training</td><td>11</td></tr><tr><td>Data Collection</td><td>11</td></tr><tr><td>Rating Schedule and Assignments</td><td>11</td></tr><tr><td>Disseminating Stimulus Materials</td><td>12</td></tr><tr><td>Recording the Ratings</td><td>12</td></tr><tr><td>Rating Analysis</td><td>13</td></tr><tr><td>Batch Analysis</td><td>13</td></tr><tr><td>Cycle Analysis</td><td>13</td></tr><tr><td>Summary</td><td>15</td></tr><tr><td>References</td><td>16</td></tr><tr><td>Appendix A - Skill Definitions by Category</td><td>A-1</td></tr><tr><td>Appendix B - Job Zone Descriptions</td><td>B-1</td></tr><tr><td>Appendix C - Knowledge Domain Definitions</td><td>C-1</td></tr><tr><td>Appendix D - Linkage Exercise: Participant Instructions</td><td>D-1</td></tr><tr><td>Appendix E - Generalized Work Activity and Work Context Linkage Results</td><td>E-1</td></tr></table>
<table><tr><td>Appendix F - Rating Instructions</td><td>F-1</td></tr><tr><td>Appendix G - Sample Stimulus Material</td><td>G-1</td></tr></table>

List of Tables

<table><tr><td>Table 1. General Batch Assignments</td><td>12</td></tr></table>

List of Figures

<table><tr><td>Figure 1. O*NET Analyst Data Collection Process</td><td>2</td></tr></table>
# O*NET ANALYST OCCUPATIONAL SKILL RATINGS: PROCEDURES

## Introduction

The Occupational Information Network (O*NET) is a comprehensive system developed by the U.S. Department of Labor that provides information for 965 occupations within the U.S. economy. This information is maintained in a comprehensive database, developed to replace the Dictionary of Occupational Titles (DOT) (U.S. Department of Labor, 1991). In order to keep the database current, the National Center for O*NET Development is involved in a continual data collection process aimed at identifying and maintaining current information on the characteristics of workers and jobs. The information that populates the O*NET database is collected from three primary sources: incumbents, occupational experts, and occupational analysts. Targeted job incumbents provide ratings on occupational tasks, generalized work activities (GWA), knowledge, education and training, work styles, and work context areas. Importance and level information regarding the abilities and skills associated with these occupations is collected from occupational analysts.

Although skill information initially was collected from job incumbents, recent research suggests that trained occupational analysts understand the skill constructs better than incumbents (Morgeson & Campion, 1997; Tsacoumis, 2007), and therefore it follows that they should provide the skill data. In these instances, it is imperative that the occupational analysts have detailed occupation information in order to rate the skill constructs. It has also been suggested that incumbents are more likely than analysts to deliberately inflate their ratings to influence policy decisions such as those associated with compensation and training (Harvey, 1991; Morgeson, Delaney-Klinger, Mayfield, Ferrara, & Campion, 2004). Skill ratings may be particularly vulnerable to such effects given that they are more abstract and thus more difficult to verify than more observable descriptors such as job tasks (Morgeson & Campion, 1997; Morgeson et al., 2004). Given these considerations, occupational analysts as opposed to incumbents provided the skill information in the O*NET database.

Skills are proficiencies that are developed through training or experience. The 35 skills in the O*NET database are grouped into seven categories: content, process, social, complex problem solving, technical, systems, and resource management (see Appendix A). To facilitate the skill rating process, occupational analysts are provided relevant occupational information from incumbents. The purpose of this report is to describe the entire analyst data collection process, from preparation of the materials describing occupational data to analysis of the final skill ratings. A flow diagram of this process is presented in Figure 1.

It should be noted that to ensure a controlled data collection and management process, occupational data are being collected in groups or “cycles.” This report describes the data collection process that generalizes across all 965 occupations. These procedures are nearly identical to those described in *O*\*NET Analyst Occupational Abilities Ratings: Procedures (Donsbach, Tsacoumis, Sager, & Updegraff, 2003), with only minor modifications to streamline the process.
<figure><img src="image_1.png" /><figcaption>Figure 1. O*NET Analyst Data Collection Process</figcaption></figure>
## Skill Rating Process

As noted above, the skill rating process followed the procedures that were previously established to rate abilities. Similar considerations were given to the development of the rating process and stimulus materials. The ultimate goal in developing the stimulus material was to identify the data that would provide sufficient information about the occupation as a whole so the occupational analysts can make accurate skill importance and level ratings. At the same time, it was important to balance the desire to present all available information with the possibility of overwhelming the occupational analysts with more data than can be reasonably processed. Based on a review of the O*NET Content Model (Peterson, Mumford, Borman, Jeanneret, Fleishman, 1995; Peterson, Mumford, Borman, Jeanneret, Fleishman, & Levin, 1997; Peterson, Mumford, Borman, Jeanneret, Fleishman, 1999), it was determined that the following information available in the O*NET database would be informative for the occupational analysts and was included in the stimulus materials:

* Title, definition, and job zone of the occupation

* Task statements and data

* Knowledge statements and data

* Generalized Work Activity (GWA) statements and data

* Work Context items and data

These pieces of information offer a comprehensive description of the occupation and provide insight about how the skills may be applied on the job. In contrast, the other domains (work styles) were not deemed to provide critical information required to make judgments about the importance or level of the skills to an occupation.

Note Job Zone information was included in the stimulus materials to describe occupations beginning in Cycle 9. Job zone is a classification system that describes the amount of preparation that can lead to successful job performance (National Center for O*NET Development, 2008). Occupations are sorted into five job zones based on similar levels of vocational preparation (Appendix B). The O*NET center, HumRRO, and occupational analysts agreed that providing job zone information would assist in understanding the scope of different occupations.

The presentation of occupation title, definition, and job zone is straightforward. In contrast, there are a number of different pieces of information that could be presented for the tasks, knowledge statements, GWAs, and Work Context items. For example, incumbents provide a variety of ratings (e.g., importance, frequency, level) for tasks, knowledge statements, GWAs, and Work Context items. Given this, the next step was to determine what data, specifically, would be included in the stimulus materials to facilitate the occupational analyst rating process. The issues and final decisions associated with each key piece of occupational information included in the stimulus materials are presented on the following pages.
## Tasks

A task is defined as an activity that occurs in order to produce a product or outcome required on the job (Peterson, et al., 1999). In reviewing the options for what task data should be presented, it was determined that it would be meaningful for the occupational analysts to be aware of the importance associated with each task, although frequency and relevance were deemed less informative. With regard to importance, it seemed insufficient to provide a list of the important tasks, without a clear understanding of the degree to which each task was important to the specific occupation. It is likely, however, that the importance level of the various tasks may influence the judgments about the importance and level of a skill. Thus, the mean task importance ratings, rounded to the nearest tenth, were provided in the stimulus material.

Clearly, knowing the tasks' importance to the occupation is informative in rating a skill's importance and level. The other two pieces of information to consider when reviewing tasks is the frequency with which a task is performed and the relevance of the task to the occupation. Frequency data do not provide sufficient unique information relevant to the skill rating process to warrant inclusion in the stimulus materials. The relevance data, though, are different. Relevance data were captured in a fairly indirect manner – the incumbent either marked “not relevant” or provided a rating. Therefore, a task was considered relevant if (1) “not relevant” was *not* marked, and (2) either an importance or frequency rating was provided. Although it does not seem critical to share the exact percentage of incumbents that indicated a task is relevant to their job, there may be some benefit to communicating, in general, the relevance of the task. Thus as a means to help interpret the importance ratings, tasks were grouped into three categories: core, supplemental, and non-relevant. More specifically, statements rated on relevance or importance by 15 or more incumbents were classified into one of the three categories:

* **Core Tasks:** (a) relevance ≥ 67% and (b) a mean importance rating ≥ 3.0

* **Supplementary Tasks:** (a) tasks rated > 67% on relevance but < 3.0 on importance, or (b) tasks rated between 10% and 66% on relevance, regardless of mean importance rating

* **Non-Relevant Tasks:** relevance < 10% regardless of mean importance

Tasks that fell into the non-relevant category were not identified as meaningful data for the occupational analyst's rating process and therefore omitted from the stimulus materials. Presenting the task data in terms of either core or supplementary tasks was considered beneficial, since occupational analysts can appreciate the incumbent perspective regarding the relative importance of tasks and thus, increase the accuracy of their ratings.

## Knowledge

Knowledge is defined as organized sets of principles and facts applying in general domains (Peterson, et al., 1999). Through discussions with occupational analysts after multiple rating cycles, it was determined that it would be meaningful for the analysts to be aware of the
most important knowledge domains associated with each occupation. This helped to clarify the
need for certain skills, such as math and science. Similar to the tasks, it was deemed informative
to include the importance associated with each knowledge domain. Thus, beginning with Cycle
10, knowledge domains with importance ratings of 3.0 or greater were also provided in the
stimulus materials accompanied by the mean knowledge importance ratings, rounded to the
nearest tenth. Only the first 10 knowledge statements meeting the importance criteria were listed
in an effort to provide meaningful information that is not overwhelming. Appendix C lists the
knowledge domain definitions.

Generalized Work Activities (GWAs)

GWAs are defined as “a set of similar actions that are performed together in many different occupations” (Peterson, et al., 1999). Similar to the skills, GWAs are divided into several taxonomic categories (i.e., Information Input, Mental Processes, Work Output, and Interacting with Others). Incumbents provide both importance and level ratings for each GWA. Given this, several pieces of GWA data were considered for inclusion in the stimulus materials. The first was whether to include data on all GWAs for each occupation or only those that were deemed important to the target occupation. It was determined that listing all GWAs would be excessive and therefore only those important to the occupation (i.e., mean importance rating of 3.0 or greater) were included in the stimulus materials. This judgment was, of course, at the occupational level (i.e., the particular GWA is important to the occupation). However, one also can think about GWAs at the individual skill level (i.e., the particular GWA needs the target skill to be performed successfully). Therefore, the project team decided to only present GWAs that are important to the occupation and require the specific skill to be performed successfully. That is, if the skill is not required to perform a particular GWA, then it would be irrelevant to present information about that GWA for that specific skill. Thus, the stimulus materials for a particular skill included those GWAs (1) found to be important for the occupation (mean ≥ 3.00) and (2) that require the targeted skill to be performed successfully. The process of identifying the skills needed to successfully perform each GWA involved collecting skill/GWA linkage data and is described later in this report.

As noted above, in addition to providing importance ratings, incumbents were asked to
rate each GWA in terms of level. Initially, the inclusion of both importance and level data was
deemed appropriate, and potentially informative. Pilot results and feedback from pilot
occupational analysts, however, indicated that the level ratings significantly increased the
complexity of the cognitive task associated with making skill importance and level ratings.
Occupational analysts also found it difficult to interpret level ratings since the benchmarks were
not provided in the stimulus materials. For these reasons, and because level rating information
can be highly correlated with importance information, a decision was made to drop the level
ratings from the stimulus materials.

In summary, the GWA data that were presented in the stimulus materials are those
GWAs that were (1) important to the occupation (mean ≥ 3.00) and (2) require the targeted skill
for successful performance. Importance means were rounded to the nearest whole number to
simplify the cognitive process. The GWAs are presented in the order that they occur on the
incumbent questionnaire and were organized by their highest order taxonomic categories
([http://www.onetcenter.org/content.html](http://www.onetcenter.org/content.html)). Presenting the GWAs in this order communicated information about the meaning of the descriptors, provided a consistent format across skill pages, and facilitated occupational analyst understanding of this information.

## Work Context Descriptors

Work Context descriptors (WCs) are conditions under which job activities must be carried out including physical conditions (e.g., temperature and noise) and social-psychological conditions (e.g., time pressure and dependence on others) that have the potential to influence how people perform certain work activities (Peterson, et al., 1999). Incumbents rated each WC on a five-point scale, although the benchmarks vary depending on the nature of the statement (e.g., frequency, importance, amount of responsibility, time spent).

Based on a review of the WCs and their associated benchmarks, the most meaningful information for the occupational analyst's rating task would be those WCs that meet a certain threshold on their respective rating scale (e.g., at least 3.0 mean importance). This would avoid presenting WCs that received low ratings, suggesting that they were not sufficiently relevant to the target occupation and therefore, had little or no impact on the occupational analyst's judgment. In addition, to ensure only the most pertinent information is included in the stimulus materials, only the WCs that require the specific skill in order to perform work in that context should be included in the stimulus material. Like the GWAs, this information was collected by conducting a skill/WC linkage study (described below). Thus, the WC data that were included in the stimulus materials for a particular skill were those that (1) received a mean rating of at least 3.0 for the specific occupation and (2) needed the target skill in order for work to be performed in that context. To facilitate interpretation, the WCs were organized by their scale (e.g., frequency, length of time, and level of responsibility) and the mean ratings were rounded to the nearest whole number.

## Skill/GWA and WC Linkages

As mentioned above, a precursor to the development of the stimulus materials was the identification of the skills that are linked to each GWA and WC outside the context of a particular occupation (i.e., skills required to perform each GWA successfully or to conduct work in each work context). Then, this information could be used to narrow the list of GWAs and WCs presented in the stimulus materials to only those relevant to the target skill.

Linkage data were collected from eight experienced industrial/organizational psychologists. All participants had doctorates in industrial/organizational psychology or a closely related field, and had extensive experience in job analysis and knowledge of O*NET. For each skill and GWA (or WC) combination, the participant made a dichotomous judgment regarding whether the skill was needed to perform each GWA (or perform work in this context). The complete linkage exercise instructions are presented in Appendix D. The results were recorded into a matrix and summarized across all participants.

After each participant made his/her independent linkage judgments, everyone met to review and discuss the results. An a priori decision was made that a majority of the judges (i.e.,
at least five) had to indicate that the skill was linked to a specific GWA/WC for a linkage to be established. If four or fewer judges considered the pair linked, a linkage was not established. All borderline situations (i.e., four judges suggested a linkage), as well as cases that a particular judge wanted to review, were discussed among the judges. Following this discussion, judges had the opportunity to modify their original linkage judgment. The results of this exercise indicated that, on average, each skill was linked to five GWAs (min = 0 and max = 22) and two WCs (min = 0 and max = 10). The final linkage matrix depicting the skills required to perform each GWA and WC is presented in Appendix E.

## Skill Rating Procedures

Given the potential complexity of the skill rating process, a set of standardized steps for reviewing and interpreting the data presented in the stimulus material was established. The detailed instructions for making ratings are presented in Appendix F. A brief overview of the procedures is presented below.

First, occupational analysts reviewed the occupation title, definition, and job zone. Then, they proceeded with the importance and level rating as follows:

### Making Importance Ratings

Step 1. Consider the Construct: Review the title, definition and three *Level* scale anchors tailored to the specific construct

Step 2. Consider the Tasks: Review importance ratings, make preliminary importance rating on skill

Step 3. Consider the Knowledge Domains: Review importance ratings, review/revise preliminary rating

Step 4. Consider GWAs: Review importance ratings, review/revise preliminary rating

Step 5. Consider Work Context: Review mean ratings

Step 6. Document Final Skill Importance Rating

### Making Level Ratings

Step 1. Determine Whether to Provide a Level Rating: If you rated the construct as *Not Important* (i.e., 1), give the construct a *Level* rating of 0 and move on to the next construct. If you rated this construct as at least *Somewhat Important* (i.e., ≥ 2), provide a *Level* rating for the target construct

Step 2. Consider the Level Anchors

Step 3. Consider the Tasks: Make a preliminary level rating on skill

Step 4. Consider the Knowledge Domains: Review importance ratings, review/revise preliminary rating
Step 5. Consider GWAs: Review/revise preliminary level rating

Step 6. Consider Work Context: Review mean ratings

Step 7. Document Your Rating

## Stimulus Material Development

The ultimate goal was to generate stimulus materials that present meaningful data aimed at facilitating the skill rating process, without being burdensome or excessive. Example materials are included in Appendix G. As described above, the information presented in the stimulus materials for each occupation, and each skill within an occupation, is as follows:

* Title, definition, and job zone of the occupation

* Mean importance of core and supplementary tasks for the targeted occupation

* Mean importance of knowledge domains that have a mean importance ≥ 3.0.

* Mean importance of Generalized Work Activities (GWAs) that (1) have a mean importance for the occupation ≥ 3.0, and (2) require the targeted skill to be performed

* Mean rating of Work Context (WC) statements that (1) have a mean ratings for the targeted occupation ≥ 3.0, and (2) require the targeted skill to work in that context

### Occupation Title, Definition, and Job Zone

The occupational titles, eight-digit codes, definitions, and job zone categories were presented on the first page of stimulus materials (for a given occupation) above the tasks. For Cycle 9, the job zone information was initially obtained from O*NET Online. Occupational titles, codes, definitions, and job zone information for subsequent cycles were provided to HumRRO in a Microsoft Excel spreadsheet by the O*NET Center staff.

### Tasks

As described above, all relevant tasks were divided into two categories: core and supplementary, with core tasks listed first, followed by the supplementary tasks. Accompanying each task was the mean importance rating, rounded to the nearest tenth. The task data required to include in the stimulus material for each occupation was provided to HumRRO in a Microsoft Excel spreadsheet. Data were then inserted in a Microsoft Access database that was created to house all of the stimulus material information. The access database allowed for the information to be merged and printed in a specified format.

### Knowledge

Important knowledge domains (≥ 3.0) were listed below the task statements on the first page of the stimulus materials. Accompanying each knowledge was the mean importance rating, rounded to the nearest tenth. The knowledge data required to include in the stimulus material for
each occupation was provided to HumRRO in a Microsoft Excel spreadsheet. Data were then inserted in the Microsoft Access database that was created to house all of the stimulus material information.

## Generalized Work Activities

Following the occupational data on the first page of the stimulus materials, information specific to each of the 35 skills for the targeted occupation presented. First, the skill title, definition, and level scale, with its job-related activities anchors, were presented. Then, the GWAs that required the targeted skill to be performed successfully and had a mean importance rating for the particular occupation of at least 3.0 were listed. The mean importance ratings, rounded to the nearest whole number, were listed next to the corresponding GWA.

GWA data were delivered to HumRRO in a Microsoft Excel spreadsheet. Once formatted, the data from the spreadsheet were imported into the access database. This step was completed after the database was already populated with the occupation and task information.

## Work Context Statements

The WC statements were presented below the GWA data on each of the 35 skill sheets for each occupation. The WCs that required the targeted skill to work in that context and had mean ratings for the targeted occupation ≥ 3.0 were listed. The mean importance ratings, rounded to the nearest whole number, were listed next to the corresponding WCs.

The same procedures used to manage the GWA data used for the WC data. The WC data were delivered to HumRRO in a Microsoft Excel spreadsheet. Once formatted, the data from the spreadsheet were imported into the Microsoft Access database. This step was completed after the database was already populated with the occupation and task information.

## Occupational Analysts

Sixteen trained occupational analysts were responsible for rating the importance and level of the 35 skills for each of the O*NET occupations. A minimum of eight raters per occupation was required. This number was based on the number of raters needed to ensure the target level of interrater reliability. The type of reliability of most interest in this situation is the extent to which raters agree about the order of and relative distance between occupations on a particular scale for a particular construct. For example, is there consistency across raters in how they differentiate among occupations on the required level of the skill *Critical Thinking*?

Our target level of interrater reliability is that the median *ICC (3, k)* across the construct ratings for a particular domain on a particular scale be .80 or greater (e.g., the median reliability across 35 Skill Level ratings should be at least .80). The value of .80 is judged to be a good rule-of-thumb that has been used previously in the O*NET context (e.g., Clement, Chauvot, Philipp, & Ambrose, 2003; McCloy, Waugh, & Medsker, 1998; Rase & Tognetti-Stuff, 1983). The need for eight raters per occupation was based on the reliability values observed in the prototype O*NET
project (Peterson, et al., 1997) and the work HumRRO performed to generate occupational reinforcement patterns for O*NET (McCloy, et al., 1998).

## Occupational Analyst Criteria

To ensure that the occupational analysts were qualified to complete the rating task, minimum criteria for serving as an analyst were established. An occupational analyst must have:

* At least two years work experience. This work could be full or part time work, but it could not be an internship, summer job, or research assistantship position in school. The work experience requirement was set to ensure that the analysts were highly familiar with a work environment and job responsibilities.

* Completed two years of graduate education in either Industrial/Organizational Psychology, Vocational Psychology, Human Resources (business department), or Industrial Relations.

* Completed courses in both job analysis (or something comparable) and research methods (or something comparable).

The education and course requirement was set to ensure that the occupational analyst had training and experience working with occupational or job analytic terminology and constructs and measurement methodology. For the current effort, all 16 occupational analysts met (and many exceeded) the criteria listed above.

## Occupational Analyst Training

HumRRO project staff used a program to train occupational analysts to make reliable and valid ratings of occupational skills. This program was developed for the ability ratings and was modified to update information for subsequent cycles. After initial training, experienced occupational analysts attended abbreviated training sessions for subsequent rating cycles, which is described in the next section.

Initial occupational analyst training lasted about 12 hours and was conducted over one and a half days. The training followed an analyst training manual that included five modules:

**Module 1: History of O*NET.** Overview of the O*NET database structure and uses; also includes a general review of the rating process along with common rating errors.

**Module 2: Overview of the Stimulus Materials.** Detailed introduction and discussion of the stimulus materials including the associated terminology and concepts.

**Module 3: Making Your Ratings.** Step-by-step description of the rating process and practice making ratings on several of the O*NET skills for a sample occupation.

**Module 4: Recording Your Ratings.** Introduction of the electronic rating form used to enter and submit skill ratings.

**Module 5: Appendices.** Materials related to Modules 1-4.
Each training module incorporated hands-on exercises and quizzes. In addition, a manual for the trainers with instructions for presenting information was developed and provided in each module. Imbedded in the training modules was repeated exposure and discussion of the skill constructs. Also, to facilitate the rating process, occupational analysts were provided detailed instructions for making their ratings (see Appendix F). In general, the rating process involved three main steps:

Step 1. Review the occupational title, definition, job zone, incumbent task ratings and incumbent knowledge ratings and make a preliminary rating of the skill.

Step 2. Review the GWAs and ratings and modify their preliminary rating as necessary.

Step 3. Review the relevant WC descriptors, make any needed adjustments to their rating, and then record a final rating.

After the occupational analysts completed all of the training modules and practice exercises, they were provided with a complete set of stimulus materials and asked to rate an entire set of constructs for an occupation. The instructor provided feedback and guidance as necessary.

## Experienced Occupational Analyst Training

Although unnecessary to review the entire training program with experienced occupational analysts, it was deemed beneficial to conduct abbreviated training sessions at the beginning of each subsequent rating cycle. The abbreviated training included a detailed review of the rating process as well as a targeted review of potentially problematic construct definitions. Occupational analysts were given the opportunity to practice rating constructs for multiple occupations and to discuss their judgments with the group. Feedback from the abbreviated training was positive, as analysts felt the session was useful and assisted in maintaining a process that resulted in reliable and valid ratings.

## Data Collection

### Rating Schedule and Assignments

Sixteen raters were assigned to Rater Group A or B such that each group consisted of eight raters. Ratings were collected for approximately 100 occupations during each cycle. Although the number of occupations varied for different cycles, each of the occupations was randomly assigned to one of approximately 20 sets of occupations, where each set consisted of approximately five occupations. Table 1 depicts the assignment of 100 occupations to the two groups of raters. As can be noted, Rater Group A rated occupation sets 1, 2, 3, 5, 7, 9, 11, 13, 15, 17, and 19, and Rater Group B rated occupation sets 1, 2, 4, 6, 8, 10, 12, 14, 16, 18, and 20. The table also shows that within the first two batches both groups rated the same sets of occupations (i.e., 1 and 2). This allowed for preliminary assessments of the reliability of the ratings to be based on 16 raters for 10 occupations.
<table><caption>**Table 1. General Batch Assignments**</caption><thead><tr><th></th><th>Group A</th><th>Group B</th></tr></thead><tbody><tr><td>Batch 1</td><td>Set 1</td><td>Set 2</td></tr><tr><td>Batch 1-Reverse</td><td>Set 2</td><td>Set 1</td></tr><tr><td>Batch 2</td><td>Set 3</td><td>Set 4</td></tr><tr><td>Batch 3</td><td>Set 5</td><td>Set 6</td></tr><tr><td>Batch 4</td><td>Set 7</td><td>Set 8</td></tr><tr><td>Batch 5</td><td>Set 9</td><td>Set 10</td></tr><tr><td>Batch 6</td><td>Set 11</td><td>Set 12</td></tr><tr><td>Batch 7</td><td>Set 13</td><td>Set 14</td></tr><tr><td>Batch 8</td><td>Set 15</td><td>Set 16</td></tr><tr><td>Batch 9</td><td>Set 17</td><td>Set 18</td></tr><tr><td>Batch 10</td><td>Set 19</td><td>Set 20</td></tr></tbody></table>

## Disseminating Stimulus Materials

As described above, each group of raters received a set of five occupations at a time. Given this, the stimulus materials were produced in batches of 10 occupations (five in each set). Each batch of stimulus materials was generated, distributed, and returned in a series of steps:

Step 1. The stimulus materials were generated and automatically formatted using the access database and sent to the printer.

Step 2. The materials were printed, proofed, packaged, and shipped to the occupational analysts.

Step 3. Occupational analysts returned their completed ratings for the batch of five occupations seven days after the day they were distributed.

The stimulus materials were prepared one to two weeks prior to distribution. For a given cycle, occupational analysts received stimulus materials on Wednesday and returned their ratings electronically on Wednesday of the following week. Occupational analysts were shipped stimulus materials using an express mail service (e.g., FedEx) that is able to track shipments. This helped ensure that the materials arrived on time at the appropriate location.

## Recording the Ratings

The occupational analysts entered their importance and level ratings into an electronic rating form designed in Microsoft Excel. This form contained special features to enhance the accuracy of the data entry process. For example, a warning message would appear if the analyst
accidentally entered a number that was out of range. Each week, the occupational analysts returned this electronic rating form to HumRRO staff via email by the scheduled due date.

## Rating Analysis

Each batch of ratings was compiled and analyzed to provide feedback to occupational analysts. This feedback informed analysts of any rating tendencies that they should note. Also, skills with large values for the standard error of the mean (SE<sub>M</sub>) were identified so that analysts could revisit their ratings. At the end of the cycle, a number of reliability statistics were computed along with additional information described in the cycle report.

### Batch Analysis

First, ratings were reviewed for errors (e.g., missing ratings). Forms submitted with errors were returned to the occupational analysts for correction. Then, the ratings were examined for common rating errors (e.g., leniency, severity). Occupational analysts who showed any sort of rating bias or tendency across multiple occupations received feedback regarding the relative rating error. For example, some raters consistently provided higher ratings than others across numerous occupations. This information was shared with the particular raters and they were instructed to keep this in mind when making future ratings.

Then, ratings were analyzed for interrater agreement. These indices were computed to examine the level of absolute agreement among the occupational analysts in ratings within a construct, regardless of how they rank ordered the relative importance or level of the abilities for a particular occupation. For example, these indices were calculated to look at the extent to which eight raters provided the same rating regarding the level of the Skill *Critical Thinking* required to perform a particular occupation. To look at the agreement, we calculated the standard deviation (SD<sub>x</sub>) of ratings across occupational analysts for a given construct and scale for each occupation and the standard error of the mean (SE<sub>M</sub>) of these ratings.

If the SE<sub>M</sub> of the importance ratings for a given construct within an occupation was large (i.e., >.51), it was deemed to have insufficient agreement across raters. An SE<sub>M</sub> greater than .51 means that the upper and lower bounds of the confidence interval are more than one scale point away from the observed mean. For these constructs, all raters in the rating group were asked to reconsider their ratings. Although not required to change their initial ratings, occupational analysts were encouraged to provide their final ratings after reconsideration, along with a brief rationale in support of the new ratings, to all other raters in the group.

### Cycle Analysis

After all of the batches were analyzed, the rating data were compiled into one database for analyses. In addition to computing interrater agreement on the combined cycle data (as described above), two different types of interrater reliability were also calculated.

**Interrater Reliability: Across Constructs Within Occupations.** These indices were computed to look at the consistency across constructs within occupations. This type of reliability
explains the extent to which raters agree about the order of and relative distance between constructs on a particular scale *within* a particular domain and occupation. For example, these indices were computed to determine if there was consistency across raters in terms how they rated the relative importance of the 35 skill constructs to performance in a particular occupation. To look at this type of reliability, we calculated Shrout and Fleiss' (1979) *ICC* (3, *k*) for each occupation on each scale. *ICC*(3, *k*) is an intraclass correlation. Our target level of interrater reliability for this coefficient was that the median *ICC*(3, *k*) across occupations for each domain and scale be .80 or greater (e.g., the median reliability across occupations for Skill Level ratings should be at least .80). The value of .80 is judged to be a good rule-of-thumb for this type of reliability and has been used for previous O*NET analyses (e.g., Clement, Chauvot, Philipp, & Ambrose, 2003; McCloy, Waugh, & Medsker, 1998; Rase & Tognetti-Stuff, 1983).

***Interrater Reliability: Across Occupations Within Constructs.*** These indices were calculated to measure the consistency across occupations within constructs. This type of reliability determines the extent to which raters agree about the order of and relative distance between occupations on a particular scale for particular construct. For example, is there consistency across raters in how they differentiate among occupations on the required level of the Skill *Critical Thinking*? In this circumstance, we computed Shrout and Fleiss' (1979) *ICC*(3, *k*) for each construct on each scale (instead of for each occupation on each scale as described above). Therefore, each of the 35 Skill Importance Scale ratings will have a reliability value. Our target level of interrater reliability for this coefficient is that the median *ICC*(3, *k*) across the construct ratings for a particular domain on a particular scale be .80 or greater (e.g., the median reliability across 35 Skill Level ratings should be at least .80). As with the previously discussed *ICC*(3, *k*), the value of .80 is judged to be a good rule-of-thumb.

This type of reliability could not be used to evaluate raters during the rating process because it cannot be calculated until a reasonable number of occupations have been rated by a given group of analysts. Also, it is important to note that this reliability is dependent on the sample of occupations being rated. That is, all else being equal, the *ICC*(3, *k*) based on ratings of a sample of homogeneous occupations will be lower than the *ICC*(3, *k*) based on ratings of a sample of heterogeneous occupations. It is important to keep this point in mind when interpreting the reliability results.

After completing all analyses, a final summary dataset of the Cycle ratings was prepared in Microsoft Excel and delivered to the O*NET Center. Reports describing additional description of skills analysis and results are available at http://www.onetcenter.org/resData.html#waves.
## Summary

The process of collecting occupational skill ratings from analysts involved designing, preparing, and distributing stimulus materials containing data associated with the target occupation, and collecting and analyzing the analyst data. The stimulus materials included the following pieces of information:

* Title, definition, and job zone of the occupation

* Tasks and mean importance rating for the targeted occupation

* Knowledge domains and the mean importance rating for the targeted occupation

* Generalized Work Activities (GWAs) to which the target skill are linked and mean importance for the linked GWAs with means ≥ 3.0

* Work Context (WC) statements to which the target skill are linked and the mean rating for the linked WC statements with means ≥ 3.0

Trained occupational analysts followed standardized procedures to review the information and make skill ratings. Occupational analysts' performance and ratings were monitored and evaluated throughout the process and, if necessary, remedial training and guidance was provided. Final importance and level ratings for each occupation were saved in an Excel spreadsheet and delivered to the O*NET Center.
Clement, L., Chauvot, J., Philipp, R., & Ambrose, R. (2003). A method for developing rubrics for research purposes. In N. A. Pateman, B. J. Dougherty, & J. T. Zilliox (Eds.), *Proceedings of the 2003 joint meeting of PME and PMENA* (Vol. 2, pp. 221–227). Honolulu: CRDG, College of Education, University of Hawaii.

Donsbach, J., Tsacoumis, S., Sager, C., & Updegraff, J. (2003). *O*\*NET analyst occupational abilities ratings: *Procedures* (DFR-03-22). Alexandria, VA: Human Resources Research Organization.

Harvey, R. J. (1991). Job analysis. In M. D. Dunnette & L. M. Hough (Eds.), *Handbook of industrial and organizational psychology* (2nd ed., pp. 71-163). Palo Alto, CA: Consulting Psychologists Press.

McCloy, R., Waugh, G., & Medsker, G. (1998). *Determining the occupational reinforcer patterns for O*\*NET occupational units. Alexandria, VA: Human Resources Research Organization.

Morgeson, F. P., & Campion, M. A. (1997). Social and cognitive sources of potential inaccuracy in job analysis. *Journal of Applied Psychology, 82*, 627-655.

Morgeson, F. P., Delaney-Klinger, K., Mayfield, M. S., Ferrara, P., & Campion, M. A. (2004). Self-presentation processes in job analysis: A field experiment investigating inflation in abilities, tasks, and competencies. *Journal of Applied Psychology, 89*, 674-686.

National Center for O*NET Development (2008). *Procedures for O*\*NET Job Zone Assignment*. Raleigh, NC: National Center for O*NET Development.

Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., Fleishman, E. A. (1995). *Development of prototype Occupational Information Network (O*\*NET) content model*. (Vols. 1-2). Salt Lake City: Utah Department of Workforce Services.

Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., Fleishman, E. A., & Levin, K. Y. (eds.). (1997). *O*\*NET final technical report. Salk Lake City, UT: Utah Department of Workforce Services through a contract with American Institutes for Research.

Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., Fleishman, E. A. (1999). *An occupational information system for the 21st century: The development of O*\*NET*. Washington, DC: American Psychological Association.

Rase, C.W. & Tognetti-Stuff, R. K. (1983). Reliability of the Auditing Process at the University of Montana's Physical Therapy Department. *Physical Therapy, 64*(7), 1088-1090.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. *Psychological Bulletin, 86*, 420-428.
Tsacoumis, S. (2007). The feasibility of using O*NET to study skill changes. Presented at the Workshop on Research Evidence Related to Future Skill Demands organized by the National Academies Center for Education, Washington, D.C.

U.S. Department of Labor (1991). Dictionary of occupational titles. (Rev. 4th ed.). Washington, DC: U.S. Government Printing Office.
Appendix A – Skill Definitions by Category

Skills are proficiencies that are developed through training or experience. The 35 skills in the O*NET database (italicized) are divided into basic skills and cross-functional skills. Basic skills, such as reading, facilitate the acquisition of new knowledge. Cross-functional skills, such as problem solving, extend across several domains of activities. Under these two broad headers, the skills are grouped into smaller categories.

**Basic Skills:** Developed capacities that facilitate learning or the more rapid acquisition of knowledge.

❖ Content — Background structures needed to work with and acquire more specific skills in a variety of different domains

> *Reading Comprehension* — Understanding written sentences and paragraphs in work related documents.

> *Active Listening* — Giving full attention to what other people are saying, taking time to understand the points being made, asking questions as appropriate, and not interrupting at inappropriate times.

> *Writing* — Communicating effectively in writing as appropriate for the needs of the audience.

> *Speaking* — Talking to others to convey information effectively.

> *Mathematics* — Using mathematics to solve problems.

> *Science* — Using scientific rules and methods to solve problems.

❖ Process — Procedures that contribute to the more rapid acquisition of knowledge and skill across a variety of domains

> *Critical Thinking* — Using logic and reasoning to identify the strengths and weaknesses of alternative solutions, conclusions or approaches to problems.

> *Active Learning* — Understanding the implications of new information for both current and future problem-solving and decision-making.

> *Learning Strategies* — Selecting and using training/instructional methods and procedures appropriate for the situation when learning or teaching new things.

> *Monitoring* — Monitoring/Assessing performance of yourself, other individuals, or organizations to make improvements or take corrective action.
**Cross-Functional Skills:** Developed capacities that facilitate performance of activities that occur across jobs.

* **Social Skills** — Developed capacities used to work with people to achieve goals

* **Social Perceptiveness** — Being aware of others' reactions and understanding why they react as they do.

* **Coordination** — Adjusting actions in relation to others' actions.

* **Persuasion** — Persuading others to change their minds or behavior.

* **Negotiation** — Bringing others together and trying to reconcile differences.

* **Instructing** — Teaching others how to do something.

* **Service Orientation** — Actively looking for ways to help people.

* **Complex Problem Solving Skills** — Developed capacities used to solve novel, ill-defined problems in complex, real-world settings

* **Complex Problem Solving** — Identifying complex problems and reviewing related information to develop and evaluate options and implement solutions.

* **Technical Skills** — Developed capacities used to design, set-up, operate, and correct malfunctions involving application of machines or technological systems

* **Operations Analysis** — Analyzing needs and product requirements to create a design.

* **Technology Design** — Generating or adapting equipment and technology to serve user needs.

* **Equipment Selection** — Determining the kind of tools and equipment needed to do a job.

* **Installation** — Installing equipment, machines, wiring, or programs to meet specifications.

* **Programming** — Writing computer programs for various purposes.

* **Operation Monitoring** — Watching gauges, dials, or other indicators to make sure a machine is working properly.

* **Operation and Control** — Controlling operations of equipment or systems.

* **Equipment Maintenance** — Performing routine maintenance on equipment and determining when and what kind of maintenance is needed.

* **Troubleshooting** — Determining causes of operating errors and deciding what to do about it.
* Repairing — Repairing machines or systems using the needed tools.

* Quality Control Analysis — Conducting tests and inspections of products, services, or processes to evaluate quality or performance.

* Systems Skills — Developed capacities used to understand, monitor, and improve sociotechnical systems

* Judgment and Decision Making — Considering the relative costs and benefits of potential actions to choose the most appropriate one.

* Systems Analysis — Determining how a system should work and how changes in conditions, operations, and the environment will affect outcomes.

* Systems Evaluation — Identifying measures or indicators of system performance and the actions needed to improve or correct performance, relative to the goals of the system.

* Resource Management Skills — Developed capacities used to allocate resources efficiently

* Time Management — Managing one's own time and the time of others.

* Management of Financial Resources — Determining how money will be spent to get the work done, and accounting for these expenditures.

* Management of Material Resources — Obtaining and seeing to the appropriate use of equipment, facilities, and materials needed to do certain work.

* Management of Personnel Resources — Motivating, developing, and directing people as they work, identifying the best people for the job.
# Appendix B - Job Zone Descriptions

A Job Zone is a group of occupations that are similar in

* how much education people need to do the work,

* how much related experience people need to do the work, and

* how much on-the-job training people need to do the work.

## Job Zone One: Little or No Preparation Needed

<table><tr><td>Experience:</td><td>No previous work-related skill, knowledge, or experience is needed for these occupations. For example, a person can become a cashier even if he/she has never worked before.</td></tr><tr><td>Job Training:</td><td>Employees in these occupations need anywhere from a few days to a few months of training. Usually, an experienced worker could show you how to do the job.</td></tr><tr><td>Examples:</td><td>These occupations involve following instructions and helping others. Examples include taxi drivers, amusement and recreation attendants, counter and rental clerks, cashiers, and waiters/waitresses.</td></tr><tr><td>Education:</td><td>These occupations may require a high school diploma or GED certificate. Some may require a formal training course to obtain a license.</td></tr></table>

## Job Zone Two: Some Preparation Needed

<table><tr><td>Experience:</td><td>Some previous work-related skill, knowledge, or experience may be helpful in these occupations, but usually is not needed. For example, a teller might benefit from experience working directly with the public, but an inexperienced person could still learn to be a teller with little difficulty.</td></tr><tr><td>Job Training:</td><td>Employees in these occupations need anywhere from a few months to one year of working with experienced employees.</td></tr><tr><td>Examples:</td><td>These occupations often involve using your knowledge and skills to help others. Examples include sheet metal workers, forest fire fighters, customer service representatives, pharmacy technicians, salespersons (retail), and tellers.</td></tr><tr><td>Education:</td><td>These occupations usually require a high school diploma and may require some vocational training or job-related course work. In some cases, an associate's or bachelor's degree could be needed.</td></tr></table>
## Job Zone Three: Medium Preparation Needed

<table><tr><td>Experience:</td><td>Previous work-related skill, knowledge, or experience is required for these occupations. For example, an electrician must have completed three or four years of apprenticeship or several years of vocational training, and often must have passed a licensing exam, in order to perform the job.</td></tr><tr><td>Job Training:</td><td>Employees in these occupations usually need one or two years of training involving both on-the-job experience and informal training with experienced workers.</td></tr><tr><td>Examples:</td><td>These occupations usually involve using communication and organizational skills to coordinate, supervise, manage, or train others to accomplish goals. Examples include funeral directors, electricians, forest and conservation technicians, legal secretaries, interviewers, and insurance sales agents.</td></tr><tr><td>Education:</td><td>Most occupations in this zone require training in vocational schools, related on-the-job experience, or an associate's degree. Some may require a bachelor's degree.</td></tr></table>

## Job Zone Four: Considerable Preparation Needed

<table><tr><td>Experience:</td><td>A minimum of two to four years of work-related skill, knowledge, or experience is needed for these occupations. For example, an accountant must complete four years of college and work for several years in accounting to be considered qualified.</td></tr><tr><td>Job Training:</td><td>Employees in these occupations usually need several years of work related experience, on-the-job training, and/or vocational training.</td></tr><tr><td>Examples:</td><td>Many of these occupations involve coordinating, supervising, managing, or training others. Examples include accountants, human resource managers, computer programmers, teachers, chemists, and police detectives.</td></tr><tr><td>Education:</td><td>Most of these occupations require a four-year bachelor's degree, but some do not.</td></tr></table>

## Job Zone Five: Extensive Preparation Needed

<table><tr><td>Experience:</td><td>Extensive skill, knowledge, and experience are needed for these occupations. Many require more than five years of experience. For example, surgeons must complete four years of college and an additional five to seven years of specialized medical training to be able to do their job.</td></tr><tr><td>Job Training:</td><td>Employees may need some on-the-job training, but most of these occupations assume that the person will already have the required skills, knowledge, work-related experience, and/or training.</td></tr><tr><td>Examples:</td><td>These occupations often involve coordinating, training, supervising, or managing the activities of others to accomplish goals. Very advanced communication and organizational skills are required. Examples include librarians, lawyers, aerospace engineers, physicists, school psychologists, and surgeons.</td></tr><tr><td>Education:</td><td>A bachelor's degree is the minimum formal education required for these occupations. However, many also require graduate school. For example, they may require a master's degree, and some require a Ph.D., M.D., or J.D. (law degree).</td></tr></table>
Appendix C – Knowledge Domain Definitions

*   **Business and Management** — Knowledge of principles and facts related to business administration and accounting, human and material resource management in organizations, sales and marketing, economics, and office information and organizing systems

o *Administration and Management* — Knowledge of business and management principles involved in strategic planning, resource allocation, human resources modeling, leadership technique, production methods, and coordination of people and resources.

o *Clerical* — Knowledge of administrative and clerical procedures and systems such as word processing, managing files and records, stenography and transcription, designing forms, and other office procedures and terminology.

o *Economics and Accounting* — Knowledge of economic and accounting principles and practices, the financial markets, banking and the analysis and reporting of financial data.

o *Sales and Marketing* — Knowledge of principles and methods for showing, promoting, and selling products or services. This includes marketing strategy and tactics, product demonstration, sales techniques, and sales control systems.

o *Customer and Personal Service* — Knowledge of principles and processes for providing customer and personal services. This includes customer needs assessment, meeting quality standards for services, and evaluation of customer satisfaction.

o *Personnel and Human Resources* — Knowledge of principles and procedures for personnel recruitment, selection, training, compensation and benefits, labor relations and negotiation, and personnel information systems.

*   **Manufacturing and Production** — Knowledge of principles and facts related to the production, processing, storage, and distribution of manufactured and agricultural goods

o *Production and Processing* — Knowledge of raw materials, production processes, quality control, costs, and other techniques for maximizing the effective manufacture and distribution of goods.

o *Food Production* — Knowledge of techniques and equipment for planting, growing, and harvesting food products (both plant and animal) for consumption, including storage/handling techniques.

*   **Engineering and Technology** — Knowledge of the design, development, and application of technology for specific purposes.

o *Computers and Electronics* — Knowledge of circuit boards, processors, chips, electronic equipment, and computer hardware and software, including applications and programming.

o *Engineering and Technology* — Knowledge of the practical application of engineering science and technology. This includes applying principles, techniques,
procedures, and equipment to the design and production of various goods and services.

o *Design* — Knowledge of design techniques, tools, and principles involved in production of precision technical plans, blueprints, drawings, and models.

o *Building and Construction* — Knowledge of materials, methods, and the tools involved in the construction or repair of houses, buildings, or other structures such as highways and roads.

o *Mechanical* — Knowledge of machines and tools, including their designs, uses, repair, and maintenance.

* Mathematics and Science — Knowledge of the history, theories, methods, and applications of the physical, biological, social, mathematical, and geography

o *Mathematics* — Knowledge of arithmetic, algebra, geometry, calculus, statistics, and their applications.

o *Physics* — Knowledge and prediction of physical principles, laws, their interrelationships, and applications to understanding fluid, material, and atmospheric dynamics, and mechanical, electrical, atomic and sub- atomic structures and processes.

o *Chemistry* — Knowledge of the chemical composition, structure, and properties of substances and of the chemical processes and transformations that they undergo. This includes uses of chemicals and their interactions, danger signs, production techniques, and disposal methods.

o *Biology* — Knowledge of plant and animal organisms, their tissues, cells, functions, interdependencies, and interactions with each other and the environment.

o *Psychology* — Knowledge of human behavior and performance; individual differences in ability, personality, and interests; learning and motivation; psychological research methods; and the assessment and treatment of behavioral and affective disorders.

o *Sociology and Anthropology* — Knowledge of group behavior and dynamics, societal trends and influences, human migrations, ethnicity, cultures and their history and origins.

o *Geography* — Knowledge of principles and methods for describing the features of land, sea, and air masses, including their physical characteristics, locations, interrelationships, and distribution of plant, animal, and human life.

* Health Services — Knowledge of principles and facts regarding diagnosing, curing, and preventing disease, and improving and preserving physical and mental health and well-being

o *Medicine and Dentistry* — Knowledge of the information and techniques needed to diagnose and treat human injuries, diseases, and deformities. This includes symptoms, treatment alternatives, drug properties and interactions, and preventive health-care measures.
* *Therapy and Counseling* — Knowledge of principles, methods, and procedures for diagnosis, treatment, and rehabilitation of physical and mental dysfunctions, and for career counseling and guidance.

* **Education and Training** — Knowledge of principles and methods for curriculum and training design, teaching and instruction for individuals and groups, and the measurement of training effects.

* **Arts and Humanities** — Knowledge of facts and principles related to the branches of learning concerned with human thought, language, and the arts.

* *English Language* — Knowledge of the structure and content of the English language including the meaning and spelling of words, rules of composition, and grammar.

* *Foreign Language* — Knowledge of the structure and content of a foreign (non-English) language including the meaning and spelling of words, rules of composition and grammar, and pronunciation.

* *Fine Arts* — Knowledge of the theory and techniques required to compose, produce, and perform works of music, dance, visual arts, drama, and sculpture.

* *History and Archeology* — Knowledge of historical events and their causes, indicators, and effects on civilizations and cultures.

* *Philosophy and Theology* — Knowledge of different philosophical systems and religions. This includes their basic principles, values, ethics, ways of thinking, customs, practices, and their impact on human culture.

* **Law and Public Safety** — Knowledge of regulations and methods for maintaining people and property free from danger, injury, or damage; the rules of public conduct established and enforced by legislation, and the political process establishing such rules.

* *Public Safety and Security* — Knowledge of relevant equipment, policies, procedures, and strategies to promote effective local, state, or national security operations for the protection of people, data, property, and institutions.

* *Law and Government* — Knowledge of laws, legal codes, court procedures, precedents, government regulations, executive orders, agency rules, and the democratic political process.

* **Communications** — Knowledge of the science and art of delivering information

* *Telecommunications* — Knowledge of transmission, broadcasting, switching, control, and operation of telecommunications systems.

* *Communications and Media* — Knowledge of media production, communication, and dissemination techniques and methods. This includes alternative ways to inform and entertain via written, oral, and visual media.

* **Transportation** — Knowledge of principles and methods for moving people or goods by air, rail, sea, or road, including the relative costs and benefits.
### Background

As you know the Occupational Information Network (O*NET) is a comprehensive conceptual framework designed to serve as the foundation for a variety of human resource programs, such as school curriculum development, job placement, and training. The National O*NET Consortium's National Center for O*NET Development is getting ready to collect what we would call job analytic ratings from incumbents for a number of occupations in a number of conceptual domains (e.g., Generalized Work Activities [GWAs], Work Context, Education, Work Styles, Knowledges, etc.). These areas make up what is referred to the O*NET Content Model. While the O*NET Center plans to collect most of its job analytic information from incumbents, it has contracted with HumRRO to develop, pilot-test, and implement a method of collecting Ability and Skill ratings of occupations from occupational analysts.

The idea is that incumbents are in the best position to rate their occupations on characteristics like the activities they perform (i.e., GWAs) and the physical and social context in which their work takes place (i.e., Work Context). Opportunity to observe the job directly is the most salient advantage of the incumbents' perspective. In contrast, occupational analysts might in better position to rate occupations on the underlying capabilities (e.g., Abilities and Skills) relevant to performing the activities required by an occupation.

Relevant occupation data will be provided to facilitate the analyst rating process. Among other things, we plan to show them incumbent ratings on GWAs and Work Context descriptors that are relevant to each ability or skill. How do we determine which GWAs and Work Context descriptors are relevant to each ability and skill? That is where you come in. Your task, along with seven of your colleagues, is to help us determine which Skills and Abilities should be linked to each GWA and Work Context variable. So that, for example, when an analyst is considering incumbent ratings on the GWAs for the purpose of rating an occupation on a particular ability the analyst will only see the GWAs that are likely to be influenced by that ability.

### Materials

1. Linkage Exercise: Participant Instructions (Your reading them right now.)

2. GWAs & Abilities Linkage Workbook

3. GWAs & Skills Linkage Workbook

4. Work Context & Abilities Linkage Workbook

5. Work Context & Skills Linkage Workbook

Each workbook is set up so that you consider one GWA or Work Context variable at a time. For example, the first two pages of the GWAs/Abilities Linkage Workbook presents GWA #1 – *Getting Information* and all of the Abilities with check boxes for you to indicate which Abilities should be linked to this GWA.
## Specific Linkage Instructions

How do you determine whether an ability or skill should be linked to a GWA or Work Context descriptor?

Ask yourself,

* “Is this Ability/Skill needed to perform this GWA?”

* “Is this Ability/Skill needed to perform work in this context?”

If the answer is yes, place a check mark in the box next to the Ability/Skill in your workbook.

If this question is difficult to answer, another way to think about it is,

* “Would an individual be able to perform this GWA or perform in this context if he/she were very low on this ability or skill?”

If the answer is “no,” then the ability or skill is needed to perform the GWA or to perform in that context and you should check the box.

Some things to keep in mind:

* Do not think about the other GWAs or Work Context descriptors when making your linkage judgment. Each judgment should be independent.

* When making the linkages, ask yourself whether an Ability/Skill is needed to perform a GWA or work in a particular context, in general, not in reference to a particular occupation.

* If you think about it carefully and you are still not sure about whether to check the box, force yourself to “check” or “not check” and put a “?” next to the box. You also might want to make a brief note next to the “?” describing your concern. You will be able to refer your worksheets during the focus group that will follow this exercise.

## Next Steps

After you and your colleagues have completed your ratings we will analyze the results of the ratings including an assessment of interrater agreement. During a focus group meeting including you and the other judges we will review these results and finalize the linkages including:

(a) any necessary modifications to the linkages (e.g., six of the eight judges individually linked an ability to a GWA, but on careful examination of the group agrees that the linkage doesn't make theoretical sense) and

(b) reaching consensus on close calls (e.g., four of the eight judges linked the GWA to the skill).
# Appendix E - Generalized Work Activity and Work Context Linkage Results

## Generalized Work Activity Statements

1. Getting Information: Observing, receiving, and otherwise obtaining information from all relevant sources.

2. Identifying Objects, Actions, and Events: Identifying information by categorizing, estimating, recognizing differences or similarities, and detecting changes in circumstances or events.

3. Monitoring Processes, Materials, or Surroundings: Monitoring and reviewing information from materials, events, or the environment, to detect or assess problems.

4. Inspecting Equipment, Structures, or Materials: Inspecting equipment, structures, or materials to identify the cause of errors or other problems or defects.

5. Estimating the Quantifiable Characteristics of Products, Events, or Information: Estimating sizes, distances, and quantities; or determining time, costs, resources, or materials needed to perform a work activity.

6. Judging the Qualities of Objects, Services, or People: Assessing the value, importance, or quality of things or people.

7. Evaluating Information to Determine Compliance with Standards: Using relevant information and individual judgment to determine whether events or processes comply with laws, regulations, or standards.

8. Processing Information: Compiling, coding, categorizing, calculating, tabulating, auditing, or verifying information or data.

9. Analyzing Data or Information: Identifying the underlying principles, reasons, or facts of information by breaking down information or data into separate parts.

10. Making Decisions and Solving Problems: Analyzing information and evaluating results to choose the best solution and solve problems.

11. Thinking Creatively: Developing, designing, or creating new applications, ideas, relationships, systems, or products, including artistic contributions.

12. Updating and Using Relevant Knowledge: Keeping up-to-date technically and applying new knowledge to your job.

13. Developing Objectives and Strategies: Establishing long-range objectives and specifying the strategies and actions to achieve them.

14. Scheduling Work and Activities: Scheduling events, programs, and activities, as well as the work of others.

15. Organizing, Planning, and Prioritizing Work: Developing specific goals and plans to prioritize, organize, and accomplish your work.

16. Performing General Physical Activities: Performing physical activities that require considerable use of your arms and legs and moving your whole body, such as climbing, lifting, balancing, walking, stooping, and handling of materials.

17. Handling and Moving Objects: Using hands and arms in handling, installing, positioning, and moving materials, and manipulating things.

18. Controlling Machines and Processes: Using either control mechanisms or direct physical activity to operate machines or processes (not including computers or vehicles).
19. Working with Computers: Using computers and computer systems (including hardware and software) to program, write software, set up functions, enter data, or process information.

20. Operating Vehicles, Mechanized Devices, or Equipment: Running, maneuvering, navigating, or driving vehicles or mechanized equipment, such as forklifts, passenger vehicles, aircraft, or water craft.

21. Drafting, Laying Out, and Specifying Technical Devices, Parts, and Equipment: Providing documentation, detailed instructions, drawings, or specifications to tell others about how devices, parts, equipment, or structures are to be fabricated, constructed, assembled, modified, maintained, or used.

22. Repairing and Maintaining Mechanical Equipment: Servicing, repairing, adjusting, and testing machines, devices, moving parts, and equipment that operate primarily on the basis of mechanical (not electronic) principles.

23. Repairing and Maintaining Electronic Equipment: Servicing, repairing, calibrating, regulating, fine-tuning, or testing machines, devices, and equipment that operate primarily on the basis of electrical or electronic (not mechanical) principles.

24. Documenting/Recording Information: Entering, transcribing, recording, storing, of maintaining information in written or electronic/magnetic form.

25. Interpreting the Meaning of Information for Others: Translating or explaining what information means and how it can be used.

26. Communicating with Supervisors, Peers, or Subordinates: Providing information to supervisors, coworkers, and subordinates by telephone, in written form, e-mail, or in person.

27. Communicating with People Outside the Organization: Communicating with people outside the organization, representing the organization to customers, the public, government, and other external sources. The information can be exchanged in person, in writing, or by telephone or e-mail.

28. Establishing and Maintaining Interpersonal Relationships: Developing constructive and cooperative working relationships with others, and maintaining them over time.

29. Assisting and Caring for Others: Providing personal assistance, medical attention, emotional support, or other personal care to others such as coworkers, customers, or patients.

30. Selling or Influencing Others: Convincing others to buy merchandise/goods or to otherwise change their minds or actions.

31. Resolving Conflicts and Negotiating with Others: Handling complaints, settling disputes, and resolving grievances and conflicts, or otherwise negotiating with others.

32. Performing for or Working Directly with the Public: Performing for people or dealing directly with the public. This includes serving customers in restaurants and stores, and receiving clients or guests.

33. Coordinating the Work and Activities of Others: Performing for people or dealing directly with the public. This includes serving customers in restaurants and stores, and receiving clients or guests.

34. Developing and Building Teams: Encouraging and building mutual trust, respect, and cooperation among team members.

35. Training and Teaching Others: Identifying the educational needs of others, developing formal educational or training programs or classes, and teaching or instructing others.

36. Guiding, Directing, and Motivating Subordinates: Providing guidance and direction to subordinates, including setting performance standards and monitoring performance.
37. Coaching and Developing Others: Identifying the developmental needs of others and coaching, mentoring, or otherwise helping others to improve their knowledge or skills.

38. Providing Consultation and Advice to Others: Providing guidance and expert advice to management or other groups on technical, systems-, or process-related topics.

39. Performing Administrative Activities: Performing day-to-day administrative tasks such as maintaining information files and processing paperwork.

40. Staffing Organizational Units: Recruiting, interviewing, selecting, hiring, and promoting employees in an organization.

41. Monitoring and Controlling Resources: Monitoring and controlling resources and overseeing the spending of money.

## Work Context Statements

1. Having face-to-face discussions with individuals and within teams

2. Speaking in public

3. Having telephone conversations

4. Using Electronic mail

5. Writing letters and memos

6. Having contact with others (by telephone, face-to-face, or otherwise)

7. Working with or contributing to a work group or team

8. Dealing with external customers (as in retail sales) or the public in general (as in police work)

9. Coordinating or leading others in accomplishing work activities

10. Being responsible for the health and safety of other workers

11. Being responsible for work outcomes and results of other workers

12. Being in conflict situations

13. Dealing with unpleasant, angry, or discourteous people

14. Dealing with violent or physically aggressive people

15. Working indoors in an environmentally controlled environment (like a warehouse with air conditioning)

16. Working in an environment that is not environmentally controlled (like a warehouse without air conditioning)

17. Working outdoors, exposed to all weather conditions

18. Working outdoors, under cover (like in an open shed)

19. Working in an open vehicle or operating equipment (like a tractor)

20. Working in a closed vehicle or operating enclosed equipment (like a car)

21. Being physically close to other people

22. Being exposed to sounds and noise levels that are distracting and uncomfortable

23. Being exposed to very hot (above 90° F) or very cold (under 32° F) temperatures

24. Being exposed to extremely bright or inadequate lighting conditions

25. Being exposed to contaminants (such as pollutants, gases, dust or odors)

26. Being exposed to cramped work space that requires getting into awkward positions

27. Being exposed to whole body vibration (like operating a jackhammer or earth moving equipment)

28. Being exposed to radiation
29. Being exposed to diseases or infection (This can happen with workers in patient care, some laboratory work, sanitation control, etc.)

30. Being exposed to high places (This can happen for workers who work on poles, scaffolding, catwalks, or ladders longer than 8 feet in length.)

31. Being exposed to hazardous conditions (This can happen when working with high voltage electricity, flammable material, explosives, or chemicals. Do not include working with hazardous equipment.)

32. Being exposed to hazardous equipment (This includes working with saws, close to machinery with exposed moving parts, or working near vehicular traffic, but not including driving a vehicle.)

33. Being exposed to minor burns, cuts, bites, or stings

34. Sitting

35. Standing

36. Climbing ladders, scaffolds, poles, etc.

37. Walking or running

38. Kneeling, crouching, stooping, or crawling

39. Keeping or regaining balance

40. Using hands to handle, control, or feel objects, tools, or controls

41. Bending or twisting body

42. Making repetitive motions

43. Wearing common protective or safety equipment such as safety shoes, glasses, gloves, hearing protection, hard hats, or life jacket

44. Wearing specialized protective or safety equipment such as breathing apparatus, safety harness, full protection suits, or radiation protection

45. Making a serious mistake (one you can't easily correct)

46. Making decisions that affect other people or the image or reputation or financial resources of employer

47. DELETE

48. Being free to make decisions without supervision

49. Performing automated work

50. Being very exact or highly accurate

51. Performing continuous, repetitious physical activities (like key entry) or mental activities (like checking entries in a ledger)

52. Being free to determine the tasks, priorities, or goals

53. Being in a competitive environment

54. Meeting strict deadlines

55. Keeping a pace set by machinery or equipment

56A. Keeping a regular work schedule (established routine, set schedule)

56B. Keeping an irregular work schedule (changes with weather conditions, production demands, or contract duration)

56C. Keeping a seasonal work schedule (only during certain times of the year)

57A. Working less than 40 hours in a typical week

57B. Working 40 hours in a typical week

57C. Working more than 40 hours in a typical week
GWAs/Skills

<figure><img src="image_2.png" /></figure>

= Linked by consensus

<figure><img src="image_3.png" /></figure>

= Linked by >4 raters pre-consensus

<figure><img src="image_4.png" /></figure>

= De-linked by consensus

<table><thead><tr><th>GWA</th><th>oral comp</th><th>writ comp</th><th>oral exp</th><th>writ exp</th><th>flu ideas</th><th>originality</th><th>prob sens</th><th>ded reas</th><th>ind reas</th><th>info order</th><th>cat flex</th><th>math reas</th><th>num facil</th><th>memoriz</th><th>speed clos</th><th>flex clos</th><th>perc sp</th><th>spat orient</th><th>visual</th></tr></thead><tbody><tr><td>1</td><td>8</td><td>8</td><td>X</td><td>4</td><td>2</td><td>1</td><td>1</td><td>3</td><td>X</td><td>3</td><td>3</td><td>1</td><td>1</td><td>3</td><td>4</td><td>6</td><td>4</td><td>1</td><td>1</td></tr><tr><td>2</td><td>5</td><td>5</td><td></td><td></td><td>2</td><td>2</td><td>5</td><td>5</td><td>6</td><td>7</td><td>8</td><td>2</td><td>1</td><td>2</td><td>5</td><td>7</td><td>7</td><td>3</td><td>X</td></tr><tr><td>3</td><td>6</td><td>7</td><td></td><td></td><td>2</td><td>1</td><td>8</td><td>6</td><td>5</td><td>X</td><td>6</td><td>2</td><td>2</td><td>3</td><td>3</td><td>6</td><td>5</td><td>1</td><td>3</td></tr><tr><td>4</td><td>1</td><td>1</td><td></td><td></td><td>1</td><td>1</td><td>7</td><td>5</td><td>6</td><td>2</td><td>X</td><td>2</td><td>2</td><td>2</td><td>3</td><td>5</td><td>6</td><td>1</td><td>5</td></tr><tr><td>5</td><td>1</td><td>2</td><td></td><td></td><td></td><td>1</td><td>6</td><td>X</td><td>6</td><td>3</td><td>7</td><td>6</td><td>2</td><td>4</td><td>X</td><td>3</td><td>2</td><td>X</td><td></td></tr><tr><td>6</td><td>5</td><td>5</td><td></td><td></td><td>1</td><td></td><td>3</td><td>5</td><td>6</td><td>2</td><td>5</td><td></td><td></td><td>2</td><td>2</td><td>2</td><td></td><td></td><td>1</td></tr><tr><td>7</td><td>6</td><td>7</td><td></td><td></td><td></td><td></td><td>6</td><td>7</td><td>5</td><td>6</td><td>3</td><td>3</td><td>2</td><td>3</td><td>3</td><td>3</td><td>4</td><td>1</td><td>2</td></tr><tr><td>8</td><td>5</td><td>5</td><td>1</td><td>1</td><td>2</td><td>2</td><td>3</td><td>X</td><td>5</td><td>7</td><td>6</td><td>7</td><td>8</td><td>2</td><td>4</td><td>6</td><td>6</td><td>1</td><td>2</td></tr><tr><td>9</td><td>4</td><td>5</td><td></td><td></td><td>2</td><td>2</td><td>1</td><td>7</td><td>6</td><td>X</td><td>6</td><td>4</td><td>3</td><td>2</td><td>2</td><td>3</td><td>1</td><td>1</td><td>2</td></tr><tr><td>10</td><td>#</td><td>6</td><td></td><td></td><td>5</td><td>5</td><td>X</td><td>7</td><td>8</td><td>X</td><td>X</td><td>3</td><td>2</td><td>1</td><td>4</td><td>X</td><td>2</td><td>1</td><td>3</td></tr><tr><td>11</td><td>2</td><td>3</td><td>1</td><td>1</td><td>6</td><td>8</td><td>3</td><td>3</td><td>5</td><td>4</td><td>7</td><td>2</td><td>2</td><td>1</td><td>3</td><td>4</td><td>1</td><td>1</td><td>6</td></tr><tr><td>12</td><td>7</td><td>8</td><td>2</td><td>2</td><td>3</td><td>3</td><td>2</td><td>7</td><td>5</td><td>3</td><td>2</td><td>2</td><td>1</td><td>3</td><td>2</td><td>2</td><td>1</td><td></td><td>2</td></tr><tr><td>13</td><td>2</td><td>2</td><td>5</td><td>5</td><td>6</td><td>6</td><td>X</td><td>7</td><td>6</td><td>4</td><td>X</td><td>1</td><td>1</td><td>1</td><td>3</td><td>3</td><td>1</td><td></td><td>3</td></tr><tr><td>14</td><td>2</td><td>2</td><td>2</td><td>2</td><td>1</td><td>1</td><td>X</td><td>X</td><td>4</td><td>8</td><td>6</td><td>1</td><td>3</td><td>1</td><td>3</td><td>1</td><td>1</td><td>1</td><td>2</td></tr><tr><td>15</td><td>1</td><td>2</td><td></td><td></td><td>3</td><td>3</td><td>X</td><td>6</td><td>5</td><td>7</td><td>7</td><td>1</td><td>1</td><td>1</td><td>2</td><td>1</td><td>1</td><td></td><td>1</td></tr><tr><td>16</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>17</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>18</td><td>1</td><td>1</td><td></td><td></td><td></td><td>1</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td>2</td><td>2</td><td>3</td><td>2</td><td>3</td></tr><tr><td>19</td><td>1</td><td>7</td><td></td><td></td><td>4</td><td>X</td><td>5</td><td>5</td><td>8</td><td>4</td><td>7</td><td>5</td><td>7</td><td>3</td><td>5</td><td>3</td><td>3</td><td>2</td><td>1</td><td>2</td></tr><tr><td>20</td><td>3</td><td>4</td><td>1</td><td></td><td></td><td>X</td><td>3</td><td>1</td><td>1</td><td>1</td><td></td><td></td><td></td><td>1</td><td>3</td><td>3</td><td>1</td><td>X</td><td>6</td><td>3</td></tr><tr><td>21</td><td>1</td><td>4</td><td>5</td><td>7</td><td>1</td><td>1</td><td>2</td><td>5</td><td>4</td><td>6</td><td>4</td><td>2</td><td>1</td><td>2</td><td>3</td><td>2</td><td>1</td><td>1</td><td>6</td><td></td></tr><tr><td>22</td><td></td><td>2</td><td></td><td></td><td>2</td><td>3</td><td>6</td><td>8</td><td>6</td><td>5</td><td>2</td><td>1</td><td>2</td><td>X</td><td>X</td><td>3</td><td>2</td><td>2</td><td>6</td><td></td></tr><tr><td>23</td><td></td><td>3</td><td></td><td></td><td>2</td><td>3</td><td>6</td><td>8</td><td>6</td><td>5</td><td>2</td><td>3</td><td>3</td><td>X</td><td>4</td><td>3</td><td>2</td><td>1</td><td>5</td><td></td></tr><tr><td>24</td><td>3</td><td>5</td><td>8</td><td></td><td></td><td>1</td><td>1</td><td>1</td><td>5</td><td>4</td><td></td><td></td><td></td><td>1</td><td>1</td><td>2</td><td>3</td><td>2</td><td>1</td></tr><tr><td>25</td><td>8</td><td>8</td><td>8</td><td>2</td><td>2</td><td>2</td><td>5</td><td>6</td><td>X</td><td>5</td><td>2</td><td></td><td></td><td>1</td><td>1</td><td>3</td><td>2</td><td>1</td><td>1</td><td></td></tr><tr><td>26</td><td>6</td><td>5</td><td>8</td><td>8</td><td></td><td>1</td><td>1</td><td>1</td><td>X</td><td>3</td><td>1</td><td>1</td><td>2</td><td>3</td><td>3</td><td>1</td><td>1</td><td></td><td>1</td><td></td></tr><tr><td>27</td><td>8</td><td>8</td><td>8</td><td>8</td><td>1</td><td>1</td><td>2</td><td>1</td><td>1</td><td>X</td><td>3</td><td>1</td><td>1</td><td>2</td><td>3</td><td>1</td><td>1</td><td></td><td></td><td></td></tr><tr><td>28</td><td>8</td><td>4</td><td>8</td><td>4</td><td></td><td>X</td><td>2</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>29</td><td>8</td><td>4</td><td>8</td><td>3</td><td>1</td><td>2</td><td>5</td><td>X</td><td>3</td><td>2</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>30</td><td>6</td><td>3</td><td>8</td><td>6</td><td>6</td><td>7</td><td>3</td><td>6</td><td>5</td><td>5</td><td>X</td><td></td><td></td><td>3</td><td>2</td><td>1</td><td>3</td><td></td><td></td><td></td></tr><tr><td>31</td><td>8</td><td>5</td><td>8</td><td>X</td><td>6</td><td>7</td><td>7</td><td>6</td><td>7</td><td>5</td><td>X</td><td></td><td></td><td>2</td><td>X</td><td>2</td><td>1</td><td></td><td></td><td></td></tr><tr><td>32</td><td>8</td><td>3</td><td>8</td><td>2</td><td>2</td><td>X</td><td>6</td><td>X</td><td>3</td><td>3</td><td>3</td><td></td><td></td><td>1</td><td>3</td><td>3</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>33</td><td>7</td><td>2</td><td>8</td><td>X</td><td>5</td><td>5</td><td>6</td><td>5</td><td>7</td><td>6</td><td>6</td><td></td><td></td><td>1</td><td>1</td><td>2</td><td>2</td><td>1</td><td></td><td>2</td></tr><tr><td>34</td><td>7</td><td>2</td><td>8</td><td>5</td><td>X</td><td>X</td><td>6</td><td>5</td><td>5</td><td>2</td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>35</td><td>7</td><td>6</td><td>8</td><td>6</td><td>X</td><td>6</td><td>6</td><td>7</td><td>7</td><td>6</td><td>5</td><td>1</td><td>1</td><td>X</td><td>3</td><td>2</td><td>1</td><td>1</td><td></td><td></td></tr><tr><td>36</td><td>8</td><td>5</td><td>8</td><td>7</td><td>X</td><td>5</td><td>7</td><td>7</td><td>7</td><td>3</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>37</td><td>8</td><td>5</td><td>8</td><td>7</td><td>X</td><td>5</td><td>7</td><td>7</td><td>7</td><td>7</td><td>3</td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>38</td><td>8</td><td>7</td><td>8</td><td>8</td><td>6</td><td>7</td><td>7</td><td>7</td><td>5</td><td>5</td><td>3</td><td>1</td><td>2</td><td>4</td><td>2</td><td>2</td><td>1</td><td>1</td><td></td><td></td></tr><tr><td>39</td><td>3</td><td>7</td><td>2</td><td>5</td><td></td><td></td><td>3</td><td>5</td><td>2</td><td>7</td><td>6</td><td></td><td>2</td><td>3</td><td>2</td><td>1</td><td>3</td><td>1</td><td>1</td><td></td></tr><tr><td>40</td><td>8</td><td>7</td><td>8</td><td>6</td><td>1</td><td>2</td><td>5</td><td>6</td><td>7</td><td>X</td><td>5</td><td></td><td></td><td>1</td><td>2</td><td>1</td><td>1</td><td></td><td>1</td><td></td></tr><tr><td>41</td><td>6</td><td>7</td><td>3</td><td>4</td><td></td><td>3</td><td>5</td><td>6</td><td>4</td><td>5</td><td>3</td><td>3</td><td>5</td><td>2</td><td>3</td><td>2</td><td>1</td><td></td><td>1</td><td></td></tr></tbody></table>
GWAs/Skills

<figure><img src="image_5.png" /></figure>

= Linked by consensus

<figure><img src="image_6.png" /></figure>

= Linked by >4 raters pre-consensus

<figure><img src="image_7.png" /></figure>

= De-linked by consensus

<table><thead><tr><th>GWA</th><th>selec att</th><th>time share</th><th>arm-hand st</th><th>manual dex</th><th>finger dex</th><th>control pre</th><th>multi-lim</th><th>resp orien</th><th>rate cont</th><th>reac time</th><th>wrist-fing</th><th>speed limb</th><th>static st</th><th>expl str</th><th>dynam st</th><th>trunk st</th><th>stamina</th><th>extent flex</th><th>dynam flex</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>3</td><td></td><td>1</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>2</td><td>3</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>3</td><td>X</td><td>5</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>4</td><td>3</td><td>2</td><td>1</td><td>2</td><td>2</td><td>1</td><td>1</td><td></td><td></td><td></td><td></td><td>1</td><td>2</td><td></td><td>1</td><td>1</td><td>1</td><td>2</td><td>1</td></tr><tr><td>5</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>6</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>7</td><td>4</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>8</td><td>3</td><td>1</td><td></td><td></td><td>1</td><td></td><td></td><td></td><td></td><td></td><td>1</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>9</td><td>X</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>10</td><td>6</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>11</td><td>5</td><td>1</td><td>1</td><td>1</td><td>1</td><td></td><td>1</td><td></td><td></td><td></td><td></td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>12</td><td>3</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>13</td><td>1</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>14</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>15</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>16</td><td></td><td></td><td>5</td><td>4</td><td>2</td><td>2</td><td>5</td><td>2</td><td>3</td><td>2</td><td>1</td><td>3</td><td>6</td><td>4</td><td>6</td><td>6</td><td>5</td><td>6</td><td>3</td><td></td></tr><tr><td>17</td><td></td><td></td><td>7</td><td>7</td><td>6</td><td>3</td><td>7</td><td>2</td><td>4</td><td>1</td><td>4</td><td>2</td><td>X</td><td>3</td><td>3</td><td>2</td><td>2</td><td>3</td><td>3</td><td></td></tr><tr><td>18</td><td>3</td><td>3</td><td>5</td><td>4</td><td>4</td><td>8</td><td>X</td><td>6</td><td>7</td><td>5</td><td>5</td><td>4</td><td>3</td><td>2</td><td>2</td><td>2</td><td>3</td><td>2</td><td>2</td><td></td></tr><tr><td>19</td><td>5</td><td></td><td>1</td><td>2</td><td>5</td><td></td><td></td><td></td><td></td><td></td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>20</td><td>X</td><td>5</td><td>5</td><td>X</td><td>1</td><td>7</td><td>8</td><td>8</td><td>7</td><td>7</td><td>1</td><td>X</td><td></td><td></td><td>1</td><td>1</td><td>1</td><td>2</td><td>2</td><td></td></tr><tr><td>21</td><td>2</td><td></td><td></td><td>1</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>22</td><td>3</td><td>1</td><td>7</td><td>8</td><td>7</td><td>5</td><td>2</td><td></td><td>1</td><td>2</td><td>1</td><td>1</td><td>3</td><td>1</td><td></td><td>2</td><td></td><td>2</td><td></td><td></td></tr><tr><td>23</td><td>3</td><td>1</td><td>5</td><td>8</td><td>7</td><td>4</td><td>2</td><td>1</td><td>2</td><td>2</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>2</td><td></td></tr><tr><td>24</td><td>3</td><td>2</td><td></td><td>3</td><td>7</td><td>2</td><td></td><td></td><td></td><td>1</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td>1</td><td></td><td></td><td></td></tr><tr><td>25</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>26</td><td>1</td><td></td><td></td><td>1</td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>27</td><td>2</td><td>1</td><td></td><td>1</td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>28</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>29</td><td>3</td><td></td><td>1</td><td>1</td><td>1</td><td></td><td>1</td><td></td><td></td><td></td><td>1</td><td></td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>30</td><td>2</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>31</td><td>3</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>32</td><td>5</td><td>X</td><td></td><td></td><td>1</td><td></td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>33</td><td>3</td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>34</td><td>3</td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>35</td><td>5</td><td>4</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>36</td><td>2</td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>37</td><td>1</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>38</td><td>3</td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>39</td><td>3</td><td>2</td><td></td><td>3</td><td>2</td><td></td><td>1</td><td></td><td></td><td></td><td>1</td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>40</td><td>2</td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>41</td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>
GWAs/Skills

<figure><img src="image_8.png" /></figure>

= Linked by consensus

<figure><img src="image_9.png" /></figure>

= Linked by >4 raters pre-consensus

<figure><img src="image_10.png" /></figure>

= De-linked by consensus

<table><thead><tr><th>GWA</th><th>grs bdy coord</th><th>grs bdy equil</th><th>near vision</th><th>far vision</th><th>vis col disc</th><th>night vis</th><th>perip vis</th><th>depth perc</th><th>glare sens</th><th>hear sens</th><th>audit att</th><th>sound local</th><th>speech rec</th><th>speech clar</th></tr></thead><tbody><tr><td>1</td><td></td><td></td><td>7</td><td>5</td><td>4</td><td>2</td><td>2</td><td>3</td><td>2</td><td>3</td><td>3</td><td>2</td><td>8</td><td>3</td></tr><tr><td>2</td><td></td><td></td><td>6</td><td>5</td><td>X</td><td>3</td><td>3</td><td>4</td><td>2</td><td>5</td><td>4</td><td>3</td><td>4</td><td></td></tr><tr><td>3</td><td></td><td></td><td>7</td><td>5</td><td>X</td><td>3</td><td>3</td><td>4</td><td>2</td><td>5</td><td>X</td><td>3</td><td>4</td><td></td></tr><tr><td>4</td><td>1</td><td>1</td><td>8</td><td>5</td><td>X</td><td>2</td><td>3</td><td>5</td><td>2</td><td>6</td><td>5</td><td>3</td><td>1</td><td></td></tr><tr><td>5</td><td></td><td></td><td>6</td><td>5</td><td>2</td><td>2</td><td>2</td><td>X</td><td>1</td><td>1</td><td></td><td></td><td>1</td><td></td></tr><tr><td>6</td><td></td><td></td><td>5</td><td>2</td><td>3</td><td></td><td>1</td><td>1</td><td></td><td>2</td><td></td><td></td><td>3</td><td></td></tr><tr><td>7</td><td></td><td></td><td>5</td><td>2</td><td>3</td><td>1</td><td>2</td><td>3</td><td>2</td><td>2</td><td>2</td><td>1</td><td>4</td><td></td></tr><tr><td>8</td><td></td><td></td><td>X</td><td></td><td>1</td><td></td><td></td><td>1</td><td></td><td></td><td>1</td><td></td><td>2</td><td></td></tr><tr><td>9</td><td></td><td></td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td>1</td><td>1</td><td>2</td><td></td></tr><tr><td>10</td><td></td><td></td><td>3</td><td></td><td>1</td><td></td><td></td><td></td><td></td><td>1</td><td>2</td><td>2</td><td>3</td><td></td></tr><tr><td>11</td><td></td><td></td><td>3</td><td></td><td>2</td><td></td><td>2</td><td>2</td><td></td><td>2</td><td>2</td><td>1</td><td>2</td><td></td></tr><tr><td>12</td><td></td><td></td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>2</td><td></td></tr><tr><td>13</td><td></td><td></td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>14</td><td></td><td></td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td>1</td></tr><tr><td>15</td><td></td><td></td><td></td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td></td></tr><tr><td>16</td><td>8</td><td>8</td><td>2</td><td>1</td><td></td><td></td><td>2</td><td>2</td><td></td><td></td><td></td><td>1</td><td></td><td></td></tr><tr><td>17</td><td>2</td><td>2</td><td>3</td><td>1</td><td>1</td><td></td><td>2</td><td>2</td><td></td><td>1</td><td>1</td><td>1</td><td></td><td></td></tr><tr><td>18</td><td>3</td><td>1</td><td>5</td><td>4</td><td>3</td><td>2</td><td>3</td><td>X</td><td>3</td><td>X</td><td>3</td><td>3</td><td>1</td><td></td></tr><tr><td>19</td><td></td><td></td><td>8</td><td></td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td></td></tr><tr><td>20</td><td>1</td><td></td><td>7</td><td>6</td><td>X</td><td>5</td><td>5</td><td>6</td><td>X</td><td>3</td><td>1</td><td>5</td><td>2</td><td>1</td></tr><tr><td>21</td><td></td><td></td><td>X</td><td></td><td>1</td><td></td><td></td><td>1</td><td></td><td></td><td></td><td></td><td>2</td><td>3</td></tr><tr><td>22</td><td>1</td><td>1</td><td>7</td><td>1</td><td>2</td><td></td><td>2</td><td>X</td><td></td><td>X</td><td>2</td><td>3</td><td></td><td></td></tr><tr><td>23</td><td>1</td><td>1</td><td>6</td><td></td><td>X</td><td></td><td></td><td>2</td><td></td><td>4</td><td>4</td><td>4</td><td>1</td><td></td></tr><tr><td>24</td><td></td><td></td><td>8</td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td>2</td><td></td><td>3</td><td></td></tr><tr><td>25</td><td></td><td></td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>5</td><td>5</td></tr><tr><td>26</td><td></td><td></td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>5</td><td>8</td></tr><tr><td>27</td><td></td><td></td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td></td><td>8</td><td>8</td></tr><tr><td>28</td><td></td><td></td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td>1</td><td></td><td>8</td><td>8</td></tr><tr><td>29</td><td>1</td><td></td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td>1</td><td>1</td><td>8</td><td>8</td></tr><tr><td>30</td><td></td><td></td><td>1</td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td></td><td></td><td>6</td><td>7</td></tr><tr><td>31</td><td></td><td></td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td>2</td><td></td><td>8</td><td>8</td></tr><tr><td>32</td><td>1</td><td></td><td>3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>2</td><td></td><td>6</td><td>7</td></tr><tr><td>33</td><td></td><td></td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>2</td><td></td><td>8</td><td>8</td></tr><tr><td>34</td><td></td><td></td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>7</td><td>8</td></tr><tr><td>35</td><td></td><td></td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>2</td><td></td><td>7</td><td>8</td></tr><tr><td>36</td><td></td><td></td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>7</td><td>8</td></tr><tr><td>37</td><td></td><td></td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td></td><td>6</td><td>6</td></tr><tr><td>38</td><td></td><td></td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td></td><td>8</td><td>8</td></tr><tr><td>39</td><td></td><td></td><td>7</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>2</td><td>1</td></tr><tr><td>40</td><td></td><td></td><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td>1</td><td></td><td></td><td>7</td><td>7</td></tr><tr><td>41</td><td></td><td></td><td>4</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>2</td><td>1</td></tr></tbody></table>
# Appendix F - Rating Instructions

## Review the Occupation

Before beginning the ability and skill ratings, review the occupation title, definition, job zone, and tasks to get a full picture of the occupation.

## Making Importance Ratings

(Follow Steps 1 – 5 for every construct to be rated.)

### Step 1 – Considering the Construct

* Review the title, definition, and *Level* scale anchors of the construct you are about to rate. This will help you understand the construct and what behaviors are like that requires only a little or a lot of the particular Ability/Skill.

### Step 2 – Considering the Tasks

* Consider the *Importance* Rating scale and its anchors.

1 = Not Important

2 = Somewhat Important

3 = Important

4 = Very Important

5 = Extremely Important

* Look at the tasks and their incumbent mean *Importance* ratings. Read each task carefully keeping in mind that they are presented in order of importance, within their designation of Core or Supplementary Tasks.

* Focus primarily on the Core Tasks (i.e., tasks that are critical to the occupation, and have both (a) relevance of ≥ 67% and (b) mean importance rating of ≥ 3.0).

* However, you should also review the Supplementary Tasks (i.e., tasks that are less relevant and/or important to the occupation and either (a) tasks rated > 67% on relevance but < 3.0 on importance, or (b) tasks rated between 10% and 66% on relevance, regardless of mean importance rating).

* Based on your review of the tasks and their *Importance* ratings, think of a preliminary rating for the *Importance* of this construct to performance of this occupation. Your preliminary rating should reflect the importance of this particular construct to the overall performance of this occupation.

### Step 3 – Considering Knowledge

* Look at the knowledge list and the incumbent mean Importance ratings. Read each knowledge, keeping in mind that they are presented in order of importance. Note that only important (mean importance rating of ≥ 3.0) knowledges are provided.
* If necessary, adjust your preliminary rating for the Importance of this construct to performance of this occupation.

### Step 4 – Considering GWAs

* Now move on to the GWAs that are relevant to this construct for this occupation. Only GWAs that were linked to this construct and that received incumbent mean *Importance* ratings of 3.0 or greater are shown. Review the linked GWAs and their mean *Importance* ratings. (Note: If there are no GWAs linked to this construct/ occupation, move on to Step 4.)

* If necessary, adjust your preliminary rating for the *Importance* of this construct to performance of this occupation.

### Step 5 – Considering Work Context

* Now move on to the Work Context statements that are relevant to this construct. Only Work Context statements that were linked to this construct and that received incumbent mean ratings indicating sufficient relevance are shown. Review the statements and their mean ratings. (Note: If there are no Work Context statements linked to this construct/occupation, base your rating on Steps 1 – 3.)

### Step 6 – Documenting Your Rating

* If necessary, adjust preliminary rating for the *Importance* of this construct to performance of this occupation and enter your final rating in the spreadsheet for this occupation. Remember that the *Importance*-rating cells in the spreadsheet will only accept values in the appropriate range (i.e., 1 – 5).

* Now that you have completed the *Importance* rating for this construct, complete the *Level* rating (if your *Importance* rating was ≥ 2) for this construct before moving on to the next construct.

#### Making Level Ratings

*(Follow Steps 1 – 6 for every construct to be rated.)*

### Step 1 – When to Provide a Level Rating

* If you rated this construct as at least *Somewhat Important* (i.e., ≥ 2), follow Steps 2 – 4 to provide a *Level* rating between 1 and 7.

* If you rated this construct as *Not Important* (i.e., 1), give the construct a *Level* rating of 0 and move on to the next construct.

### Step 2 – Considering the Level Anchors

* Consider the rating scale and its anchors.

* Review the *Level* scale anchors and at what point on the scale they fall.

* Remember *Level* scale anchors increase (from 1 – 7) in the amount of the particular Ability/Skill required to perform the behavior.
### Step 3 – Considering the Tasks

* Consider the *Level* Rating scale and its anchors.

* Think about the Core and Supplementary Tasks.

* Based on these tasks, think of a preliminary rating for the *Level* of this construct needed to perform this occupation. Remember, this rating reflects the *Level* of this particular construct needed to perform this occupation as a whole.

### Step 4 – Considering Knowledge

* Look at the knowledges and their incumbent mean Importance ratings. Read each knowledge, keeping in mind that they are presented in order of importance. Note that only important (mean importance rating of ≥ 3.0) knowledges are provided.

* If necessary, adjust your preliminary rating about the *Level* of this construct needed to perform this occupation.

### Step 5 – Considering GWAs

* Now move on to the GWAs. Think about the GWA statements and their mean *Level* ratings. (Note: If there are no GWAs linked to this construct/occupation, move on to Step 4.)

* If necessary, adjust your preliminary rating about the *Level* of this construct needed to perform this occupation.

### Step 6 – Considering Work Context

* Now move on to the Work Context statements. Think about the statements and their mean ratings. (Note: If there are no Work Context statements linked to this construct/occupation, base your rating on Steps 1 – 3.)

### Step 7 – Documenting Your Rating

* If necessary, adjust your preliminary rating about the *Level* of this construct needed to perform this occupation and enter your final rating in the spreadsheet for this occupation. Remember that the *Level*-rating cells in the spreadsheet will only accept values in the appropriate range. That is, 1 – 7 if your *Importance* rating for that construct was ≥ 2, and 0 if your *Importance* rating for that construct was 1.

### Next Steps

* Now that you have completed the *Level* rating for this construct, move on to the next construct.

* Review rating tendency feedback from previous batches and check them against your current ratings.

* Remember to refer regularly to the Ability, Skill, GWA and WC definitions.

* Remember to refer regularly to the *Clarifying Potentially Misunderstood Ability and Skill Definitions* handout.
Appendix G - Sample Stimulus Material
29-2011.00 Medical and Clinical Laboratory Technologists

Job Zone: 4

Perform complex medical laboratory tests for diagnosis, treatment, and prevention of disease. May train or supervise staff.

<table><thead><tr><th></th><th></th><th>1 = Not Important</th><th>2 = Somewhat Important</th><th>3 = Important Important</th><th>4 = Very Important</th><th>5 = Extremely Important</th><th>Importance</th></tr></thead><tbody><tr><td colspan="8"><strong>Core Task</strong></td></tr><tr><td>1</td><td>Analyze laboratory findings to check the accuracy of the results.</td><td></td><td></td><td></td><td></td><td></td><td>4.8</td></tr><tr><td>2</td><td>Conduct chemical analysis of body fluids, including blood, urine, and spinal fluid, to determine presence of normal and abnormal components.</td><td></td><td></td><td></td><td></td><td></td><td>4.8</td></tr><tr><td>3</td><td>Enter data from analysis of medical tests and clinical results into computer for storage.</td><td></td><td></td><td></td><td></td><td></td><td>4.7</td></tr><tr><td>4</td><td>Operate, calibrate and maintain equipment used in quantitative and qualitative analysis, such as spectrophotometers, calorimeters, flame photometers, and computer-controlled analyzers.</td><td></td><td></td><td></td><td></td><td></td><td>4.6</td></tr><tr><td>5</td><td>Establish and monitor quality assurance programs and activities to ensure the accuracy of laboratory results.</td><td></td><td></td><td></td><td></td><td></td><td>4.6</td></tr><tr><td>6</td><td>Set up, clean, and maintain laboratory equipment.</td><td></td><td></td><td></td><td></td><td></td><td>4.4</td></tr><tr><td>7</td><td>Provide technical information about test results to physicians, family members and researchers.</td><td></td><td></td><td></td><td></td><td></td><td>4.3</td></tr><tr><td>8</td><td>Supervise, train, and direct lab assistants, medical and clinical laboratory technicians and technologists, and other medical laboratory workers engaged in laboratory testing.</td><td></td><td></td><td></td><td></td><td></td><td>4.1</td></tr><tr><td colspan="8"><strong>Supplementary Task</strong></td></tr><tr><td>9</td><td>Collect and study blood samples to determine the number of cells, their morphology, or their blood group, blood type, and compatibility for transfusion purposes, using microscopic techniques.</td><td></td><td></td><td></td><td></td><td></td><td>4.7</td></tr><tr><td>10</td><td>Obtain, cut, stain, and mount biological material on slides for microscopic study and diagnosis, following standard laboratory procedures.</td><td></td><td></td><td></td><td></td><td></td><td>4.6</td></tr><tr><td>11</td><td>Analyze samples of biological material for chemical content or reaction.</td><td></td><td></td><td></td><td></td><td></td><td>4.6</td></tr><tr><td>12</td><td>Cultivate, isolate, and assist in identifying microbial organisms, and perform various tests on these microorganisms.</td><td></td><td></td><td></td><td></td><td></td><td>4.6</td></tr><tr><td>13</td><td>Select and prepare specimen and media for cell culture, using aseptic technique and knowledge of medium components and cell requirements.</td><td></td><td></td><td></td><td></td><td></td><td>4.5</td></tr><tr><td>14</td><td>Develop, standardize, evaluate, and modify procedures, techniques and tests used in the analysis of specimens and in medical laboratory experiments.</td><td></td><td></td><td></td><td></td><td></td><td>4.1</td></tr><tr><td>15</td><td>Harvest cell cultures at optimum time based on knowledge of cell cycle differences and culture conditions.</td><td></td><td></td><td></td><td></td><td></td><td>4.1</td></tr><tr><td>16</td><td>Conduct medical research under direction of microbiologist or biochemist.</td><td></td><td></td><td></td><td></td><td></td><td>3.1</td></tr><tr><td colspan="8"><strong>Knowledge</strong></td></tr><tr><td>1</td><td>Biology</td><td></td><td></td><td></td><td></td><td></td><td>4.2</td></tr><tr><td>2</td><td>Chemistry</td><td></td><td></td><td></td><td></td><td></td><td>3.8</td></tr><tr><td>3</td><td>Customer and Personal Service</td><td></td><td></td><td></td><td></td><td></td><td>3.7</td></tr><tr><td>4</td><td>Medicine and Dentistry</td><td></td><td></td><td></td><td></td><td></td><td>3.6</td></tr><tr><td>5</td><td>English Language</td><td></td><td></td><td></td><td></td><td></td><td>3.5</td></tr><tr><td>6</td><td>Mathematics</td><td></td><td></td><td></td><td></td><td></td><td>3.5</td></tr><tr><td>7</td><td>Clerical</td><td></td><td></td><td></td><td></td><td></td><td>3.3</td></tr><tr><td>8</td><td>Computers and Electronics</td><td></td><td></td><td></td><td></td><td></td><td>3.2</td></tr><tr><td>9</td><td>Mechanical</td><td></td><td></td><td></td><td></td><td></td><td>3.1</td></tr></tbody></table>
## Skill 1 Reading Comprehension

Understanding written sentences and paragraphs in work-related documents.

### Level Scale Anchors

<table><thead><tr><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th></tr></thead><tbody><tr><td></td><td>Read step-by-step instructions for completing a form</td><td></td><td>Read a memo from management describing new personnel policies</td><td></td><td>Read a scientific journal article describing surgical procedures</td><td></td></tr></tbody></table>

### Incumbent Ratings for Relevant Descriptors

<table><thead><tr><th>Generalized Work Activity:</th><th>Importance</th></tr></thead><tbody><tr><td><strong>Information Input</strong></td><td></td></tr><tr><td>1 Getting Information</td><td>4</td></tr><tr><td>2 Identifying Objects, Actions, and Events</td><td>4</td></tr><tr><td>3 Monitor Processes, Materials, or Surroundings</td><td>4</td></tr><tr><td><strong>Mental Processes</strong></td><td></td></tr><tr><td>6 Judging the Qualities of Things, Services, or People</td><td>4</td></tr><tr><td>7 Evaluating Information to Determine Compliance with Standards</td><td>4</td></tr><tr><td>8 Processing Information</td><td>4</td></tr><tr><td>9 Analyzing Data or Information</td><td>4</td></tr><tr><td>10 Making Decisions and Solving Problems</td><td>4</td></tr><tr><td>12 Updating and Using Relevant Knowledge</td><td>4</td></tr><tr><td><strong>Work Output</strong></td><td></td></tr><tr><td>19 Working With Computers</td><td>4</td></tr><tr><td><strong>Interacting with Others</strong></td><td></td></tr><tr><td>25 Interpreting the Meaning of Information for Others</td><td>3</td></tr><tr><td>35 Training and Teaching Others</td><td>3</td></tr></tbody></table>

<table><thead><tr><th>Work Context:</th><th>Rating</th></tr></thead><tbody><tr><td><strong>Frequency</strong><br>(3 = Once a month or more but not every week; 4 = Once a week or more but not every day; 5 = Every day)</td><td></td></tr><tr><td>4 How frequently does your current job require electronic mail?</td><td>4</td></tr></tbody></table>