f!. ;j

Ai~

rill

,I!:F 

u

ta~diI

pWI

. ... ....

#Fqmg'L f

r. re J.K

tli
Planning for the Future:

Progress Toward an NCO

Competency Assessment Program

Roy C. Campbell

and

Deirdre J. Knapp

Human Resources Research Organization

Tonia S. Heffner

U.S.Army Research Institute

for the Behavioral and Social Sciences

DISTRIBUTION STATEMENT A

Approved for Public Release

Distribution Unlimited

Special Report 61

November 2005

U.S.Army Research Institute

for the Behavioral and Social Sciences

www.ari.army.mil

Z0060120 IZO
SPlanning for the Future: Progress Toward an NCO Competency Assessment Program

ii
U.S. Army Research Institute

Foreword

n April 2002, the Army Training and Leader Development Panel (ATLDP) released

the results of its survey of 35,000 noncommissioned officers (NCOs). The ATLDP

recommendations included the need for regular assessment of Soldiers' technical,

tactical, and leadership skills. The need for regular assessment of Soldiers' skills

coincides with a series of research programs on NCO assessment and development

executed by the U.S. Army Research Institute for the Behavioral and Social Sciences

(ARI). ARI's research programs begin with an effort to identify potential knowledge,

skills, and attributes (KSAs) for future Soldiers, Soldier Characteristics of the 2 1st

Century (Soldier 21). They continued with an effort to identify and validate potential

indicators of the KSAs for use in junior NCO promotion, Maximizing 2 ist Century

Noncommissioned Officers Performance (NC021). Finally, the research program

described in this report provides a three-phase investigation of the issues and possible

resolutions for developing a viable Army assessment program that would be linked

to NCO promotions. This program is Performance Measures for 21st Century Soldiers

Assessment (PerformM21).

Information on PerformM21 was briefed to the Deputy Chief of Staff, Army G- 1 on

8 Oct 2003, the Director, Military Personnel Management, Army G- 1, on 25 Jan 2004, and

Chief, Enlisted Professional Development, Directorate of Military Personnel Management

on 13 Nov 2003 and 29 Dec 2004, as well as the Deputy Chief of Staff, Operations and

Training, Training and Doctrine Command, on 3 Mar 2005. It was briefed to the Sergeant

Major of the Army on 28 Jan 2003 and 30 Mar 2004. It has been briefed quarterly

to senior NCO representatives from U.S. Army Training and Doctrine Command

(TRADOC), Office of the Sergeant Major of the Army, Office of the Army G- 1, U.S.

Forces Command (FORSCOM), U.S. Army Reserve (USAR), and the Army National

Guard (ARNG) as members of the Army Test Program Advisory Team (ATPAT).

The PerformM21 research program is still in progress. This special report to the field

presents a non-technical overview of the current status of the program and outlines

the steps yet to be taken. It is intended to inform, stimulate, and invoke discussion

and response.

ZITA M. SIMUTIS

Director and Chief Psychologist

of the United States Army

III
O PJannin#1ofr the Future: Progress Tew~ft an NCO Competen~y Assessment Proram

Acknowledgement As a group, we are indebted to the Army Test Program Advisory Team (ATPAT),

currently chaired by SGM Michael Lamb, Sergeant Major at the Training

and Doctrine Command (TRADOC) Deputy Chief of Staff for Operations

and Training (DCSOPS&T), and assisted by SGM (R) Gerald Purcell, Office of the

Army G- 1. The following is a list of the individuals who have served on a volunteer

basis as members of the ATPAT since its inception in April 2003. Their willingness to

get involved, to contribute time and resources, and to support project goals was, and

continues to be, an immeasurable asset to this research effort.

Army Test Program Advisory Team (ATPAT) Members

CSM Cynthia Pritchett, Chair Emeritus SGM Christopher Miele

SGM Michael Lamb, Chair SGM Pamela A. Neal

MAJ Chris Keller CSM Rock A. Orozco

SGM Thomas Clark SGM Tim Ozman

SGM Fredrick Couch CSM Nick Piacentini

SGM John T. Cross CSM Doug Piltz

SGM Osvaldo Del Hoyo SGM (R) Gerald Purcell

CSM George D. DeSario CSM Robie Roberson

SGM Daniel Dupont CSM Otis Smith Jr.

SGM (R) Julian Edmondson SGM Irene Torkildson

CSM Dan Elder CSM (R) Clifford R. West

CSM (R) Victor Gomez MSG Daphne Angell

SGM John Griffin MSG Robert Bartholomew

SGM (R) Paul Harzbecker MSG Monique Ford

SGM James Herrell MSG (R) Fred Liggett

SGM John Heinrichs MSG Matt Northen

SGM Enrique Hoyos 1 SG Edwin Padilla

SGM (R) Dan Hubbard MSG Jerome Skeim

SGM David J. Litteral MSG Edward Herbert

SGM Michael Magee SFC Kevin Barney

SGM John W. Mayo SFC Martha Chavez

SGM Tony McGee SFC Chad McNeish

SGM Steven R. Merrill Mr. Jeff Colimon

iv
U.S. Army Research Institute

Table of Contents

Army Test Program Advisory Team (ATPAT) Members .............................. iv

Assessing NCO Skills .............................................................................................. 1

W hat is Assessm ent? ........................................................... ................................ . . . 1

W hy Do Assessm ents? .......................................................................................... 1

O u r M issio n ....................................................................................................................... 2

The Assessment Research Program ................................................................. 3

PERFORMM21: Getting Assessment Right ................................................. 5

The Army-Wide Core Assessment: A Prototype for All Soldiers .............. 7

Soldier Preparation: Critical To the Mix ............................................................ 9

Technical Assessments: A Real-Life Approach ............................................ 11

The Army Test Program AdvisoryTeam (ATPAT): The Army's Voice ......... 13

Past Army Test Experiences: What We've Learned .................................. 15

What Happens Next? The Short Term Outlook - The Long Term Vision ..... 17

Your Turn: Feedback? ................................................. ................................. . . . 21

V
U.S. Army Research Institute

Assessing NCO Skills

What is Assessment?

ssessment is a term that refers to a standardized, systematic, and objective

method for evaluating individuals' performance in selected areas. Many

would refer to this type of method of evaluating as a "test. In its application, it

centers on job performance that can be scored objectively and demonstrates a level of

knowledge or skill an individual has in a given area.

Why Do Assessments?

The value of assessment is in measuring Soldier studying ata

and validating knowledge and experience desk;self-development

in order to award credit. Assessments is one pillar of the Army

can be used to enhance both personnel training system.

and training management practices. In

the personnel arena, it can help identify

who should be promoted and/or who

will succeed in a special assignment. In

the training arena, it can help validate skills and experiences and/or evaluate training

needs and effectiveness. Good assessment leads to good decisions, which leads to an

overall increase in organizational performance through a better NCO corps. Testing is

not a replacement for development and training, but it does help ensure accountability

for those activities. Assessment should drive training only to the extent that both

assessment and training are focused on the same critical job requirements.

Like anything else however, not every assessment test is equally good. To be effective,

assessments must reflect job requirements and frame questions in a realistic

manner that maximizes the relationship between the testing context and on-the-job

performance. This can be done and there is overwhelming research evidence that

effective testing does improve individual and organizational performance.

The Army has a long history of assessment, beginning with the introduction of group

testing during World War I and continuing with the current Armed Services Vocational

Aptitude Battery (ASVAB). But what makes assessing NCOs a current topic of interest?

, , i i I I I I l I l l i I
Q Planning for the Future: Progress Toward an NCO Competency Assessment Program

A Mandate from the Field

"Develop and sustain a competency assessment program for evaluating Soldiers' technical

and tactical proficiency in the military occupational specialty (MOS) and leadership skills

for their rank."

This statement comes not from the Army policymakers but from Soldiers and leaders

across the operational spectrum. In 2001, the Army Training and Leader Development

Panel (ATLDP) sought input from 35,000 NCOs and their supervisors on a variety

of topics to chart the future needs and requirements of the NCO corps. This effort

is crucial because it reflects what NCOs see as necessary to enhance and sustain a

competent, professional NCO corps to meet current and future demands. Foremost

among those requirements was the need for an objective job assessment as additional

criterion for the selection of Soldiers to positions of leadership and responsibility.

Endorsed by the Chief of Staff of the Army, the ATLDP findings became the focus of

attention for Army Leadership. In 2003, Sergeant Major of the Army (SMA) (R) Jack

Tilley outlined a potential competency assessment program that should:*

* Be incorporated into the NCO promotion system

* Be web-based

e Apply to the entire Army (Active Component, Army National Guard, Army Reserve)

* Apply to Specialist/Corporal (E4) through Sergeant First Class (E7)

* Be initially developed as a core assessment, applicable to all Soldiers regardless of

military occupational specialty (MOS); MOS-specific competency assessments to

be added later

* Include basic NCO history, Army values, leadership, and training subject matter in

the common core assessment

• Include situation-based assessments

Part of a Coptnulng Program

The.Army4,ý, t4 ..... tio W and c. m_ tin and the

M t'a supole 'totOhe

adminisrt• * sdtdNW-*i urufzdkha•p1nAISea

Repo rt 52 Sdrto For tedespeTnsrIalKVft

* The assessment program and SMA Tilley s guidonce were briefed to, and endorsed by, SMA Preston in March 2004.

2
U.S. Army Research Institute

The Assessment Research Program

To meet the Army's need for job-based measures, the U.S. Army Research Institute for

the Behavioral and Social Sciences (ARI) started a 3-year feasibility research program

to identify viable approaches for an effective and affordable Soldier assessment system.

The work is being conducted in three phases:

"* Phase I: Perform a needs analysis and examination of assessment options.

"* Phase II: Design and develop a prototype core knowledge assessment.

"* Phase III: Examine job-specific measurement alternatives and develop final

recommendations for an Army system.

Currently the 3-year ARI research program is about 50%

complete, and although it is too early to report results, Soldier working on laptop

progress to date has enabled us to develop a vision of aossie nt

what such a program might look like and what the Army computer; assessment

would need to include. As of yet, these are initial concepts 4 items are designed with

based on our research - it still remains for Army leaders to simulation to be realistic.

approve the elements of the assessment program as well as

to make a final decision regarding whether an assessment

program will be implemented. However, based on our

research, the following design features seem likely:

" The performance assessment system would consist of

core assessments (all Soldiers - Army-wide skills and

knowledge) plus technical (MOS) assessments, tested

separately. Core assessments could be implemented

fairly quickly while technical assessments would

require a somewhat longer lead time.

"* Computer scorable items would be designed to be realistic rather than textbook-like.

"* Tests would be administered annually (although individual Soldiers might only

have required assessments every three years consistent with the brigade combat

team lifecycle, or in conjunction with promotion eligibility). The assessment

program would be phased in gradually (e.g., starting with only one or two pay

grades). The first assessments would most likely be part of a program pilot and

not be for-record.

" Defined windows would be established for test preparation, test taking, and test

make-up or retesting. Emphasis would be on maximizing Soldier access to test

taking within the defined windows.

3
@ Planning for the Future: Progress Toward an NCO Competency Assessment Program

" Core assessments would be scheduled and administered on-line or server-based

through proctored, Army-sponsored portals.

" Scores would be converted to promotion points and entered into Soldier records.

Individual feedback reports would be provided to Soldiers.

" Test preparation guides would be electronically available to Soldiers and would be

supplemented by a self-assessment program that refers Soldiers to training and job

aids as part of their preparation. A self-assessment program would operate before

for-record core assessments were instituted.

Advantages of an Army Job Assessment

* Provides a new, oWepromo tool to supplement other promoton aiteria.

* ndrefere material. meril.tne 

* 4af k my mission.

0 Add empasi onstaftre to the NCO corps.

* Drhvs deveopm~entand d~stufion of up-to-aterderences and learning materials.

4
U.S. Army Research Institute

PERFORMM21: Getting Assessment Right

nstituting an Army wide assessment program is a significant undertaking

involving virtually every aspect of Army functions - personnel, training,

operations, communications, and computers - and all of the major Army

commands. Moreover, competency assessment itself is a complex issue requiring a

systematic foundation in job and performance analysis, development of test plans or

blueprints, tryouts and validation of items, and specifications for scoring and feedback.

The availability of computer-based, computer-administered testing, while affording

tremendous new opportunities, also requires new paradigms of test development and

administration. Army assessment requires a collusion of many players - test developers,

analysts, subject matter experts (SMEs), policymakers, trainers, and systems designers

and administrators.

Above all, assessment is a serious and

sensitive issue. Leaders and Soldiers must

have confidence in their testing system. Soldiers working on laptop

Confidence comes from knowledge and computers; assessment

understanding - failure to communicate involves many aspects of

all aspects of the system to the user breeds Army functions- personnel,

misconceptions and suspicions. A hastily .. training and development,

developed assessment system, which has • operations, communications,

not been thoroughly investigated for flaws 4 . ' and computers.

and complications, cannot succeed. A

competency assessment campaign is just

like any commander's battle operation -

it must be solidly grounded in expert

input, detailed planning, rehearsals, and

anticipation of branches and sequels.

NCOs in the Expeditionary Army

"IAs westrive to develop flexible, adaptive, competent leaders it is imperative that we have the right tools

to identify and prepare Soldiers to perform, lead, and excel in an Expeditionary Army. The 6lobal War on

Terrorism herein Afghanistan presents challenges that cause Soldiers to use an array of skills: problem

solving, conceptual, communication, tactical and technical, interpersonal, mental, emotional, and team

building. They have to be agile, flexible, and adaptive, and have a keen sense of situational and selfawareness. We must find ways to identify and assess potential, and evaluate leadership and performance so

that we can teach, coach and mentor them to even higher performance. More than ever, we need to ensure

our Soldiers have the capabilities to lead, perform, and excel in an Expeditionary Army."

CSM Cynthia Pritchett

Command Sergeant Major

Headquarters, Combined Forces Command - Afghanistan

5
QPlanning for the Future: Progress Toward an NCO Competency Assessment Program

Many remember the Army experience with the Skill Qualification Test/Soldier

Development Test (SQT/SDT) of the 1980s and early 1990s. Because the lessons learned

from SQT/SDT programs are so valuable, a section is devoted to this experience at the

end of the report.

The project takes a deliberative approach to researching the foundation of an effective

assessment system. An overview of our approach is shown below. The method involves

gathering data and information, and identifying and exploring options, but maintaining

an eye towards the operational implications. This allows design and testing of an

operational system using prototypes to discover pieces that work, as well as potential

pitfalls for the operational implementation.

From Research to Operation

ER C OPERATIONAL CONCEPT

S1d. Am(oide AssAmen Program

2.Soldier Test Guie ndSew-ssssen toleaton 

Technical ~ ~ ~ Sollt illgttet ~$pre qides

1. Army-Widcore Assessments

2Soldier Prparation: T i S s nta

3. Technical Assessments

6
U.S. Army Research Institute

The Army-Wide Core Assessment: A Prototype for All Soldiers

In 2004 a prototype test was designed to measure basic soldiering performance and

knowledge requirements. Approximately 280 computer-based test items (traditional

multiple-choice, matching, ranking, and drag-and-drop) were developed for

administration to E4 Soldiers. The content of the test is:

"* Common Tasks Skill Level 1 (46%)

"• Common Tasks Skill Level 2 (14%)

"• Army and NCO History/Army Values (15%)

"* NCO Leadership (13%)

"* Conduct of Training (12%)

This web-based prototype test was administered to 571 Soldiers in late 2004 using the

Army Digital Training Facilities (DTF) computer suites. Analysis of data from these

administrations is on-going.

A room of Soldiers taking

the prototype competency

assessment in the Fort

Riley digital training

facility.

Although the prototype could serve as the groundwork for an operational assessment,

it was developed without a complete and systematic job analysis that would provide

the best foundation for such an operational test. A new test blueprint plan, based on

incumbent and supervisor survey data, and prioritized and reviewed by knowledgeable

training, doctrine, measurement, and policy groups, needs to be prepared to support

an operational test. Such a procedure would insure timeliness and relativity in a

changed operational climate. The next generation test should also reflect technological

capabilities including advanced media and graphics potentials to include sound and

motion for more realistic and versatile test item presentations. These capabilities were

explored in the MOS technical phase described later in this section.

7
@Planning far the Future: Progress Toward an NCO Competency Assessment Program

What Soldiers Think

As part of the prototype Army-wide core measures, we administered a post-test questionnaire and

conducted focus group workshops with Soldiers who took the test. There were about 375 participants,

all in the grade of E4, representing both Active and Reserve Component units,and from a cross section of

combat, combat support, and combat service support jobs. Some results:

" Only about 10% of the Soldiers had used the Digital Training Facility (DTF) before the test but

75% agreed that the DTF was a good facilityfor testing.

* 73% of the Soldiers had taken computer tests before. When asked to choose between a computer

test and a paper-and-pencil test, 91% preferred computertests.

" Many test items are non-traditional;that is, something other than 4-option multiple-choice

questions. Examples include matching, drag and drop, and"select all that apply" responses.

Overall, Soldiers liked the non-traditional items. Many responded that it madethe test topics

more 'interesting"and were'different from typical multiple-cholce test items.

" Soldiers were asked woquestos about thtt content categories (1) How effective is the

test in evatuating proficiency in this area? and (2) How well did you do today on this part of the

test? The questions wee on a 5-polnt scale w4hery elfftctv.e- very well and "effective -wellt

representing thetop two points.

Salffs who.. Common Tasks listory Leadership Troiing Values

Although Soldiers knewtheywere taking part In a research project and the te results would not affect

them or be reported, proctorsperceived that almost all appeared to ake the test seriously. Based on

the Soldlers! reactions,the demonstration core test was not perceived to be an 'easy'test. However,

despite concerns about how wefl they performed, Soldiers did not blame the test. Their appraisal of the

effectiveness of the test content was, overall, very positive.

8
U.S. Army Research Institute

The Criticality of Job Analysis

It is an axiom in the performance measurement field that you don't have to do job analysis for all the

assessments being developed - only for the ones you want done right. This applies to both the core

assessments (all Soldiers) and to the MOS technical assessments. In its simplest form,job analysis tells

you what (content) and how much (weight) to include in the assessment.

Effective job analysis for purposes of assessment measurement should be based on up-to-date and

representative survey data from incumbents and supervisors, reflecting all venues in which the job is

performed. These survey results then must be analyzed and interpreted by knowledgeable job experts.

Final content selection for testing is a combination of all known factors, applied by knowledgeable and

experienced reviewers, and policymakers at a high level of authority.

Current Army job analysis is in a varied and inconsistent state. Since the job analysis that is being

conducted is not done with an assessment goal, revitalization of the Army job analysis program will

likely be a necessary start point for the introduction of an assessment program. Assessments that are not

based on an up-to-date job analysis risk being irrelevant to the incumbents and will be seen as outside

of the Soldier's job.

Soldier Preparation: Critical to the Mix

Essential to a Soldier assessment program is the concept of Soldier preparation.

Soldiers must be given information about what will be expected of them and the

tools to help them prepare. For the prototype core assessment, we developed a Test

Preparation Guide. This Guide - available both electronically and in hard copy - was

used as combination read-ahead, study guide, and test familiarization tool. It includes

information about what is covered on the assessment, sample items, and links to

relevant study resources (e.g., Soldier Manuals). While pre-pilot test notification

problems precluded a full trial of the intended use of the Guide, Soldier reactions to the

usefulness of the Guide were highly favorable.

We also have completed a prototype self-assessment system designed to accompany the

introduction of an Army assessment program. Such a system would provide individuals

with a means to judge their strengths and weaknesses in preparation for promotion

and provide opportunities for learning how to study. It would be web-based, accessed

through AKO, with links to other related Army sites and sources. The prototype selfassessment system has three components:

* An Information Component that offers specifics about:

- Testing program goals and eligibility requirements

- The test environment and test process

- Test preparation including test-taking strategies (this incorporates features of

the Test Preparation Guide described above)

- Links to a description of the semi-centralized promotion system including an

interactive promotion points worksheet (PPW)

9
@Planning for the Future: Progress Toward an NCO Competency Assessment Program

"A Knowledge Component consisting of practice items similar to those from the

operational test. This would not be a "practice test" because it would not give a

predictive score, but it would allow Soldiers to identify test topic areas that are

weaknesses for them.

" A Feedback Component consisting of identification of sources or study references.

Feedback needs to be specific to the type of item. Straightforward procedural

or knowledge items need minimal explanations, primarily just references to

the source documents. Leadership, situational, and problem solving exercises,

designed to tap Soldier's critical thinking skills, however, need more detailed

feedback including reasons and rationales supporting choices, along with

principles and doctrine that govern performance. Overall, the goal is drive

Soldiers to use training materials and resources that already exist. These materials

and resources will be provided as on-line links.

The prototype self-assessment is being designed to meet the immediate needs of the

initial Army-wide core assessment. As the assessment system matures, it will be capable

of adjustment to include providing MOS-specific information, sample job measures,

and promotion status or comparisons with peers within the MOS.

Thoughts on Assessment

"The•Army is i eed for a pr01? UW pvyW e edbackto theIdidual unit leades, and institutions. Suc a ýt*W Istegh n ekess •oW 

"tMeARftP lSuwn ft WqiwoWev :om 

been testde o tbW o ss thersW anst theip•ei One

consideratioat Ik lee repafdInementaiy

and sec aboutknowigtheirS in

more ways thonjusta subjetiWIaio~n byasuipayor. This generdon Is not lee•yof tests - they

welcome them as a way of cerIAg their know* end wviue."

SGM Michael I Lamb

TRADOC Deputy Chief of Staff for Operations and Training

10
U.S. Army Research Institute

Technical Assessments: A Real-Life Approach

An integral part of the Army assessment program includes obtaining measures of a

Soldier's job proficiency - in short, technical testing. Yet this is the most problematic

aspect. MOS vary greatly by incumbent size, complexity, performance requirements,

training, employment, and environment. Even within MOS, there can be differences in

utilization, assignments, equipment, and job tasks. Finally, plans for MOS consolidation

and reorganization, the increased use of assignment oriented training (AOT), increased

reliance on additional skill identifiers (ASI), and the very nature of Future Force Army

structure, indicate the turbulence will continue for the foreseeable future.

Nevertheless, technical assessment is too

important to be dismissed as "too hard" S

One approach is to throw out all the old Soldier entering data:

ideas about technical assessments - mainly assessmentwill provide a

that all MOS, and indeed, all Skill Levels comprehensive evaluation.

within an MOS, need to be treated the

same. Since promotion competition is

by MOS, not between MOS, no issue of

fairness should exist if one MOS has a test

and another does not, or if different MOS

have different assessment methods, or if

different MOS have different assessment

designs and cycles. Precedence already

exists for requiring or rewarding Army

jobs differently as part of the promotion -

criteria. For example, the Ordnance Corps

awards promotion points for civilian

certifications.

As part of the ongoing research, we investigated prototype options available to

assessments in five different MOS:

"• 63B Wheeled Vehicle Mechanic

"* 91W/68W Health Care Specialist

"* 19K Abrams Armor Crewman

"* 31B Military Police

"* 14E Patriot Fire Control Enhanced Operator/Maintainer

11
, Planning for the Future: Progress Toward an NCO Competency Assessment Program

These MOS were selected because they posed representative challenges and

opportunities for different assessment methods. The purpose was to explore the

assessment possibilities, rather than to develop complete tests. Part of this work

involved technological explorations, including computer-based simulations that

approximate job conditions without the expense and demands of the typical handson evaluation with real equipment. Other potential measures considered include

situational assessments, job product evaluations, virtual and constructive simulations,

and existing job measures. Discoveries during this research identified a number of

approaches that individual MOS could develop as a means of assessment.

Not A Pass or Fail Evaluation

A promotion relted assessm SJ ic with other similar Solimin the promotion pool, not

with a preet sntdad Thr scong Inov Sl•diers who perform much

better than their 3 *dtmax mprootion points allowed for the assessment

factor while those who n0N Wed to their peen wol receive fewer promotion points.

Whatever the outcome, the movement towards technical assessments will likely be

much slower than that towards an Army core assessment. It is likely that efforts will be

initiated individually, by MOS or by Career Fields, rather than on an Army-wide scale

involving all MOS. Indeed, some Career Fields have already made an initial move and

many more will follow. This independence and distinctiveness in the approach towards

technical testing is viewed as a positive. This does not mean that development will be

without boundaries or that practices can be instituted without constraint or without

coordination. But if individual programs comply with professional testing standards

and with Army policy, they can and should differ across jobs.

12
U.S. Army Research Institute

The Whole Soldier Concept

Competency assessment results would be another tool on which to assess Soldiers' readiness for

promotion - but it would not be the sole tool. For the semi-centralized promotion system (promotion to

grades E5 and E6), military and civilian education, awards and decorations, and military training (Army

Physical Fitness Test and weapons qualification) will continue as weighted factors. Commanders would

continue to recommend (or not recommend) Soldiers for promotion and Promotion Board appearances

may continue as a requirement. The weight (i.e., points) of a competency assessment test within the

overall promotion points worksheet is a policy decision that is yet to be determined.

In the centralized promotion system (to grades E7 and E8), competency assessment results become one

more factor available for centralized selection boards to consider. As they do now, boards would continue

to receive specific guidance for promotion considerations.

The Army Test Program Advisory Team (ATPAT): The Army's Voice

As indicated, an Army assessment program encompasses many functions and facets.

Research and expertise from the testing community can provide some of the answers

and identify the design requirements for much of a potential Army program. But an

operational program will involve many policy decisions and require organizational and

cultural considerations that transcend purely test program-oriented research. To insure

that the research and program development can lead to an acceptable, implementable

operational program, a special team, the ATPAT, has been in action since April 2003.

Composed primarily of Sergeants Major and Master Sergeants representing a wide

variety of Army commands and functions, the ATPAT meets quarterly with the research

team. Although the specific contributions of the ATPAT are many and varied, the

general functions of the team are to:

"* Serve as an oversight group providing insights into operational implications and

real-world feasibility of an implementable assessment program.

"* Provide continuing input and update to the analysis and definition of needs

requirements for an Army assessment program.

"* Provide product reviews, subject matter expertise, and external assistance and

contacts (as needed) in the process of developing prototype measures and trial

procedures.

" Serve as a conduit to explain the assessment research program to various Army

agencies and constituencies.

" Provide real time feedback on changing conditions and anticipated activities

within the Army that affect the assessment program.

13
Q Planning for the Future: Progress Toward an NCO Competency Assessment Program

ATPAT Members

The ATPAT was established In April2003 largejy thr.ugthe Initiative n4 recommendations of

CSM Cynthia Pritchettthen Commend Serge Center and Fort

Leavenworth. CSM l'rlthett served is the ATMI eroputfl heresslgnment as Command

Sergeant MujorHeadquartenComblned forc( aqi.Afghau1stan.5te coptinues to hold position in the ATPAT as Chair Emeritus.

ATPAT participation varies with commltmentsasslg tsdpIomntsind retirements. There have

been 46 ATPAT members to date. Even thoug embrs bec &jactlw, we consider ATPAT

membership to be permatentftragen4Jng

The current ATPAT Chatrp*ison is S4M MlchoI tl* iM 1n#aIid Doctrine Command,

Deputy Chief ofiaff for mrn4Trlln&11ortMqnroeVA (151-788-5109) michael.lamb@us.annmfl.

The ATPAT gewrally mevethree iths.iu Ad t.ilrgle$a ?tl4patIon In meetings is

sponsored bythe member's mwnd&aeocy.

More abQut the ATPAT

ATPAT members have dkect In lwnienZ.i4 pokein ri. PQUI4 like to leam more about

the ATPAT orwoil4Ilkttt4 1ntacts ptvlded above,

or DrTonla Heffnei4SI ISII* (ARI), Arlington,

VA U03-602-7948)totlIa4V qda*flrNI.

Iwo Soldiers field training.

Well-trained Soldiers

make good decisions,

whkhleadstoanoverall

increase in organizational

performance through

a better NCO corps.

Knowledge gained

through training and

quality of decision making

are just two aspects of

competency assessment.

14
U.S. Army Research Institute

Past Army Test Experiences:

What We've Learned

esting for purposes of proficiency measurement is not a new concept in the Army.

From the post-Korean War period to about the mid-1970s, NCOs (in grades E5

through E7) took an annual MOS test, both to qualify in their MOS and to draw

extra pay for the highest performers. This was followed by the Skill Qualification Test/Soldier

Development Test (SQT/SDT) period, which lasted into the early 1990s. The SQT/SDT

was very much an evolutionary process - it was many things over a period of time, encompassing written tests, hands-on tests, certifications, and a variety of personnel applications

for scoring. Almost everyone's recollection of SQT/SDT differs by when they experienced it.

We have spent considerable time researching the history of promotion testing, particularly

the SQT/SDT experiences - both the good and the bad. We have identified some lessons

learned to take forward into a new Army testing program. Specifically:

" Minimize the logistical load. SQT/SDT was a completely paper-based, mail-out, and

returns system.

" Do the research. SQT/SDT broke a lot of new ground, and very quickly. We know a lot

more about large-scale job testing - in all applications - than we did 30 years ago.

" Perceptions count - whether they are accurate or not. Make sure Soldiers, leaders,

and commanders understand the program and have a way to get answers to their

questions. Actively push information to the field and install programs that maximize

individual Soldier contacts.

" Minimize the administrative burden, particularly those functions that require Soldiers

and in-house staffing. SQT/SDT was a task-based, high maintenance system relying

almost completely on in-house test development expertise, resources and production,

plus SME augmentation. Technology, contracting, and new test design strategies can

make a difference.

" Recognize testing as one part of the whole Soldier performance system. Testing

can enhance accountability and provide both individual and unit measures of

performance, but individual assessment testing needs to be a part of a larger Soldier

development program - testing is not an end-all in itself.

Some may perceive that the SQT/SDT program was a failure and, from an operational

standpoint, the logistical requirements made the program unwieldy. From a testing

viewpoint, however, it was a highly successful program. Hundreds of thousands of Soldiers

were accurately and consistently measured in critical job performance categories. Low

performers were identified and their skills were improved or the Soldiers were removed

from the system. Equally as important, high performers were recognized and promoted.

Soldiers were provided with specific expectations and leaders had a means of measuring

achievement of those expectations. In the end, operational problems, including costs and

support requirements, forced almost constant revision in the program, and, ultimately,

many of these unresolved factors motivated decision makers to abandon skills testing in

the Army. The intrinsic quality and value of the program, however, is still recognized, both

inside and outside the Army. 15
@ Planning for the Future: Progress Toward an NCO Competency Assessment Program

Assessment In the Other Services:

We have spent some time with the other Services - particularly the Air Force and the Navy - examining their promotion testing

systems and procedures. We continue this working relationship as a means of exchanging ideas that may have Army applications.

US Air Force - The Air Force administers promotion examinations annually to all Airmen starting at E4 and continuing for promotion

to E9. The system (called the Weighted Airman Promotion System - WAPS) tests Airmen with two tests - a Promotion Fitness

Examination of common tasks, history, leadership, conduct, and customs and courtesies and a Specialty Knowledge Test that Airmen

take in their job specialty. Both tests are knowledge tests and are currently paper-and-pencil-bsed. Airmen are provided Study

Guides and references but all ttudy and preparation is an individual Airman's responsibility; group preparation Is prohibited. Test

development is centralized atRandolph Air Force Base,TX at a Flight with about 20 military and 35 dvilan members permanently

assigned. These are augmented on an annual bash by enlisted subject expes who are brought in 30-day temporary duty periods.

The Air Force administers about 200,000 tests annually to about 104,00 individual Airmen.

US Navy - The Navy administers promotion tests to all Sailors in grades E4 through ET. Tests are administered twice a year to E4 - E6

Sailors and annually to E7 Sailors. Two tests are administered - a Professional Military Knowledge test consisting of common subjects

such as personal safety and damage control, nudmblsiogicaland c it leadershilptrainlng, and Navy programs and

policies, and an Occupational test which Is job-speck The NavyteM apploximatuy 1S0job specialties (ratings) with a different test

in each pay grade for each rating. Test development is centralize at the Navy Advancement Center In Pensacola, FL with about 50

personnel assigned permanent party. This is augmented byalibut 85 enlisted subjectlexpertswhontlI very recently,were brought

Into Pensacola on 2-year test writing assignments. The Navy administers about 29,OOO tests each year. They test all E4- E6 Sailors

in a 3-hour window on the same day in March and In September and all E7 Sailors on the sume day in January. The Navy has recently

started prototyping computer testing.

US Marine Corps - Although it is not diretyused in theMarine jet4M promotion O system,the F1. Marne thog Corps EMrrnt( administers Ofcers an annual W01antest

called the Marine Corps Common Skills (MCCS) testO and

CW02, and Officrs 01 through0. O3ainTtteso The *Aol q 'Ot Incld n aiiio1 ctics,land navlgation,nudear,

biological1, and ch emica I s,W dose order drill.Aands-on evaluation is

provided; however, the for-record test Isa knowfedge.*ot4 aublaion. A minip passng soM of 0% Is required. Marines

who fall are mediated; however, th e firsttimere 4 lt te ofrecord. Scores are entered on profidency markings (E4) and

Fitness Reports for ES Marines and above. Unit test pesomancels aggregatedand cmmanders can obtain historical test records on

individuais. The MCCS comes in paper-and-pedl format. A electronicversion, With immediate soring and feedback, is also available

over distance learning (MarineNet) at the Marine Learning Resources Centers. Proponent for the MCCS is the Marine Corps Institute at

the Marine Corps Barracks in Washington, DC.

US Coast Guard - The Coast Guard is currently phasing into operation a revised education and qualification program, Enlisted

Professional Military Education (E-PME). Under this program all members of 20 rates (the Coast Guard equivalent of MOS) in grades

E4, E6, and E8 will be administered the Advancement Qualification Exam (AQE) which covers general military knowledge including

seamanship, first aid, survival, customs and courtesies, leadership and supervision, small arms, and security. All members must

complete a series of rating-specific hands-on performance qualification tasks and members in grades E4 and B. also must pass a

rating specific correspondence course. Members who meet all ether eligibility requirements and satisfactorily complete the E-PME

and qualification requirements may participate in the Service Wide Exam for advancementL The Coast Guard publishes official study

guides as part of the E-PME as well as a checkdist of all performance and knowledge requirements for each pay grade. Members

are encouraged to study collectively and there is local study In the unit. Test development is centralized at three Training Centers in

Petaluma, CA, Cape May, NJ, and Yorktown, VA. Subject matter experts assigned to the Training Centers on 3-4 year tours develop the

tests as part of their duty assignment. The Advancement Branch of the Personnel Service Center administers the tests.

16
U.S. Army Research Institute

What Happens Next?

The Short Term Outlook - The Long Term Vision

The PerformM21 project is in progress

and much work remains to be done. The NCO using on-board

first 18 months of the project have been comusing ard

very productive in demonstrating both computer; assessment

what is needed and what can be done helps ensure

in the area of an Army core promotion accountability and

assessment. Consistent with the "Research meeting performance

to Operation" conceptshown on page 5, standards.

the time is fast approaching for this aspect

of the assessment to be handed over to Army leadership for implementation decisions.

Meanwhile, the second half of the project should see concentrated efforts in the

following areas:

"* Continuation of ATPAT meetings on a quarterly basis to provide up to date field

and user input.

"* Development of a cost analysis of various implementation options along with the

identification of potential benefits criteria.

"• Review and incorporation of operational requirements and lessons learned from

Iraq, Afghanistan, and the Global War on Terrorism as a basis of recreating the test

blueprint for the Army core assessment.

"• Refinement of the specifications for technology and system requirements to

support Army-wide computer-based testing.

"• Exploration of alternatives in MOS assessment to include:

- Candidate jobs for full prototype development.

- Testing methods and enhancements through technology and low-cost

simulation.

- Situational judgment and decision-making testing.

- Approaches for skill level implementation and job coverage.

- Job analysis and test content definition strategies.

17
Planning for the Future: Progress Toward an NCO Competency Assessment Program

One Army

From the outset, an operational principle of the Army's new assessment program has been that it

include, at the start, all eligible Soldiers, including AC as well as RC (US Army Reserve and Army National

Guard). All Soldiers will take the same test and encounter the same requirements.

This does not mean that there would be a single program or that there would be Just one path to

implementation. The RC is different in many ways and preliminary planning is addressing the operational

implications of inclusion, while still recognizing and working with the unique factors affecting the RC.

"Wemust insure that the USAR and the ARNG are included in the planning and given adequate time to

react to the implementation of a et am, to integrate the requirem ents Into our training schedules,

and to conduct the pMwrotton an ftns w4W nor ape tional sd*Is. What the Active Component

can do in a vetyshort period of tfime is moanhled by' two or three due to the characeristics of the Reserve

Components. It Is something we can live with, but it needs recognition and planning.'

"Conversey this is a program that USAR mnht take the lead on in implementaton, rather than waiting to

follow the practkes of the AC Our recent eperientes with mobilbro and deployments within the USAR

have caused concern about readiness in some ef the Soldier and Leadership skill sets. This test program

could boost both the emphasis and accountability of performance requireients that are critical to the

USAR mission.'

CSM Nick Piacentini

United States Army Reserve Command (USARC)

18
U.S. Army Research Institute

The closing point of our long-term vision is best made by referring to the work

performed by a small group of senior NCOs who have done some serious thinking

about the future of the NCO corps. Called "A Concept for Growing the 21st Century

NCO Corps'" this model of development is shown below. Like all concepts, this is

emerging, subject to frequent updating, and does not emulate recent doctrine or policy.

But it does stimulate thinking and demonstrates the need for long-term goals. It is

critical for readers to comprehend the theme of changing requirements and career

development paths for NCOs depicted in the model. All the elements portrayed in

the NCO Development Model are crucial for NCO growth and Army success; no one

element is intrinsically more important than another. Promotion testing is one of those

elements and the focus of our work. But it is important not to view assessment testing

as disconnected, divergent, or an option. Like all the other pieces, it helps make up the

whole that is the NCO, and the Army of the future - a future that has already begun.

NCO Development Model

n A nt Tests

Cdla~• on So e p tier/ncOe

Vit a e ssIn ssie sndvua competene na

Leader-Tra ner Job Linked

to-Proomgtnon

Leader &5.21dier efarteor ccin t eader Coturse

yIntemmeeior Leader Corurse

"Ceveulaboearion Plans achieved tr - anabesecesto

NCOFR~ounselin waro skilnt Tie topromotiony

OpiptottuniNC forrps deeo

, on , & • ,! 

tor ) PerscloefinscSuecs arnd 

essionad = on D ev t rElaten Col y a r IAm Eoinbteles oco issioand O

ie oversOn lenti prt is OpLeOne; Ed on s te Or nopment

atO hoetto and Cg -r ealitic

Virevitoaaan constructive1 C

peraonmnn dutiesion Devmultiple

"Asgne and ts -leading

Soldier dotigmdployent

a TO/O ilo Assigvment!Functio arire 

0 SGO/BSNEoCATSEt MACGM Staff/G

Technica StanchSpecifi

anamofesionaDeedlopment PrE relati Stem; CMrmy CaerMaaee Field t C JomaTrinin et er&S is egat SCC BfeSafNncm isoe fie

ATIA-M = Army Training Information Architecture - Migrated; NCOFR Noncommissioned Offlrer Evauotion Report; NCOPD = Noncammissioned Officer Professionai Development.

19
Planning for the Future: Progress Toward an NCO Competency Assessment Program

20
U.S. Army Research Institute

Your Turn: Feedback?

This research program is important to decisions the Army may make that affect Soldiers

in the future. Equally important is that you get a chance to be heard. Comments and

reactions are welcomed. Please take the time to let us know your thoughts by sending

them to ARI, as follows:

Mail:

Dr. Tonia Heffner USAR Soldier completing

DAPE-ARI-RS the competency

2511 Jefferson Davis Highway assessment in a digital

Arlington, VA 22202-3926 training facility.

Email:

tonia.heffner@hqda.army.mil "

Costs of an Army Assessment Program

No change comes without a price tag, and costs are not always dollars and cents issues. Some associated

testing requirements:

* An Army infrastructure to support testing is required.

• Soldiers will need time and resources to prepare for testing and to test.

, Personnel and promotion policies will have to be changed; Soldiers will need to adjust.

• Accurate, up-to-date study materials must be made available.

An Army at War

When the initial interest in Army job testing was generated in 2002, the operational commitments and

priorities of the Army were far different than what exists in 2005. Is testing a viable concept for an Army

at war? The answer is YES. Testing drives rapid dissemination of Lessons Learned and tasks derived

from combat experiences. And, if anything, the demands of increased operational tempo highlight the

requirement to promote first those who are best qualified to advance to the NCO ranks.

Without doubt, increased operational commitments and deployments involve special testing issues that

must be addressed. Wartime demands may alter the schedule for advancing and adopting the Soldier

assessment program, but competency assessment is a long-term program focused on meeting challenges

and providing for the continuing betterment of the NCO corps. Development needs to advance even

when priorities are concentrated elsewhere - the program needs to be ready when the Army is ready.

21
, Planning for the Future: Progress Toward an NCO Competency Assessment Program

22
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from... to)

November 2005 Interim November 2003 - February 2005

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

Planning for the Future: Progress Toward an NCO Competency DASWO1-03-D-0015/DO #5

Assessment Program

5b. PROGRAM ELEMENT NUMBER

622785

6. AUTHOR(S) 5c. PROJECT NUMBER

Roy C. Campbell (Human Resources Research Organization), A790

Tonia S. Heffner (U.S. Army Research Institute), and

Deirdre J. Knapp (Human Resources Research Organization) 104

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

Human Resources Research Organization

66 Canal Center Plaza, Suite 400

Alexandria, VA 22314

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

U. S. Army Research Institute for the Behavioral & ARI

Social Sciences

2511 Jefferson Davis Highway 11. MONITOR REPORT NUMBER

Arlington, VA 22202-3926 Special Report 61

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

Contracting Officer's Representatives: Peter Greenston and Tonia Heffner. Subject Matter POC: Tonia S. Heffner.

14. ABSTRACT (Maximum 200 words):

In the early 1990s, the Department of the Army abandoned its Skill Qualification Test (SQT) program due primarily

to maintenance, development, and administration costs. This left a void in the Army's capabilities for assessing job

performance qualification. To meet this need, the U.S. Army Research Institute for the Behavioral and Social

Sciences (ARI) instituted a 3-year program of feasibility research related to development of a Soldier assessment

system that is both effective and affordable. The PerformM21 program has two mutually supporting tracks. The

first focuses on the design of a testing program and identification of issues related to its implementation. The

second track is a demonstration of concept - starting with a prototype core assessment targeted to all Soldiers

eligible for promotion to Sergeant, followed by job-specific prototype assessments for several Military

Occupational Specialties (MOS). This report describes the PerformM21 feasibility research program, which is

now at the end of the second year of its 3-year plan.

15. SUBJECT TERMS

Behavioral and social science Personnel Job performance measurement

Manpower

SECURITY CLASSIFICATION OF 19. LIMITATION OF 20. NUMBER 21. RESPONSIBLE PERSON

ABSTRACT OF PAGES (Name and Telephone Number)

16. REPORT 17. ABSTRACT 18. THIS PAGE Unlimited Ellen Kinzer

Unclassified Unclassified Unclassified Technical Publications Specialist

(703) 602-8047

Standard Form 298
-- I r .1

AA.;

1: j~

N Is