From policy to practice:

Insights from U.S. Army administrative 

data into Soldier performance

Federal Committee for Statistical Methodology Research and Policy Conference

October 24, 2023

Joanna Schroeder, Skylar Haskiell, Joel Thurston

Social and Decision Analytics Division (UVA-SDAD)

Army Research Institute for the Behavioral and Social Sciences (ARI)
The research described herein was sponsored by the U.S. Army Research Institute for the 

Behavioral and Social Sciences, Department of the Army (Cooperative Agreement No. 

W911NF19-2-0164 ). The views expressed in this presentation are those of the authors and do 

not reflect the official policy or position of the Department of the Army, DOD, or the U.S. 

Government.
Project: Use modern data science techniques to develop models that integrate 

existing DOD data to make predictions about Soldier behavior and performance.

Payoff: Knowledge about how best to utilize data from disparate sources to form a 

holistic picture of Soldier and unit performance that can be used to: 

• Improve training

• Identify informative performance metrics 

• Optimize talent management decisions across Soldier lifecycle

Conceptual 

mapping

Year 1 Year 3

Year 2 Year 4

Year 5

Data discovery 

and preparation

Statistical 

development of 

constructs

Construction of 

statistical models

Fill gaps/

Validate models

- + +

Leveraging Archival Data to Predict Performance
Start with the literature

• 1980 congressional mandate led to seminal research 

performance (e.g., Project A)

• Widely adopted by those outside military (e.g., 

academia, business)

• Researchers developed frameworks

Frameworks: Campbell et al., 1993 & 2015 and 

Koopmans et al., 2011

• Performance is multidimensional, abstract, latent 

construct 

• Manifests as behavior  something people do and 

can be observed

• Behaviors relevant to organizational goals

What is Performance?

Soldiers with Kentucky's CBRN Enhanced Response Force package 

(CERFP) breech and break team use a hammer drill to extract a victim 

from a rubble pile during an exercise evaluation at Camp Blanding, Fla., 

9 January 2019. U.S. Army National Guard photo by Sgt. Taylor Tribble. 

https://www.flickr.com/photos/soldiersmediacenter/46128506494/

A U.S. Army Soldier provides perimeter security during an Iraqi police 

dismounted presence patrol in Kirkuk, Iraq, on 24 April 2007. The Soldier 

is from Charlie Company, 2nd Battalion, 35th Infantry Regiment, 3rd 

Infantry Brigade, 25th Infantry Division. DoD photo by Tech. Sgt. Maria J. 

Bare, U.S. Air Force. 

https://www.flickr.com/photos/soldiersmediacenter/497978091/
Conceptual Performance Framework
Timeline of Major Events Relevant to the U.S. Army from 2000-2019
How does U.S. Army policy & doctrine impact or contextualize the 

description of behaviors related to individual and team performance?

Analyses:

• Extract performance themes from Army policy and team science 

literature

• Examine changes in the conceptualization of performance in Army 

policy over time

• Analyze Army policy for mentions of performance behaviors

Research Question
How does U.S. Army policy & doctrine impact or contextualize the 

description of behaviors related to individual and team performance?

Analyses:

• Extract performance themes from Army policy and team science 

literature

• Examine changes in the conceptualization of performance in Army 

policy over time

• Analyze Army policy for mentions of performance behaviors

Research Question
Army administrative documents

• Officer Professional Development and Career 

Management (PAM 600-3) 

• Army Leadership and the Profession (ADP 6-22)

• Evaluation Reporting System (PAM 623-3)

• Performance Evaluation Guide - Leadership 

Requirements Model and Example Behavioral 

Indicators (AR-623-23)

• Building Cohesive Teams Handbook

• Leader Development (FM 6-22)

• Leaders Guide to Team Building Handbooks

• Soldier’s Guide (TC 7-21.13)

Team science literature

• Svyantek, D. J., Goodman, S. A., Benz, L. L., & 

Gard, J. A. (1999). The relationship between 

organizational characteristics and team building 

success. Journal of Business and 

Psychology, 14(2), 265-283.

• Cohen, S. G., & Bailey, D. E. (1997). What makes 

teams work: Group effectiveness research from 

the shop floor to the executive suite. Journal of 

management, 23(3), 239-290.

Performance Themes: Corpus
Performance Themes: Methods

• Manual Qualitative Coding

• Initial coding by a team of 4 (inter-rater reliability tested with a group of 5)

• Performed in Dedoose

• Topic Modeling

• Latent Dirichlet Allocation (LDA)

• Bayesian (probabilistic), unsupervised

• Assumes topics consist of underlying sets of words 

• Performed using the topicmodels package
Qualitative Analysis Yields Performance Themes

• Identified 320 excerpts

• Tagged with 506 applications of 18 codes
Topic Modeling Triangulates on Performance Themes

Manual Codes

• Qualities of a Leader

• Leader Development

• Organizational Trust

• Diversity in Team

• Negative Qualities

• Team Success

• Shared Vision

• Team Building

• Responsibility

• Cooperation

• Influence

• Intellectual Efficiency

• Negation of Positive 

Qualities

• Adaptability

• Team Orientation

• Commitment to Serve

• Communication

• Integrity

Bolded codes also appeared in automated 

analysis

Results of the 18 

Term Topic 

Model

• Many manual 

codes appeared in 

topic models

• Adaptability

• Commitment to 

serve

• Leader 

development

• Some manual 

codes did not 

appear in topic 

model

• Negation of Positive 

Qualities

• One topic did not 

appear in manual 

codes

• Family support
How does U.S. Army policy & doctrine impact or contextualize the 

description of behaviors related to individual and team performance?

Analyses:

• Extract performance themes from Army policy and team science 

literature

• Examine changes in the conceptualization of performance in Army 

policy over time

• Analyze Army policy for mentions of performance behaviors

Research Question
• 45 Army Administrative documents from 1946-2021,

including

• ADP 6-22: Army Leadership and the Profession

• Establishes the Army's concepts of leadership and 

leader performance (~130 pages)

• Items from the Global Assessment Tool (GAT) assessing 

social characteristics of Soldiers, e.g.

• Spiritual fitness – “It is very much like me that my 

life has lasting meaning.”

• Friendship – “I do have someone to talk to when I 

feel down.”

• Bonus: Center for Army Profession and Leadership 

(CAPL) document on the thematic changes in most of 

our corpus

Performance Concept Over Time: Corpus
Performance Concept Over Time: Methods

• Large Language Models (LLMs)

• Use deep learning to analyze vast amounts of text

• Can be trained to perform general or specific tasks

• Bidirectional Encoding using Transformers (BERT)

• Captures the contextual relationships between words in a sentence and 

across documents

• Output is a numerical representation of documents (vectors)

• Performed using bert-based-uncased

• Pretrained model on English language

• Also performed using GPT-2

• We compared documents by a distance measure (cosine similarity)

• Also compared by clustering (using K-nearest neighbors )
The Concept of Performance Stays Relatively Similar Over Time

• Only three performance documents have a mean cosine similarity less than 0.6

• Functional Manuals from 1957, 2002, and 2010

• There is slightly more cohesion in performance concepts since 2012

• Example of conceptual dissimilarity of the ADP 6-22: Army Leadership and the 

Profession (2017) and the AR 600-3: The Army Personnel Development System 

(1993)

• Cosine similarity of 0.6

ADP 6-22 (2017) AR 600-3 (1993)

Leadership is the activity of 

influencing people by providing 

purpose, direction, and motivation 

to accomplish the mission and 

improve the organization

Leadership is the process of 

influencing others to accomplish 

the mission by providing purpose, 

direction, and motivation

CAPL Definitions of Leadership
The Concept of Performance Stays Relatively Similar Over Time

• Only three performance documents have a mean cosine similarity less than 0.6

• Functional Manuals from 1957, 2002, and 2010

• There is slightly more cohesion in performance concepts since 2012

• Example of conceptual dissimilarity of the ADP 6-22: Army Leadership and the 

Profession (2017) and the AR 600-3: The Army Personnel Development System 

(1993)

• Cosine similarity of 0.6

ADP 6-22 (2017) AR 600-3 (1993)

Leadership is the activity of 

influencing people by providing 

purpose, direction, and motivation 

to accomplish the mission and 

improve the organization

Leadership is the process of 

influencing others to accomplish 

the mission by providing purpose, 

direction, and motivation

• Leadership as an activity (versus 

process)

• Influencing people (versus 

others)

• Addition of improving the 

organization

CAPL Definitions of Leadership
BERT Does Not Discern between 

Positive and Counterproductive 

Performance Measures GAT Item Cosine Similarity

adaptability 1

bad coping 1

catastrophizing 0.94

character 0.94

depression 0.99

family closeness 0.92

family satisfaction 0.92

family support 0.88

friendship 0.91

good coping 0.91

life meaning 0.92

loneliness 0.9

negative affect 0.92

non-work interests 0.89

optimism 0.89

organizational trust 0.84

positive affect 0.97

work engagement 1

“I am extremely 

satisfied with my 

marriage”

“I am not at all 

satisfied with my 

marriage”

• Overall the item text of positive and 

counterproductive psychosocial characteristics are 

highly similar

• Potential issues

• Model specification

• Translation of item text

Cosine Similarity between the Positive and 

Counterproductive Characteristic for Each 

Measure of the GAT
GAT Item Cosine Sim. GAT Item Cosine Sim.

adaptability 0.99 adaptability 0.99

bad coping 0.91 bad coping 0.92

catastrophizing 0.9 catastrophizing 0.9

character 1 character 0.99

depression 0.9 depression 0.94

family closeness 0.99 family closeness 0.91

family 

satisfaction

0.91 family satisfaction 0.86

family support 0.89 family support 0.91

friendship 0.91 friendship 1

good coping 0.89 good coping 0.91

life meaning 0.89 life meaning 0.89

loneliness 0.92 loneliness 0.99

negative affect 0.89 negative affect 1

non-work 

interests

1 non-work interests 0.89

optimism 0.84 optimism 0.99

organizational 

trust

0.89 organizational trust0.89

positive affect 0.91 positive affect 0.91

work engagement1 work engagement 0.91

Performance Measures Align 

Conceptually with the 

ADP 6-22: Army Leadership 

• Overall psychosocial characteristics are 

highly conceptually similar to the AR-623

• Minimum cosine similarity of .84

• Both positive and counterproductive 

performance

“I am 

extremely 

satisfied with 

my marriage”

Cosine Similarity between Each Measure of the GAT 

and the ADP 6-22
How does U.S. Army policy & doctrine impact or contextualize the 

description of behaviors related to individual and team performance?

Analyses:

• Extract performance themes from Army policy and team science 

literature

• Examine changes in the conceptualization of performance in Army 

policy over time

• Analyze Army policy for mentions of performance behaviors

Research Question
• STP 21-1-SMCT (Soldier’s Manual of Common Tasks: Warrior Skills Level 1) from 

2021

• One of several training documents for Soldiers

• Describes training plans

• Summarizes Soldier tasks

Performance Behaviors: Corpus
Performance Behaviors: Methods

• Verb Extraction and Co-occurrence

• Tagged parts of speech and extracted verbs, extracted co-occurring words in 

sentences

• Performed using udpipe

• Distinguishes part-of-speech in context of a sentence

It is important for a Soldier to 

use good judgement*

A Soldier knows the importance 

of the use of good judgement*

*Example not reflective of text in Soldier’s Manual of Common Tasks: Warrior Skills Level 1 
Performance Behaviors: Methods

• Verb Extraction and Co-occurrence

• Tagged parts of speech and extracted verbs, extracted co-occurring words in 

sentences

• Performed using udpipe

• Distinguishes part-of-speech in context of a sentence

It is important for a Soldier to 

use good judgement*

A Soldier knows the importance 

of the use of good judgement*

Use = Verb in context Use = Noun in context

*Example not reflective of text in Soldier’s Manual of Common Tasks: Warrior Skills Level 1 
The Warrior Skills Manual Contains

Examples of Performance Behaviors Lemmatized Verb Count

use 435

figure 270

go 245

identify 214

do 202

perform 155

have 150

move 129

ensure 126

provide 123

• Overall, 13.2% of words in the Warrior 

Skills Manual are tagged as verb part of 

speech

• 1,179 unique verbs

• Co-occuring words 

• E.g. “use” cooccurs with

• People e.g. (“Soldier”, “student”)

• Objects e.g. (“map”, “compass”)

• Qualifiers e.g. (“without”, 

“must”)

Counts of Each Unique Verb in the Warrior Skills 

Manual, Ordered by Frequency 

Counts of the Top 10 Most 

Frequent Verbs in the Warrior 

Skills Manual
Summary

• We find several topics relevant to Soldier performance

• Helps identify variables for selection in performance models

• Provides a backdrop for results of performance models

• We find that the way performance has been conceptualized by the Army has 

shifted over time, but has ultimately remained very similar

• Backed up by internal Army sources, e.g., CAPL

• We find that psychosocial characteristics and performance behavior are 

somewhat difficult to measure using large language models

• Not discerning between positive performance and counterproductive 

performance

• Relatively high cohesion overall

• Could be an issue of item-text writing or the generalized training dataset 

for models
Recommendations for Future Research

• Inform the development and interpretation of performance models

• Explore other uses of large language models 

• Training models on an Army-specific corpus

• Models trained to work on short text

• Showcase computational complements to large-scale human-led metadata 

tracking projects

• i.e., Tracking of leadership themes by CAPL