NAEP Process Data: 

Past, Present, and Future 

Emmanuel Sikali, Ph.D.

Senior Research Scientist/Mathematical Statistician, NCES
Agenda 

2

• Overview • Introduction to NAEP assessments • Change of Assessment in Digital Era • NAEP Process Data • Origin • Collection and Cleaning Process • NAEP Process Data: Potential • Students’ Test Taking Behavior • Item/Test Development and Scoring • Assessment Features • Assessment Accommodations • Process Data Community 
3

Overview: NAEP and NAEP Assessments
National Assessment of Educational Progress (NAEP)

4

• NAEP is a congressionally mandated assessment and serves as an integral 

part of our nation’s evaluation of the condition and progress of education.

• NAEP is the largest nationally representative and continuing assessment of 

what America's students know and can do in various subject areas. 

• NAEP is required under the Elementary and Secondary Education Act of 

1965, which was reauthorized as the Every Student Succeeds Act of 2015.

• The first national assessments were held in citizenship, science, and writing 

to 9-, 13-, and 17-year-olds in 1969.
Subjects Assessed for NAEP

5

• NCES administers NAEP assessments in public and nonpublic (private) schools 

across the nation. 

• Four subjects- mathematics, reading, science, and writing- are assessed most 

frequently and reported at the state and district level. 
Sampling for NAEP

6

• NAEP is designed to report results at the national and state level, as 

well as for selected urban districts by creating a sampling frame. 
7

WE ARE LIVING IN A DIFFERENT ERA 

PAST PRESENT FUTURE

Item development Labor intensive Labor intensive Automatized

Item types Generic Enhanced Real-life

Test design Static Semi-static Data-driven

Test assembly Labor intensive Semi-automatized Automatized

Accessibility Limited Universal design Adaptive

Timing Not measurable Measured Used

Pathways Not observable Observable Modeled

Validity Content/Corr based Construct based Process based

Feedback Summative Summative Diagnostic
8

Overview: NAEP Process Data 
9

Examinees’ real time interactions with the digital test 

system environment that are recorded in the 

background as timestamped events. 

11
• Insights into

• students’ testing experiences,

• problem solving behaviors (e.g., guessing, skipping pattern of items)

• misconceptions

• metacognitive processes (e.g., item revisits)

• motivation, persistence, and engagement

• Data supported operational decisions

• item development, analysis and selection

• questionnaire development and validation

• block and test assembly optimization

• understanding test administration conditions

• Insights into learners’ needs, accommodation use and effect

• Enhancing the communication of what assessments measure

• Modeling cognitive and behavioral processes, advancing

psychometric methods or building new IRT models, developing a

framework for process data use

• Advancing research, item/test development, reporting,

teaching/learning practices, and decision making

Benefits and Challenges of NAEP Process Data

Collecting data with more expert input may increase potential utility of

data

Process data is noisy, complex, and requires detailed exploration

Standards are under construction

Field has limited access to NAEP process data which inhibits forming

a research community centered around process data

Limited platforms/systems that can be integrated across various

assessments

OPPORTUNITIES CHALLENGES

12
13

NAEP Process Data: Origin 
14

Logging Data Checking and 

Cleaning Variable Derivation

• Data structure: 

Structured data

• Collection of pre-decided 

actions 

• Data Codebook : 

Not standardized 

Data validation 

• Correctness and 

consistency of data sets 

• Within data set 

• monotonically increasing 

time stamps

• duplicated cases 

• Across data sets 

• consistency with item 

maps

• Security

• item content, hate speech 

• Merging with response data

Data Preparation for 

Analysis 

• Derivation of variables 

• Time spent 

• Accommodation 

Use 

• UDE use 

• ...
Data Infrastructure 

15
Example of Captured Events[Mathematics]

16
Captured vs. Derived Variables

17

Captured Derived 

Student Identifier Cumulative time 

Block Code Number of visits (e.g., 1,2,3,4 ) 

Accession Number Calculator Use (yes-no)

Item Type Code Response change (e.g., A->B)

Observable Type .... 

Extended Info

Timestamp
Data Management 

Data Security

Data Privacy

Data Quality 

18

Beyond Data 
19

NAEP Process Data: Potential 
20

Today’s Examples

Students’ Test Taking Behavior

Item/Test Development & Scoring

Assessment Features 

Assessment Accommodations

Data & Semantics 

Identify & operationalize the variables

Generate defensible variables 

Guide data collection 

Provide base for the use of the data
Students’ Test Taking Behavior: Disengagement

21

Our Research:

Can finite mixture modelling techniques 

be used on RT to identify distinct groups 

of examinees with different testing 

behaviors, including disengagement 

and speededness?

Key Findings:

Distinct behaviors can be identified. 

Meaningful and plausible interpretations 

can be made about the identified groups.

Zheng, X. & Kim, Y. Y. (2019)
Item/Test Development & Scoring : Non-Response Rates 

Hicks, J.., & Circi, R. (2019) 22

Our Research: 

Explore how process data helps to 

evaluate the appropriateness of the 

conventional definitions for NR and 

Omit 

– Do NR and Omit rates differ 

between scoring files 

(conventional) and process data? 

– Can we identify a threshold 

between NR and Omit for each 

item using RT?

Item sequence Item type Response Time Response File 

Coding

1 MatchMS 69.44 Incorrect

2 MCSS 20.26 B

3 Composite 108.44 Correct

4 FillInBlank 212.27 Incorrect

5 MCSS 45.39 A

6 MatchMS 82.32 Incorrect

7 MCMS 32.20 Partial

8 CompositeCR 348.30 Omitted

9 ZonesMS 85.45 Correct

10 CompositeCR 449.58 Partial

11 MatchMS 339.15 Not reached

12 CompositeCR NA Not reached

13 MCMS NA Not reached

14 CompositeCR NA Not reached

Key Findings: 

Non-response rates between student scoring 

file and process data differ 
Digital Assessment and Accessibility

Hicks, J., Circi, R., & Burhan, O. (2021). 23

Our Research: 

NAEP DBAs implement technologies 

to improve accessibility for all 

participants with different learning 

backgrounds.

Key Findings: 

Students with accommodations use more of 

drawing and eliminator choice tools 
Assessment Accommodations: Extended Time Accommodation

24

Our Research: 

Exploring extended time accommodation 

(ETA), by analyzing the relationship 

between ETA use and performance of 

students with ETA

Key Findings: 

Only one-third of ETA students (35%) used 

ET

On average, ETA students who used extra 

time scored 2 points higher than those ETA 

students who did not use extra time

Kim, Y. Y., & Circi, R. (2019)
Assessment Features: Calculator Use Study

25

Our Research:

• What computations students do with the 

calculators?

• How often do students use computation 

strategies or commit common errors 

identified by experts using the calculator?

Key Findings:

• Percentage of students who exactly follow 

anticipated solution strategies or errors vary.

• Most common computations were not exactly 

matching with the anticipations. Most non- matching computations were one character 

different than anticipated. Unanticipated 

errors were also found.

Sahin, F.., & Liao, M. (2019)
Process Data Community Building: 2019 - 2021

26
Process Data Community Building: Special Interest Groups 

(SIGs)

27

NCME: Big Data in Educational Measurement 

(Join: https://form.jotform.com/ncme/SIGIMIESIGNUP )

Leverage the availability of big data from a 

variety of sources to inform the study of 

education and educational measurement
For your questions 

28

Contact information:

emmanuel.sikali@ed.gov