Click to edit Master subtitle style

12/7/2023 1 1

NAEP Alliance

Linking NAEP and COVID Data Hub

Katherine Castellano

Carol Eckerly

Helena Jia

Educational Testing Service
2

Background

• Schools abruptly shifted to remote instruction in 

March 2020 due to COVID-19

• School reopening policies varied widely by locale 

• Infrastructure wasn’t in place to collect data on 

school learning mode (i.e., in-person, hybrid, 

virtual) in a systematic way across the country

• Instructional mode data is complex to collect

• What is the definition of mode?

• Is mode available at all or to specific groups?

• What is actual enrollment by mode?

• How often does this information change?

• Is the source of the data reliable and complete?

2
3

Background

• Despite challenges, there has been great public interest in 

linking learning mode to student achievement to evaluate 

the impact of the pandemic related closures

• The National Center for Education Statistics (NCES) has 

two projects currently underway to link external data 

sources on learning mode to 2022 National Assessment of 

Education Progress (NAEP) achievement data

• 2021 Monthly School Survey (MonthlySurvey) Linking Study

• Covid School Data Hub (DataHub) Linking Study

• With researchers using different data sources on 

instructional mode to address similar research questions, it 

is imperative to:

• Understand relationships between the data sources

• Effectively communicate potential nuance in conclusions to 

the public

3
4

Outline

• Data sources providing information about 

mode of enrollment (e.g., in-person, hybrid, or 

remote)

• COVID School Data Hub

• NAEP Monthly School Survey

• NAEP assessments

• Describe relationships between data sources

• Spotlight on two states

• Conclusion

4
12/7/2023 5

Data Sources

5
6

Understanding the variables

• Data sources generally get at one of two 

dimensions related to instructional modes during 

the 2020-21 school year 

• Time an instructional mode was offered

• Such as percentage of school year or number of months 

• Volume of an instructional mode

• Such as student enrollment or number of classes for a 

particular instructional mode 

6
7

• The COVID School Data Hub (DataHub) is a publicly 

accessible database of COVID-related data collected and 

disseminated for the 2020-2021 school year by Brown 

University researchers

• Provides percentage of the school year students primarily 

had access to in-person, hybrid, or remote instruction 

• Time offered variable 

• Available for most schools in 48 states

• The interval of data collection varies from weekly to biannually

• Reflects the most common learning model in use for the interval

• Each state used their own processes (e.g., surveys, data system) 

to collect information on offered instructional modes 

• Note that 100% of the year in-person does not mean that all 

students were in-person learning all school year but just that it was 

offered all year and a common mode (though “common” was 

defined differently by state)

COVID School Data Hub
8

COVID School Data Hub

Source: covidschooldatahub.com
9

NAEP Monthly School Survey

• On January 21, 2021, President Biden issued the Executive Order 

on Supporting the Reopening and Continuing Operation of 

Schools and Early Childhood Education Providers

• Among other tasks about supporting the safe reopening of schools, 

the order tasked the Secretary to 

coordinate with the Director of the Institute of Education Sciences 

to facilitate...the collection of data necessary to fully understand the 

impact of the COVID-19 pandemic on students and educators

• IES made use of the already-drawn NAEP sample to survey 

schools about the current status of instruction. 

• Only reached out to 50% of sample due to pandemic and associated 

costs

• The 2021 NAEP Monthly School Survey (MonthlySurvey) 

questionnaire was administered to sampled schools each month 

between February and June 2021.
10

Monthly 

School 

Survey –

Questions 

#1, 2 
11

NAEP Monthly School Survey Derived 

Variables

• The survey questions were used to derive variables 

about duration of in-person learning in the 5-month 

window

• MSS Q1: Was in-person instruction never offered, offered 

starting Feb-May, or offered all months?

• Time offered variable

• MSS Q2: What was the average percent of in-person 

enrollment across the 5 months?

• Volume variable but averaged over time 
12

NAEP Assessments

• Main NAEP (grade-based) assessed in January-March 

of 2022

• Survey question from the NAEP School Questionnaire 

administered to school administrators at participating 

NAEP schools

• Volume variable 

12
12/7/2023 13

Relationships Between 

Data Sources

13
14

Variables for comparison

• DataHub

• Percentage of 2020-2021 school year students primarily had 

access to virtual instruction

• Continuous time offered variable

• MonthlySurvey Q1

• Summary of available in-person instruction across five months 

• Categorical time offered variable

• MonthlySurvey Q2

• Average percent of student enrollment in virtual instruction 

across five months 

• Continuous volume (averaged over time) variable

• NAEP Questionnaire

• Amount of classes moved to distance learning 

• Categorical volume variable

14
15

Covid School Data Hub time offered variable vs.

Grade 4 NAEP Monthly Survey time offered variable

15

Tau = 0.50
16

Covid School Data Hub time offered variable vs.

Grade 4 2022 Main NAEP volume variable 

16
18

Grade 4 NAEP Monthly School Survey Q2 volume 

(averaged over time) variable vs.

Grade 4 2022 Main NAEP volume variable

18

Tau = 0.37
12/7/2023 19

Texas and Florida

19
20

Texas and Florida Covid-19 School Data 

Hub

• Both states had biannual reporting at the 

school level

• September 1, 2020 to December 31, 2020

• January 1, 2021 to June 1, 2021

• Instruction mode classifications: 

• In-person: On-campus enrollment counts were at 

least 10% of total enrollment

• Hybrid: Not collected

• Virtual: On-campus enrollment counts were less 

than 10% of total enrollment

20
21

Texas and Florida – February 2021

Source: covidschooldatahub.com
22

Texas grade 4: Differing Pictures

• Time offered

• DataHub

• 95% of schools (with non-missing in both DataHub and 

MonthlySurvey) have 100% of year with students primarily 

having access to in-person instruction (or 0% of year primarily 

access to virtual)

• MonthlySurvey

• Q1: 93% of schools report offering in-person instruction in all 

five Spring 2021 months

• Volume

• MonthlySurvey

• Q2: The average monthly percent remote enrollment is 32% 

with a max of 67%

• NAEP School Questionnaire

• 76% of schools (in both DataHub and NAEP) report some or 

more classes moving to distance learning

22
23

Florida grade 4: Differing Pictures

• Time offered

• DataHub: 

• 100% of the schools (with both Hub and MonthlySurvey data) 

have 100% of year with students primarily having access to inperson instruction (or 0% of year primarily access to virtual)

• MonthlySurvey: 

• Q1: 100% of common schools report offering in-person 

instruction in all five Spring 2021 months

• Volume

• MonthlySurvey

• Q2: The average monthly percent remote enrollment is 24% 

with a max of 67%

• NAEP School Questionnaire

• 69% of schools (in both DataHub and NAEP) report some or 

more classes moving to distance learning

23
12/7/2023 24

Concluding Remarks

24
25

Concluding Remarks

• Important to understand instructional mode data 

source and variable type

• Often easy to misinterpret 

• Particularly, easy to confuse time offered and volume 

• Improved data collection methods needed for 

recording school data in real time particularly for 

fast-changing data points 

• Choice of instructional mode variable can affect 

inferences, such as relationship with student 

achievement 

25
12/7/2023 26

Supplemental Slides

26
27

NAEP Monthly School Survey

Deriving MSS1 variable I

Category

Remote only

Remote + Hybrid

Hybrid only

Remote + In person

All 3 modes

Hybrid + In person

In person only

Can be reported roughly from 

least to most in person
28

NAEP Monthly School Survey

Deriving MSS1 variable II

• MSS Q1 Duration Variable: Percentage of months in which 

school offered at least some full time in-person instruction (for 

schools with responses in at least four out of the five months)

• Categories: Never offered, offered starting Feb-May, or always 

offered full time in-person instruction

• Example of school responses associated with a reporting 

category of “offered starting Feb-May”

Category Month 1 Month 2 Month 3 Month 4 Month 5

REM only X X

REM + HYB

HYB only

REM + INP X X

All 3

HYB + INP

INP only
29

Grade 4 association summary

CSDH MSS Q1 MSS Q2

MSS Q1 0.50 -- --

MSS Q2 0.72 0.41 --

NAEP 0.47 0.20 0.37

Kendall's Tao

Pearson's R

Cramer's V