Click to edit Master subtitle style

12/7/2023 1 1

NAEP Alliance

Linking NAEP 2022 to the 2021 Monthly 

School Survey

Carol Eckerly, Robert Finnegan, Helena Jia, Laura Jerry 

Educational Testing Service

October 26, 2023
2

2

In-person enrollment 

increased from 38% in 

January to 52% in May 2021

Monthly School 

Survey

Spring ‘21 Fall 

‘20 Spring ‘20 Fall ‘21 Spring ‘22 Fall ‘22

NAEP Assessments

Historic declines in student 

performance

All U.S. public 

schools closed 

by March 25, 

2020

• The National Assessment of Educational Progress (NAEP) is a 

continuing and nationally representative measure of trends in 

academic achievement of U.S. elementary and secondary 

students in various subjects 

NAEP overview in 2021 and 2022
3

2022 NAEP assessment overview

Study Population Sample Collection Dates

Long-Term Trend, Age 9 Nation 410 schools

7,400 students

Jan. – Mar. 2022

Mathematics, Gr. 4, 8 Nation, states, TUDAs 11,000 schools 

227,200 students

Jan. – Mar. 2022

Reading, Gr. 4, 8 Nation, states, TUDAs 11,000 schools 

219,500 students

Civics, Gr. 8 Nation 410 schools 

7,800 students

U.S. History, Gr. 8 Nation 410 schools 

8,000 students

Long-Term Trend, Age 13 Nation 460 schools 

8,700 students

Oct. – Nov. 2022
4

Monthly School Survey

• On January 21, 2021, President Biden issued the 

Executive Order on Supporting the Reopening and 

Continuing Operation of Schools and Early Childhood 

Education Providers

• Among other tasks about supporting the safe reopening of 

schools, the order tasked the Secretary to

• coordinate with the Director of the Institute of Education Sciences to 

facilitate...the collection of data necessary to fully understand the 

impact of the COVID-19 pandemic on students and educators, including data on the status of in-person learning. These data shall be 

disaggregated by student demographics, including race, ethnicity, 

disability, English-language-learner status, and free or reduced lunch status or other appropriate indicators of family income;

• As part of the response, IES made use of the already- drawn NAEP sample to survey schools about current 

status of instruction. 

• The 2021 NAEP Monthly School Survey (MSS) 

questionnaire was administered to sampled schools each 

month between February and June 2021.
5

Monthly School Survey Linking Study

• Using the portion of the 2022 main NAEP operational 

sample belonging to the same schools that were sampled 

for the 2021 MSS, we will link the 2021 MSS school-level 

responses to the 2022 main NAEP student-level data.

• Outcome will be an NCES report to address the following 

overarching questions:

• How does student performance in 2022 differ by schools’ responses to 

instruction or learning mode offered in 2021?

• How does student performance in 2022 differ by schools’ responses to 

enrollment in instruction modes in 2021?

• A set of plausible values for the 2022 MSS Linking sample 

will be made available for NAEP secondary research
6

Outline

• Overview of 2021 MSS

• 2022 MSS Linking Study Reporting Goals

• Linking Study Methodology

6
12/7/2023 7

Overview of 2021 MSS

7
8

Overview of survey and questionnaire

• Surveys were administered to collect five months of data 

for January – May of the 2021 school year. 

• Results were reported monthly and with a final summary report

• The survey questionnaire provided data on the following 

topics:

1. To what degree did schools offer remote only, hybrid, or full-time in-person instruction?

2. What was the proportion of students overall and by various student groups enrolled in each 

instructional mode?

3. What was the attendance rate for each instructional mode by various student groups?

4. What type of hybrid model was offered?

5. On average, how many days were students offered in-person instruction under the hybrid

instructional mode?

6. How many hours of synchronous (i.e., live) instruction did students receive on average daily 

under the remote learning mode?

7. Which student groups received priority for in-person instruction?

8. What percentage of teachers have received at least one shot of a COVID-19 vaccine?
10

Monthly School Survey: Sample Design

• Sampled schools were those originally planned for 

the 2021 operational assessment in Reading and 

Math

• Referred to as the “Slim Sample”

• “Slim” refers to the fact that the sample was reduced by 

about 50%, primarily due to the pandemic and associated 

increased costs

• State Sample Sizes:

• 40 Schools: Target sample size for non-TUDA states

• 70 Schools: Target sample size for states with TUDAs

• 30 schools per TUDA

• No state had more than 1 participating TUDA
11

Monthly 

School 

Survey –

Questions 

#1, 2 
12

Key Findings: Modes Offered in March

• Almost 90% of schools offer 

some type of in-person 

instruction AND some type of 

remote option.

• In-person offerings have 

increased over the months.

• Regional differences observed, 

with students in West and 

Northeast less likely to have inperson instruction offering for all.

• Differences by locale also 

observed, with students in city 

and suburban schools less likely 

to have in-person offering for all.

12

Percentage of public schools with a grade 4 Feb Mar

Open for full-time or hybrid instruction to ANY students 79* 88

Open full-time for ALL students 51* 56

Not open for in-person instruction (Remote only) 21* 12

Offering hybrid to any students 37* 44

Offering remote-only to any students 90 89

Percentage of public schools with a grade 8 Feb Mar

Open for full-time or hybrid instruction to ANY students 80* 89

Open full-time for ALL students 46* 51

Not open for in-person instruction (Remote only) 20* 11

Offering hybrid to any students 44* 49

Offering remote-only to any students 88 87

Percentage of grade 4 students by instructional modes offered: March 2021 Percentage of grade 4 students by instructional modes offered: March 2021
12/7/2023 14

Linking Study Methodology

14
15

Linking 2022 with 2021 Monthly School Survey

40 

Schools 

from 

2021

40 new 

Schools

80 

Schools 

2021 2022

40 

Schools

Monthly 

School 

Survey

Performance 

School Q 

Teacher Q 

Student Q

Special analysis of 

relationship 

between 2022 

performance and 

2021 Monthly 

School Survey 

Create variables for reporting 

performance by, for example 

-Mode of instruction offered

-Mode of instruction enrolled

Note: Number of schools are per 

state
16

Monthly School Survey: Methodology

• 2022 student performance can only be described 

based on their schools’ responses to the 2021 survey

• 2021 MSS data cannot be linked to 2022 students 

(e.g., we don’t know details of attendance, 

enrollment, mode of instruction for the 2022 students)
17

2022 MSS Linking Sample

• Taking the 2021 MSS sample, a rule was developed to 

determine the set of responding schools for the 2022 

MSS linking sample.

• Responding schools: sampled public schools that 

responded to the MSS in months 2 and/or 3 (March 

and/or April 2021) to MSS Question 1 and with 

students in the sampled schools assessed with the 

2022 NAEP assessment.

• NAEP Sampling and Data Collection contractor 

produced a set of student-level weights for the 2022 

NAEP sampled students in the responding schools

• Student weights were adjusted for school 

nonresponse 17
18

MSS Linking Study Score Reporting

• To estimate subgroup NAEP scores with subgroups 

defined by the MSS information, latent regression 

analysis was conducted by incorporating a set of 

variables from the 2021 MSS as model covariates

• Variables include:

• Derived variables based on school responses to MSS Q1, 

MSS Q2, and MSS Q6 (details on upcoming slides)

• Interactions with major reporting variables (e.g., gender, 

race/ethnicity, EL status, NSLP, SD status, region, school 

location, parental education).

• Analysis will support both national and state results.

• National results will include a caveat that a handful of small 

states are missing.

• State sample sizes are small; limited ability to contextualize 

state results
12/7/2023 19

MSS Linking Study 

Research Questions

19
20

Research Question #1: Mode Offered

• How does student 

performance in 2022 

differ by schools’ 

responses to 

instruction or learning 

mode offered in 

2021? 
21

Research Question #1: Mode Offered

Category

Remote only

Remote + Hybrid

Hybrid only

Remote + In person

All 3 modes

Hybrid + In person

In person only

Can be reported roughly from 

least to most in person
22

Research Question #1: Mode Offered

Category

Remote only

Remote + Hybrid

Hybrid only

Remote + In person

All 3 modes

Hybrid + In person

In person only

Multiple response paths 

form these categories
23

Research Question #2: Mode Enrollment

• How does student 

performance in 

2022 differ by 

schools’ responses 

to enrollment in 

instruction modes 

in 2021? 
24

MSS Linking Study Research Questions

• The research will allow for comparisons 

• Within jurisdiction and month, between MSS categories

• Within jurisdiction and MSS category, between months

• Within MSS category and month, between jurisdictions

• Provide information about duration of in-person learning in 

the 5-month window

• Was in-person instruction never offered, offered starting Feb-May, 

or offered all months?

• What was the average percent of in-person enrollment across the 5 

months?
25

Summary

• The 2021 NAEP Monthly School Survey (MSS) 

questionnaire was administered to schools each month 

between February and June 2021

• The 2022 MSS Linking Study will result in an NCES report 

linking student performance in 2022 to school responses 

to select 2021 MSS questions

• Secondary researchers will be invited to further explore 

how school-level variables related to instruction in 2021 

were associated with NAEP student achievement in 2022

25