AMERICAN INSTITUTES FOR RESEARCH® | AIR.ORG Copyright © 2021 American Institutes for Research®. All rights reserved.

Exploration of NAEP-SSOCS linkage: Investigating 

the Association between Behavioral Incidents, School 

Behavioral Climate, and NAEP Mathematics Achievement

FCSM| October 2022

Please do not cite or distribute without permission of the presenting authors. This project has been funded at least in part with Federal funds from the U.S. Department of Education under contract number ED-IES12-D-0002/0002. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education nor does mention of trade names, commercial products, or organizations 

imply endorsement by the U.S. Government. 

Saki Ikoma, Senior Researcher (AIR) | Paul Bailey, Senior Economist (AIR) | 

Jizhi Zhang, Principal Researcher (AIR) | Kelly Guinn, Senior Research Scientist (ICF) | 

Markus Broer, Managing Researcher (AIR)
| AIR.ORG

Agenda 

I. Contextual aspect

• Motivation of this study

• Data Linkage between NAEP and SSOCS

• Analytical approach

• Major takeaways on the criminological contexts

II. Technical aspect

• Underestimation of regression coefficients

• Plausible values and EdSurvey Dire Package

• Comparison of results between the original NAEP PVs and the Dire PVs

• Final thoughts

2
| AIR.ORG

I. Contextual aspect

3
| AIR.ORG

Motivation of this study

4

• Importance of school safety

• Criminological contexts in education

• Ongoing efforts on NCES survey linkage

• NAEP linking to the School Survey on Crime and Safety (SSOCS)
| AIR.ORG

Criminological contexts in education

5

• School behavioral environment and behavioral climate 

– Peguero & Bracy 2015; Ruiz et al. 2018

• Violence in school and surrounding school 

– Sharkey 2010; Mccoy, Raver, & Sharkey 2015; 

Carlson & Cohen 2015; Burdick-Will 2016

• School level spillover effects onto student level

– Hagan & Foster 2012; Lacoe & Steinberg 2019
| AIR.ORG

Data linkage using NCES surveys

6

• Linking NAEP to the Schools and Staffing Survey (Kaufman 1996)

• NAEP to the Trends in International Mathematics and Science Study (Johnson 1998)

• NCES surveys to state and federal administrative data (Gamoran 2016; Soldner 2017)

• Creating an ID crosswalk to link the Civil Rights Data Collection to other NCES data 

collections (Sable and Miller 2020) 

• Commissioned by NCES, AIR has examined the overlap samples of NAEP and the High 

School Longitudinal Study of 2009 as well as other datasets linked to schools.
| AIR.ORG

Why NAEP and SSOCS?

There is valuable information in SSOCS that is not included in NAEP, and this information can shed light on 

important and policy relevant associations between school level crime and behavioral indicators and 

student achievement.

7
| AIR.ORG

Data Linkage between NAEP and SSOCS

• Analytic sample description

– NAEP 2017 G8 Mathematics & SSOCS 2017-18

– ≈ 330 schools

– ≈ 8,470 students

• Bias analysis 

– distributions, t-tests

• Choosing weights

– NAEP weights

8

NAEP

SSOCS
| AIR.ORG

Analytical Approach

9

Merge NAEP 

and SSOCS

Bias analyses Exploratory 

analyses

CFA

Dire PV generation MLM Results comparison
| AIR.ORG

Major takeaways regarding the criminological contexts

• After taking into account student and school level 

characteristics, 

• no violent incidents within the school were 

significantly associated with student’s higher NAEP 

mathematics performance.

• However, the MLM results also revealed that 

the school climate index was not significantly 

associated with student’s mathematics 

performance. 

10
| AIR.ORG

II. Technical aspect

11
| AIR.ORG

Underestimation of regression coefficients

• When regression analyses are conducted with plausible values (PVs) as an outcome and 

explanatory variables from external sources, the regression coefficients will be 

underestimated due to the fact that the external variables are not included to produce 

the plausible values as regressors (Mislevy, Beaton, Sheehan, and Kaplan 1992; von 

Davier, Gonzalez, and Mislevy 2009; Wu 2015; Rubin 1987)

• The Marginal Maximum Likelihood (MML) model allows us to incorporate additional 

covariates in the model

12
| AIR.ORG

Why use Multiple Imputation on Test Scores

• To measure an individual student’s score we use a high reliability test, usually the 

variation of any student’s estimate is dominated by their ability, not noise

• These tests have too few items to be reliable for individual students

• If we applied the same test score we use for individual student assessments to the NAEP 

test form, we would get biased results (Mislevy, Beaton, Sheehan, and Kaplan 1992)

13
| AIR.ORG

Forming Plausible values: the likelihood

• This item the student got wrong, their 

likelihood is higher in lower scores.

14
| AIR.ORG

Latent Regression: the Likelihood

• A second item the student got right and 

so is more likely for higher scores

• The model allows “guessing” and so it 

does not go to zero at low scores

15
| AIR.ORG

Latent Regression: the Likelihood

• The student took 13 dichotomous items, 

this is all their likelihoods

16
| AIR.ORG

Latent Regression: the Likelihood

• Combining them we get a skewed 

normal likelihood surface

• If we wanted to assign the respondent a 

score we could take the peak or the 

expectation of this likelihood

• We do not want to assign the 

respondent a score

17
| AIR.ORG

Latent Regression: the Mean Model

• The green curve is the population model 

or mean model

yy�= XXXX + εε

εε~NN(0, σσ�)

• Fit parameters ( ) can move it to the 

left/right, or (σσ�) make the distribution 

wider or narrower

18
| AIR.ORG

Latent Regression: Final Model

• The student’s final likelihood is the 

product of the mean model and the 

likelihood

• The mean model is fit by maximizing the 

“likelihood” of this product

• This is analogous to a regression where 

the outcome is not precisely measured, 

but we still want accurate coefficients

19
| AIR.ORG

Latent Regression: 2-Dimensions

• For multiple dimensions we must 

estimate the joint density across the 

dimensions.

• Here a student’s likelihood is plotted for 

two constructs to estimate the 

correlation between the two

20

Conditioning model

Likelihood

Posterior distribution
| AIR.ORG

Latent Regression: n-Dimensions

• Because a multivariate normal 

distribution, when marginalized to two 

variables, is bivariate normal the same 

procedure used for 2-dimensions 

generalizes completely to any number of 

dimensions—only bivariate correlations 

need to be calculated

21

Conditioning model

Likelihood

Posterior distribution
| AIR.ORG

EdSurvey Dire Package

• EdSurvey is an R package developed by NCES and AIR to help users download, 

understand available covariates for, run common analyses on, and extract data for 

additional analyses

• Dire adds the ability to use the type of latent regression described above to EdSurvey, 

but it can also work with other data

22
| AIR.ORG

Plausible Values

• EdSurvey, with Dire, can use a latent regression to draw plausible values

• This is multiple imputation for the latent variable or test score

• This is typically used by data owners before distribution

• This prevents users from having to use latent regression

• Also prevents data owners from having to share all the conditioning variables 

• In addition, this allows repeated use of a conditioning model

• A conditioning model is biased if a variable of interest is not included, but not if 

unused variables are not included

23
| AIR.ORG

Selected comparison results for multiple regressions

24

NOTE: Race categories exclude persons of Hispanic origin. 

Asian includes students who are identified as Pacific Islander. 

Other includes students who are identified as either 

American Indian or Alaska Native or More than one race. 

NSLP = National School Lunch Program. Bold indicated that 

the magnitude of coefficients in the model using Dire PVs 

was larger than the model using the original NAEP PVs.

SOURCE: U.S. Department of Education, Institute of 

Education Sciences, National Center for Education Statistics, 

National Assessment of Educational Progress (NAEP), 2017 

Grade 8 Mathematics Assessment. U.S. Department of 

Education, National Center for Education Statistics, 2017–18 

School Survey on Crime and Safety (SSOCS), 2018.
| AIR.ORG

Selected comparison results for multiple regressions (cont’d)

25

NOTE: Bold indicated that the magnitude of coefficients in 

the model using Dire PVs was larger than the model using 

the original NAEP PVs.

SOURCE: U.S. Department of Education, Institute of 

Education Sciences, National Center for Education Statistics, 

National Assessment of Educational Progress (NAEP), 2017 

Grade 8 Mathematics Assessment. U.S. Department of 

Education, National Center for Education Statistics, 2017–18 

School Survey on Crime and Safety (SSOCS), 2018.
| AIR.ORG

Final Thoughts

• These methods are general to other latent constructs, such as a compound SES 

measure, or a personality measure

• Learn more about EdSurvey on the AIR website

• https://www.air.org/project/nces-data-r-project-edsurvey

26
AMERICAN INSTITUTES FOR RESEARCH® | AIR.ORG

Notice of Trademark: “American Institutes for Research” and “AIR” are registered trademarks. All other brand, product, or company names are trademarks or registered trademarks of their respective owners.

Copyright © 2021 American Institutes for Research®. All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, 

website display, or other electronic or mechanical methods, without the prior written permission of the American Institutes for Research. For permission requests, please use the Contact Us form on AIR.ORG.

Thank you

Saki Ikoma

Senior Researcher

sikoma@air.org

Paul Bailey

Senior Economist

pbailey@air.org 
| AIR.ORG

References

Burdick-Will, J. (2016). Neighborhood Violent Crime and Academic Growth in Chicago: Lasting Effects of Early Exposure. Social Forces, 95(1), 133–158. 

Carlson, D., & Cowen, J. M. (2015). Student Neighborhoods, Schools, and Test Score Growth. Sociology of Education, 88(1), 38–55.

Gamoran, A. (2016). Toward the Next Generation of U.S. National Longitudinal Surveys: Ideas From Researchers for the National Center for Education Statistics. AERA Open 2(2), 1–8.

Hagan, J., & Foster, H. (2012). Children of the American Prison Generation: Student and School Spillover Effects of Incarcerating Mothers: Children of the American Prison Generation. Law & Society Review, 

46(1), 37–69. 

Johnson, E.G. (1998). Linking the National Assessment of Educational Progress (NAEP) and the Third International Mathematics and Science Study (TIMSS): A Technical Report, NCES 98-499. U.S. Department of 

Education, National Center for Education Statistics. Washington, DC: Government Printing Office, September 1998. 

Kaufman, P. (1996). Linking Student Data to SASS: Why, When, How. Working Paper No. 96-23, U.S. Department of Education, National Center for Education Statistics. Washington, D.C.

Lacoe, J., & Steinberg, M. P. (2019). Do Suspensions Affect Student Outcomes? Educational Evaluation and Policy Analysis, 41(1), 34–62. 

McCoy, D. C., Raver, C. C., & Sharkey, P. (2015). Children’s Cognitive Performance and Selective Attention Following Recent Community Violence. Journal of Health and Social Behavior, 56(1), 19–36. 

Mislevy, R. J., Beaton, A. E., Kaplan, B., & Sheehan, K. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29(2), 133-161.

Peguero, A. A., & Bracy, N. L. (2015). School Order, Justice, and Education: Climate, Discipline Practices, and Dropping Out. Journal of Research on Adolescence, 25(3), 412–426. 

Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York: Wiley.

Ruiz, L. D., McMahon, S. D., & Jason, L. A.. (2018). The Role of Neighborhood Context and School Climate in School-Level Academic Achievement. American Journal of Community Psychology, 61(3–4), 296–309. 

Sable, J., & Miller, S., R. (2020, November 24). Building bridges: Increasing the power of the Civil Rights Data Collection (CRDC) through data linking with an ID crosswalk. 

Sharkey, P. (2010). The acute effect of local homicides on children’s cognitive performance. Proceedings of the National Academy of Sciences, 107(26), 11733–11738. 

Soldner, M. (2017). Leveraging Administrative Data to Strengthen NCES Postsecondary Sample Surveys. (NPEC 2017). U.S. Department of Education. Washington, DC: National Postsecondary Education 

Cooperative.

von Davier, M., Gonzalez, E. J., & Mislevy, R. J. (2009). What are Plausible Values and Why are They Useful? In Matthias von Davier & Dirk Hastedt (eds.). IERI Monograph Series: Issues and Methodologies in 

Large-Scale Assessments: Volume 2. 2009, 9-36.

Wu, M. (2005). The role of plausible values in large-scale surveys. Studies in Educational Evaluation, 31(2-3), 114-128.

28
| AIR.ORG

Descriptive statistics of variables of interest

29

NOTE: n ≈ 8,470. Estimates are at student level. Race categories exclude 

persons of Hispanic origin. Asian includes students who are identified as 

Pacific Islander. Other includes students who are identified as either 

American Indian or Alaska Native or More than one race. SOURCE: U.S. 

Department of Education, Institute of Education Sciences, National Center 

for Education Statistics, National Assessment of Educational Progress 

(NAEP), 2017 Grade 8 Mathematics Assessment. U.S. Department of 

Education, National Center for Education Statistics, 2017–18 School Survey 

on Crime and Safety (SSOCS), 2018.
| AIR.ORG

Descriptive statistics of variables of interest (cont’d)

30

NOTE: n ≈ 8,470. Estimates are at student level. SOURCE: U.S. Department 

of Education, Institute of Education Sciences, National Center for Education 

Statistics, National Assessment of Educational Progress (NAEP), 2017 Grade 

8 Mathematics Assessment. U.S. Department of Education, National Center 

for Education Statistics, 2017–18 School Survey on Crime and Safety 

(SSOCS), 2018.
| AIR.ORG

SSOCS items for school behavioral climate

31

SOURCE: U.S. Department of Education, National Center for Education Statistics, 

2017–18 School Survey on Crime and Safety (SSOCS), 2018.
| AIR.ORG

Results of school behavioral climate CFA

32

NOTE: The survey asked, "To the best of your knowledge, 

how often do the following types of problems occur at your 

school?". Five response categories ranging from never 

happens, happens on occasion, happens at least once a 

month, happens at least once a week to happens daily. The 

estimates of Cronbach's alpha are 0.76. 

SOURCE: U.S. Department of Education, National Center for 

Education Statistics, 2017–18 School Survey on Crime and 

Safety (SSOCS), 2018.
| AIR.ORG

MLM Results

33

NOTE: Race categories exclude persons of Hispanic origin. 

Asian includes students who are identified as Pacific Islander. 

Other includes students who are identified as either 

American Indian or Alaska Native or More than one race. 

NSLP = National School Lunch Program. 

SOURCE: U.S. Department of Education, Institute of 

Education Sciences, National Center for Education Statistics, 

National Assessment of Educational Progress (NAEP), 2017 

Grade 8 Mathematics Assessment. U.S. Department of 

Education, National Center for Education Statistics, 2017–18 

School Survey on Crime and Safety (SSOCS), 2018.
| AIR.ORG

MLM Results (cont’d)

34

SOURCE: U.S. Department of Education, Institute of 

Education Sciences, National Center for Education Statistics, 

National Assessment of Educational Progress (NAEP), 2017 

Grade 8 Mathematics Assessment. U.S. Department of 

Education, National Center for Education Statistics, 2017–18 

School Survey on Crime and Safety (SSOCS), 2018.