1

Using Bayesian Regression Models in Small Sample Size Contexts to Support System-Level Education Decision-Making

Bradley Rentz

REL Pacific 

Christina Tydeman

REL Pacific

2

Introductions and Overview

REL Pacific

3

1

2

3

REL Pacific is one of 10 Regional Educational Laboratories funded by the Institute of Education Sciences at the U.S. Department of Education. The RELs work in partnership with school districts, state departments of education, and other educational stakeholders to use data and research to improve academic outcomes for students.  The RELs provide technical support, conduct research, and and offer learning opportunities that inform changes to policy and practice in an effort to improve educational outcomes for students. REL Pacific serves the State of Hawaiʻi, the territories of Guam, CNMI, and American Samoa, as well as three freely associated states: the Republic of Palau, the Federated States of Micronesia, and the Republic of the Marshall Islands. 

Agenda

Introductions and overview.  

Bayesian regression models can help mitigate the effects of small sample sizes.

Using Bayesian regression models to support education decision-making.  

4

Objectives

Illustrate how Bayesian regression models can provide many advantages for researchers working with small sample sizes.

Share how REL Pacific has used Bayesian regression models to inform education decision making around college readiness on Pohnpei in the Federated States of Micronesia.

5

6

Bayesian regression models can help mitigate the effects of small sample sizes.

Small Sample Sizes in Quantitative Education Research

Most system-level quantitative education studies conducted in the United States have access to large datasets with tens of thousands or more students, which enables them to conduct robust analyses.

Occasionally education researchers have access to much smaller datasets; for example, when examining pilots of a new intervention in a few classrooms or schools or when working with small rural school districts. 

7

(Digital Atlas of Micronesia, 2021)

Small sample sizes are the norm in the Pacific Region.

Even when conducting system-level education research, sample sizes are generally small.

Many jurisdictions in the Pacific Region have relatively small populations.

Most of the education systems in the region have about 2,000–15,000 students total from kindergarten through post-secondary education. 

8

(Digital Atlas of Micronesia, 2021)

The exception to small sample size is the State of Hawaiʻi.

Data missingness and irregularites also contribute to small sample sizes.

Depending on the jurisdiction, education data may be incomplete, missing cohorts, lack unique identifiers, or have other irregulaties that may reduce sample sizes further.

Some of these data issues may be mitigated through standard data imputation procedures, but others cannot.

9

Small sample sizes can complicate education research.

Researchers may want to run (hierarchical) regression models that include predictor variables based on previous research and variables of interest identified by stakeholders.

Small sample sizes can lead to models not being able to computationally converge on a solution.

With small sample sizes, the model’s results can be noisy and might not be sensitive enough to measure differences in the data accurately.

Conversely, there might not be enough variation in the data if the outcome is relatively rare.

10

Small sample sizes can affect the types of statistical models one can use. Some of the variables included may be motivated by prior research that had access to much larger sample sizes.

Prior Practices for Small Sample Sizes

When faced with small sample sizes, researchers typically followed two approaches: collect more data and/or simplify the statistical analyses.

Researchers in general have simplified statistical models by

changing random effects to fixed effects,

removing predictor variables, or 

using bivariate correlations and descriptive statistics instead of regression models.

11

If the regression models were not able to computationally model the data because of the small sample sizes, researchers typically reduced model complexity by removing random effects (if using hierarchical linear modeling) and/or removing predictor variables until the models converged or had lower standard errors for regression coefficients. In some cases, researchers stopped using regression all together and instead used bivariate correlations (or similar bivariate statistical tests like t-tests) and descriptive statistics.

Consequences of Prior Practices for Small Sample Sizes

It may not be possible to collect more data, since researchers may have access to all existing data.

Simplifying statistical models purely for computational reasons may affect the model’s validity and usefulness for making policy decisions.

12

When conducting system-level research, it is not always possible to collect more data, since they may have all the data that exists for those students. Researchers could include additional cohorts to get more data, but those data might not exist or they may have been structural differences the further back one goes. The study could also be delayed years to wait for new cohorts, but that is often impractical since the research need is generally urgent and tied to a problem of practice. 

Changing the statistical models one uses simply because they would not converge or for other computational reasons, may mean that those models do not answer the research questions, may not accurately model the phenomenon being studied, and ultimately may not be useful for stakeholders in making policy decisions.

Bayesian regression models can help mitigate the effects of small sample sizes.

Bayesian regression models are much less likely to have computational convergence errors even with small sample sizes, since they are calculated probabilistically using Markov chain Monte Carlo sampling methods and use prior information to fill in the gaps when there is limited data.

13

(Kruschke & Liddell, 2018)

Using Bayesian regression models instead of simplifying frequentist regression models can be a useful alternative when using small sample sizes. One reason for this is that Bayesian regression models are much less likely to have computational convergence errors even with small sample sizes, since they are calculated probabilistically using Markov chain Monte Carlo (MCMC) sampling methods and use prior information to fill in the gaps when there is limited data. Unlike traditional frequentist models, Bayesian models require the researcher to specify some level of prior knowledge about the relationship of the predictor variables with the outcomes, usually in the form of a statistical distribution. This information is called the “priors.” The Bayesian models combine the data, the priors, and the specified model structure to create a posterior distribution for each of the model’s parameters. These distributions can then be described using descriptive statistics. Usually it is not possible to directly calculate the posterior distribution, so MCMC sampling is used to probabilistically create the posterior distributions by sampling from the data thousands of times. With large datasets, these models can be computationally intensive, but for small datasets, the MCMC sampling can help generate enough data to calculate the model’s parameters, where there simply were not enough data to do the same with frequentist models.

Bayesian regression models can also provide additional benefits to all datasets.

Bayesian models can 

incorporate prior information about the relationship between predictor variables and the outcome variable.

quantify the level of uncertainty after taking into account both the data and our prior knowledge about the data.

14

(Kruschke & Liddell, 2018)

Bayesian regression models may also more accurately produce measures of uncertainty than frequentist models, since Bayesian models can incorporate prior information about the relationship between predictor variables and the outcome variable.

Even if the small dataset is noisy and has lots of variation, Bayesian models can at the very least quantify our level of uncertainty after taking into account both the data and our prior knowledge about the data.

15

Using Bayesian Regression Models to Support Education Decision-Making

Using High School Data to Explore Early College Success on Pohnpei, Federated States of Micronesia

REL Pacific recently used Bayesian generalized hierarchical modeling to examine associations of high school student variables with five community college outcomes for public high school graduates on Pohnpei.

In the study, there were two samples:

Sample 1: 1,146 students who took the College of Micronesia-FSM Entrance Test in grade 12 (88 percent of Pohnpei public high school graduates).

Sample 2: 327 students from sample 1 who were admitted to the college and enrolled in a degree program in the summer or fall term after their high school graduation.

The study included three cohorts and students from each of the island’s three public high schools.

16

(Rentz et al., 2021)

Study Methods

The study team identified predictor variables that had posterior distributions different from the model’s intercept by using a region of practical equivalence (ROPE).

A small region close to zero was defined as the ROPE and determined to be substantively equivalent to zero. 

The percent of overlap of the 95 percent highest density interval with the ROPE for each predictor variable’s posterior distribution was calculated.

Predictor variables that had 0 percent overlap with the ROPE were identified as being statistically different from the intercept and were discussed in the report’s main findings.

For each identified statistically different predictor variable, predicted probabilities (conditional effects) of achieving the outcome were calculated.

17

(Kruschke & Liddell, 2018)

The results of the Bayesian models may also be more conservative than frequentist models especially when using ROPEs. This may help reduce false positives and give a more accurate view of the uncertainty. The 95 percent highest density interval is similar to a frequentist 95% confidence interval.

Example Results

18

(Rentz et al., 2021, p. 8)

High school GPA was found to be a predictor of placement into college credit-bearing math, reading, and writing courses, but there were different probabilities of placement for each course. Each of the three outcomes were modeled separately using Bayesian hierarchical logistic regression models.  

Using Bayesian Regression Models: Results

Using Bayesian hierarchical regression models instead of frequentists models allowed the study team to:

Include predictor variables and random effect structure motivated by past research, the structure of the local school system, interests of stakeholders, and available data.

Identify predictor variables that are statistically associated with community college outcomes after taking into account the prior uncertainty about data and the limited data due to small samples.

Stakeholders at the Pohnpei Department of Education and the College of Micronesia-FSM are currently discussing ways to use the study results inform policy and practice changes to improve student outcomes. 

19

If the study team were to have used frequentist models, there may have been model convergence issues and the researcher may not have been able to as accurately model the data. 

Some of the items under discussion by stakeholders based on the results are (1) to use a multiple measures admission and placement system at COM-FSM that uses GPA in addition to the entrance test and (2) to provide extra supports in high school and the college to support students’ English reading and writing skills to increase the number of the students who place into those credit-bearing courses at the college. 

Thank you for your time! Contact Information:

Bradley Rentz brentz@mcrel.orgChristina Tydeman ctydeman@mcrel.org

20

CT

We appreciate the opportunity to be here, and we thank you for your time with us.

The reports for the Using High School Data to Explore Early College Success on Pohnpei, Federated States of Micronesia can be found on the IES website or via this QR code:

21

QR code goes to https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=5667

For those interested, the reports...

References

Digital Atlas of Micronesia. (2021). Digital Atlas of Micronesia. https://islandatlas.org/

Kruschke, J. K., & Liddell, T. M. (2018). The Bayesian new statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25(1), 178–206. https://doi.org/10.3758/s13423-016-1221-4

Rentz, B., Holquist, S. E., Nardi, C. L., Simion, K., Martin, P., & Arens, S. A. (2021). Using High School Data to Explore Early College Success on Pohnpei, Federated States of Micronesia (REL 2021–092). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Pacific. https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=5667.

22

This presentation was prepared under Contract # ED-IES-17-C-0010 by Regional Educational Laboratory Pacific, administered by McREL International. The content does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.

23

24

Supplemental Slides

Example Table Displaying Bayesian Regression Results

25

(Rentz et al., 2021, p. C-3)

This is a table that shows how Bayesian regression results could be reported in tabular form. The main features to report are the coefficient (average value of the posterior distribution), 95 percent highest density interval (HDI) and the percent inside the ROPE. In the table notes, we identified that the ROPE was defined as -0.18 to 0.18.

# image1.emf

# image2.png

# image5.jpeg

# image6.jpeg

# image7.png

# image3.png

# image8.png

# image9.png