A General Framework for Power Analysis to Detect the Moderator Effects in Two- and Three-Level Cluster Randomized Trials.

Body Background / Context: Like studies that focus on detecting a main effect, a critical consideration in designing studies to detect moderation effects is the statistical power with which the moderation effect can be detected if they exist. The potential moderators of the intervention in cluster randomized trials (CRTs) include pretest, ethnicity, school climate, or the fidelity of implementation, which could be at different levels and have different distributions (e.g., binary, continuous). For moderator relationships in experimental studies, Bloom (2005), Spybrook (2014), and Spybrook, Kelcey, and Dong (2015) have presented procedures for conducting power analysis for binary moderators in twoto four-level cluster randomized trials (CRTs), but have not extended those procedures to include continuous moderator variables. Furthermore, no computational tools to facilitate use of these techniques by researchers have been developed. Most recently, Mathieu, Aguinis, Culpepper, & Chen (2012) conducted a comprehensive Monte Carlo simulation to estimate the statistical power to detect cross-level interaction effects, and Dong (2014) presented the formulas to calculate minimum detectable effect size (MDES) for continuous moderator analysis in two-level CRTs with a level-2 continuous moderator. However, Mathieu et al (2012) only studied two-level analyses without including covariates, and did not provide closed form formulas to estimate the statistical power, MDES, or minimum required sample size to detect meaningful effects; Dong (2014) only focused on a level-2 continuous moderator in two-level CRTs, and didn’t study cross-level moderation, include covariates, or examine three-level CRTs. In sum, there is no a systematic study investigating the power analysis to detect moderation effects in twoand three-level CRTs that includes both binary and continuous moderators, same and cross-level moderation, with covariates, and computational tools. Purpose / Objective / Research Question / Focus of Study: The purpose of this study is to propose a general framework for power analyses to detect the moderator effects in twoand three-level CRTs. Specifically, we aim to: (1) develop the statistical formulations for calculating statistical power, minimum detectable effect size (MDES) and its confidence interval to detect the moderation effects in twoand three-level CRTs, which include same and cross-level moderation, binary and continuous moderators, and covariates, and (2) operatize these formulas in the enhanced version of PowerUp! (Dong & Maynard, 2013) to create spreadsheets for calculating power, MDES, etc. Significance / Novelty of study: Educational researchers have interests in the effects of both binary and continuous moderators in CRTs. Statistical power analysis is appropriate in the planning stages to help researchers design studies with sufficient power to detect such relationships when they are large enough to have practical or theoretical significance. This study will provide a general framework and computational tool for power analyses to detect the binary and continuous moderator effect in twoand three-level CRTs. Study Design: This study covers the same and cross-level moderation in twoand three-level CRTs. The detailed study designs and analysis models are presented at Table 1 (see appendix). For the cross-level moderation, i.e., the moderator at the level lower than the treatment level, there are two options: the fixed slope and random slope of the moderator variable. The fixed slope of the SREE Spring 2016 Conference Abstract Template 1 moderator variable assumes that the effect of the moderator varies by the treatment status, but does not vary across the higher-level clusters, while the random slope of the moderator effects assume that the effect of the moderator varies by the treatment status, and varies randomly across the higher-level clusters. The power and MDES formulas are derived for all the designs and models covered in Table 1 and are applied in PowerUp! Statistical, Measurement, or Econometric Model: Due to the page limitation, we present the results of two-level CRTs with a treatment variable at Level 2 and a moderator at Level 1 below. The HLM, including one treatment variable, j T , and one level-1 moderator, ij X , with a random slope is: Level 1: ij ij j j ij r X Y + + = 1 0 β β , ) , 0 ( ~ 2 |X ij N r σ (1) Level 2: j j j j j j u T u T 1 11 10 1 0 01 00 0 + + = + + = γ γ β γ γ β , 2 | 11 | 10 | 01 2 | 00