## 5 Design Aspects
### Sample Size Considerations
Since in practice there is uncertainty around the engagement (we assume it can be quantified using \(N(\lambda,\xi^{2})\) distribution), we can repeat the entire procedure described in Step 1 (say, 1000 times) to obtain a distribution of the requisite sample size. An experimenter may then decide to use some percentile of this distribution (say, 80 th percentile), to ensure that their experiment has the desired level of power \(=1-\beta\), with reasonably high confidence, for the chosen values of trial parameters. A histogram or a box-plot of the simulated distribution of the sample size can provide valuable insights into the required sample size and the associated uncertainty. Figure 3 displays box-plot distributions of the smallest sample size to achieve 80% power of the test using 2-sided \(\alpha=5\%\). Individual responses were generated from model (1) with parameters \(\mu_{E}=-1\), \(\mu_{C}=0\), \(\gamma=-0.5\), and \(\sigma=1\). Four different distributions of engagement were considered:  
(i) \(X\sim Beta(0.5,0.5)\), which has mean \(=0.5\) and SD \(\sim 0.354\);  
(ii) \(X\sim Uniform(0,1)\), which has mean \(=0.5\) and SD \(\sim 0.289\);3. \(X\sim Normal\) with mean \(=0.6\) and SD \(=0.3\); and
4. \(X\sim Normal\) with mean \(=0.6\) and SD \(=0.2\). For each of the four engagement patterns, simulations were performed as described above. In Step 1, we ran 10,000 simulations to obtain empirical power for the total sample size \(n=2m\), where \(m=25,26,\ldots,200\), and used the smallest \(n\) for which empirical power \(\geq 80\%\) as an estimate of the required sample size. In Step 2, 1000 replications of Step 1 were performed. From Figure 3, several important observations can be made. First, the requisite sample size is smaller if the underlying distribution of engagement is more variable. This makes good sense, because from Equation (4), \(\text{var}(\widehat{\mu}_{E}-\widehat{\mu}_{C})=\sigma^{2}\bigg{(}\frac{4}{n} +\frac{\overline{\
u}_{E}^{2}}{5_{xx}}\bigg{)}\), and larger sample variance of engagement (\(S_{xx}\) term in the denominator) implies lower value of \(\text{var}(\widehat{\mu}_{E}-\widehat{\mu}_{C})\), which leads to a larger absolute value of the \(t\)-statistic in Equation (6), and therefore higher power of the test. Second, there is uncertainty in the requisite sample size, and this uncertainty is smaller if the engagement is more variable. Table 1 shows the summary statistics for the sample size distributions from Figure 3. Suppose it is plausible to assume that individual engagement values are uniformly distributed over the interval (0, 1), and an investigator wants to be 80% (or 90%) confident that the experiment has 80% statistical power for the chosen trial parameters (\(\mu_{E}=0\), \(\mu_{C}=-1\), \(\gamma=-0.5\), and \(\sigma=1\)). Then, the required total sample size is \(n=92\) (or 98). On the other hand, if it is plausible to assume that engagement follows a normal distribution with mean 0.6 and standard deviation 0.2, and we want to be 80% (or 90%) confident that the experiment has 80% statistical power, the required sample size is \(n=194\) (or 206). Figure 3: Distributions of the sample size to achieve 80% power of ANCOVA \(t\)-test of \(H_{0}:\Delta_{0}=0\) using 2-sided \(\alpha=5\%\). Responses were simulated from model (1) with \(\mu_{E}=-1\), \(\mu_{C}=0\), \(\gamma=-0.5\), and \(\sigma=1\). Four different distributions of engagement were considered (top to bottom): _Beta_(0.5,0.5); _Uniform_(0,1); _Normal_ with mean \(=0.6\) and standard deviation (SD) \(=0.3\); and _Normal_ with mean \(=0.6\) and SD \(=0.2\). Note that if we consider another research hypothesis, e.g., \(H_{0}:\Delta_{x}=0\) for some \(x>0\), then the required sample size will be different. For instance, if we wish to test \(H_{0}:\Delta_{x_{E}}=0\), then we can follow a conventional approach to sample size determination using the two-sample \(t\)-test, which requires no upfront knowledge on engagement.