## 4 Statistical Properties of Significance Tests: A Simulation Study  
To gain further insights into the properties of statistical tests described in Section 2, we ran Monte Carlo simulations under different experimental scenarios. We considered the following parameter values for the simulations:  
* \(\mu_{C}=0\) and \(\mu_{E}\) is in the range from \(-1\) to \(1\); therefore \(\Delta=\mu_{E}-\mu_{C}=-1,-0.5,0,0.5,1\). * \(\gamma\) (slope) is in the range from \(-1\) to \(1\) (\(\gamma=-1,-0.5,0,0.5,1\)). * \(\sigma^{2}=1\). * \(n=50\) (25 subjects per arm). * All tests are 2-sided, with significance level \(\alpha=0.05\). Engagement in the experimental group was simulated as a random sample of size 25 from a normal distribution with mean \(\lambda=0.6\) and standard deviation \(\xi=0.2\). The sample summary statistics were \(\overline{x}_{E}=0.62\) and \(S_{xx}=1.38\), and these values were fixed throughout simulations. For each experimental scenario defined by a combination of \(\Delta\), \(\gamma\), \(\sigma^{2}\), and \(n\), we consider testing \(H_{0}:\Delta_{\chi}=0\), where \(\Delta_{x}=\Delta+\gamma x\) is the contrast between the combined effect of the novel CBT + dCBT engaged at level \(x\) and the effect of the control treatment using the ANCOVA test (12) for several pre-specified levels of engagement \(x\), namely \(x=0,0.3,0.5,\text{ and }0.8\). Moreover, we consider testing \(H_{0}:\Delta_{\overline{x}_{E}}=0\), i.e., the significance of the difference between the effects of the experimental treatment at the average observed engagement \(\overline{x}_{E}=0.62\) and that of the control treatment. For the latter hypothesis, we consider two tests: the ANCOVA test (12) with \(x=\overline{x}_{E}=0.62\) and the two-sample \(t\)-test (17). For each experimental scenario, the type I error rate and power of statistical tests were estimated based on 10,000 Monte Carlo simulations. For each simulation, outcome data were generated from model (1) with the chosen values of trial parameters. The proportion of simulations for which a given test yielded a statistically significant result was taken as a Monte Carlo estimate of type I error rate (or power). Figure 2 shows the power patterns, which are plotted using different symbols for \(x=0\) (red square, ), \(x=0.3\) (brown circle, ), \(x=0.5\) (dark green triangle, ), \(x=0.8\) (dark green rhombus, ), \(x=0.62\), ANCOVA test (dark blue rhombus, ), and \(x=0.62\), two-sample \(t\)-test (dark blue inverted triangle, ). The \(x\)-axis displays the true values of the contrast \(\Delta_{x}=\Delta+\gamma x\). Note that several combinations of \((\Delta,\gamma,x)\) can yield the same value of \(\Delta_{x}\). For instance, \((\Delta,\gamma,x)=(-1,-1,0)\) and \((\Delta,\gamma,x)=(-0.5,-1,0.5)\) correspond to \(\Delta_{x}=-1\). The \(y\)-axis displays the values of statistical power. For \(\Delta_{x}=0\), the power is equal to 0.05 (type I error rate). For \(\Delta_{x}\
eq 0\), power \(>0.05\). Clearly, the power is monotone increasing in \(|\Delta_{x}|\). One remarkable observation from Figure 2 is that the power is larger for values of \(x\) that are closer to the average observed engagement (which is equal to 0.62 in our example). This observation is in good correspondence with the theory, which posits that the value of the ANCOVA test statistic in Equation (12) is maximized (hence, maximizing the chance  
Figure 2: Statistical power for testing \(H_{0}:\Delta_{x}=0\), where \(\Delta_{x}=\Delta+\gamma x\), for different combinations of \(\Delta=(-1,-0.5,0,0.5,1)\), \(\gamma=(-1,-0.5,0,0.5,1)\), and \(x=(0,0.3,0.5,0.62,0.8)\). It is assumed that \(n=50\) (25 subjects per arm) and outcomes are generated from model (1) with parameters \(\mu_{E}=\Delta\) in the range from \(-1\) to \(1\); \(\mu_{C}=0\); \(\gamma\) in the range from \(-1\) to \(1\); and \(\sigma=1\). of obtaining a statistically significant result, i.e., achieving maximum power) for \(x=\overline{x}_{E}\).