# Presentation: 164727

## Some Basic Issues in Determining Sample Size for Hypothesis Tests

- Diana Urbauer, M.S.

**Some Basic Issues in Determining Sample Size for Hypothesis Tests**

## Context

**null hypothesis significance testing**
- with the** **following structure:
-  a null hypothesis (typically: no difference between treatments or groups)
-  an alternative hypothesis (specifying some kind of difference between treatments or groups)
- Usually we aim to support the alternative hypothesis by showing that the data do not support the null hypothesis.

**Context**

## Some examples of null and alternative hypothesis pairs.

-    null hypothesis: group means are equal
-    alternative hypothesis: group means are not equal
- example 2
-   null hypothesis: group means are equal
-   alternative hypothesis: second group mean is larger than first

**Some examples of null and alternative hypothesis pairs.**

## Some examples of null and alternative hypothesis pairs.

- example 2 (restated)
- example 3
-    null hypothesis: no difference in group means
-    alternative hypothesis: second group mean is larger than first by at least delta

**Some examples of null and alternative hypothesis pairs.**

## Some examples of null and alternative hypothesis pairs.

-   null hypothesis: group means are equal
-   alternative hypothesis: group means differ by at least delta

**Some examples of null and alternative hypothesis pairs.**

## How do we test a hypothesis?

- I.e., full specification of the null and alternative hypotheses require identification of the probability distributions of the data under these hypotheses.
- In a typical instance of example 4
- the null hypothesis would specify that the difference in group sample means follows a normal distribution with mean 0.

**How do we test a hypothesis?**

## How do we test a hypothesis?

- our example?
- In this case, the null hypothesis is a point hypothesis (it specifies one
- single value as the mean difference), whereas the alternative hypothesis is a
- composite hypothesis.
- I.e. this alternative hypothesis is equivalent to infinitely many point hypotheses
- combined by “or”. For each of these point hypotheses, we need to specify a
- probability distribution for the difference in group means. In actual calculations,
- however, the particular point distribution(s) at the “threshold(s)” of the
- alternative hypothesis play a special role.

**How do we test a hypothesis?**

## How do we test a hypothesis?

- We often obtain the null hypothesis rejection region by specifying a confidence level for the test.
- The relationship between rejection region and confidence level is illustrated by this example:
-   at a 95% confidence level, the probability under the null hypothesis of the difference of sample means falling into the rejection region is .05 (i.e. if the null hypothesis were true, there would be a 5% probability of the null hypothesis being rejected).

**How do we test a hypothesis?**

## confidence level

**power**
- so

## Sample Size

- the test for example 4
- involves sampling from the two groups and calculating the difference between the two sample means. The higher the sample size, the smaller the variances in the distributions of the sample mean differences for both the null hypothesis and the alternative hypothesis.
- Fixing the confidence level, we observe two things with increasing sample size,
-  the rejection region gets larger and
-  the power increases

**Sample Size**

## Power equations for example 4

-  if variances are known and unequal
-  if variances are unknown and equal
- where the t-distributions have N1+N2-1 degrees of freedom and t-star is noncentral.

**Power equations for example 4**

## let’s solve for N:

- with 95% confidence and 80% power, the first case gives:

**let’s solve for N**:

## Sample Size and Effect Size

**Sample Size and Effect Size**

## Power and Effect Size

**Power and Effect Size**

## SUMMARY: Components of Sample Size Calculations for Tests of 2 Independent Means

- Statistical Significance (*α*) - the probability a researcher is willing to accept of rejecting the null hypothesis when that hypothesis is true
- Power (1-*β*) - the probability of rejecting the null hypothesis when that hypothesis is false
- Minimum Detectable Difference (Δ) - the minimum detectable difference that the researching is looking for; often considered the minimum clinical difference of interest
- Variance (*σ*2) - the variance of the phenomenon of interest

## E.g.  Effect of Variance Upon Sample Size Holding Everything Else Constant

**Notes:**
Simple 2-sample t-test; 2-tails, alpha = 5%, power = 80%, minimum detectable difference = 100

## E.g.  Effect of Power Upon Sample Size Holding Everything Else Constant

**Notes:**
Simple 2-sample t-test; 2-tails, alpha = 5%, standard deviation = 100, minimum detectable difference = 100

## E.g.  Effect of Minimum Detectable Difference Upon Sample Size Holding Everything Else Constant

**Notes:**
Simple 2-sample t-test; 2-tails, alpha = 5%, standard deviation = 100, power = 80%

## E.g.  Effect of Significance Upon Sample Size Holding Everything Else Constant

## Tests of 2 Independent Means:Var(X) known, Var(X1) = Var(X2) , n1=n2

- N per group is

## Tests of 2 Independent Means: Var(X) known, Var(X1) <> Var(X2), n1=n2

- N per group is

## Test of 2 Independent Means with Unequal Sample Sizes

- Now you need to know either the sample size in one of your two study groups or the ratio of *N*1:*N*2.
- Knowing the ratio can give a more efficient estimate of sample size

## Test of 2 Independent Means with Unequal Sample Sizes

- For variance known, Var(*X*1)=Var(*X*2), normal distribution

## Test of 2 Independent Means with Unequal Sample Sizes

- For variance known, Var(*X*1)<>Var(*X*2), normal distribution

## Tests of 2 Independent Means:Points to Ponder

- What if data do not follow a normal distribution?
- What if Var(*X*1) and Var(*X*2) are unknown?

## Tests of 2 Independent Means:Points to Ponder (cont’d)

- Note that *N* is in both sides of the equation and cannot be solved for except in an iterative fashion.

## Tests of 2 Means: Dependence

- What if means are not independent, e.g. paired t-test?
- Now need an estimate of the correlation between *X*1 and *X*2
- The variance is adjusted by 1-*r*, where *r* is the correlation between *X*1 and *X*2

## Tests of 2 Means: Dependence

- So
- becomes

-  Note that when two samples are not independent, smaller sample sizes will yield just as much power, everything else being held constant

## Power Determination for Analysis of Variance Models

- Need to know
- Significance level (*α*)
- The number of groups, *t*
- The number of observations per group, *r*
- And therefore the degrees of freedom of the F-statistics, *ν*1 and *ν*2
- The experimental error variance (*σ**2*)
- The treatment error variance (*σ**2**t*)

## Power Calculation for a One-Way ANOVA

- where F* is the MST/MSE and *λ* is the estimate of the non-centrality parameter defined as

## Sample Size Calculation for a One-Way ANOVA

- Note that a knowledge of *N* is needed to solve for *N* because it is a component of the degrees of freedom for the *F*-distribution and of the non-centrality parameter.
- Therefore, solving for *N* is an iterative process.

## E.g.  Effect of Treatment Mean Square Upon Sample Size Holding Everything Else Constant

**Notes:**
Simple 2-sample t-test; 2-tails, alpha = 5%, power = 80%, minimum detectable difference = 100

## E.g.  Effect of Error Mean Square Upon Sample Size Holding Everything Else Constant

**Notes:**
Simple 2-sample t-test; 2-tails, alpha = 5%, power = 80%, minimum detectable difference = 100

## E.g.  Effect of Power Upon Sample Size Holding Everything Else Constant

**Notes:**
Simple 2-sample t-test; 2-tails, alpha = 5%, power = 80%, minimum detectable difference = 100

## Power Calculations for Planned Comparisons in a One-Way ANOVA

- Procedure is the same as for overall determination of power but now
- The problems are still the same.

## Detecting a difference between proportions

- Our assumptions are that the data in each group consist of binary (0 or 1) variables which, upon repeated independent trials, follow a binomial distribution (the binomial variable being a count of 1’s or “successes” in N trials). The proportion p of “successes” in N trials is then the mean over those trials of the binomial count variable.

**Detecting a difference between proportions**

## Detecting a difference between proportions

- the difference of the two binomial variables.
- Unfortunately, the difference of two binomial variables is not itself a binomial variable. Its distribution is actually given by a “convolution” of two binomial distributions which can be somewhat computation intensive for large n.
- But, any binomial distribution B(n,p) can be approximated, for large n, by a normal distribution N(np,np(1-p)). Since differences of normal random variables are themselves normally distributed, the difference of two binomials can be (and commonly still is) given a normal approximation as can the difference of two proportions.

**Detecting a difference between proportions**

## Detecting a difference between proportions

- where

**Detecting a difference between proportions**

- followed by these “continuity corrections”

## Detecting a difference between proportions

- before the continuity
- correction.
- The highest power curve occurs approximately where
- p-bar=(1-Delta)/2.
- Note that the formula includes reference to the two actual proportions p1 and p2 and not just their difference (as with the continuous variable examples earlier on).
- Let’s see some of these plotted.

**Detecting a difference between proportions**

## Sample size curves for detecting difference in proportions

**Sample size curves for detecting difference in proportions**

## What’s different about the proportion case?

- This is different from the case of detecting differences in means of continuous variables in which the sample size needed to detect a difference of a given size at a given power was the same regardless of the values of the means for the two groups at the alternative hypothesis threshold.
- In this respect, the proportion case represents the more typical kind of sample size calculation scenario. This dependence on the parameter values at the alternative hypothesis threshold is seen, for example, in calculations for odds ratios and hazard ratios. In such situations, we cannot get away with simply sizing up for an “effect size” alone. An odds ratio example follows.

**What’s different about the proportion case?**

## Illustrations:

- For a null hypothesis of p1=p2 (odds ratio=1) and alternative hypothesis of log(odds ratio) greater or equal to Delta-star, the power equation is

- Note how it is necessary to supply both p1 and p2 explicitly into this formula (the underlying assumption here being that p1 and p2 are the alternative hypothesis values for the proportions whereas under the null they are both hypothesized to be equal to their weighted average).

**Illustrations:**

## Illustrations:

**Log Rank test for hazard ratios**
- Null hypothesis: lambda1=lambda2(HR=1)
- Alternative hypothesis threshold: HR= Delta

- This formula requires specification of not only the ratio Delta but also the alternative hypothesis hazard rates for the two groups (whose ratio is Delta). Moreover, estimates of censoring rates e(lambda) must be included for both hypotheses. R1 and R2 are the relative sizes of the two groups.

## Some Common Structure of These Sample Size Formulae

- Despite the diversity of statistical tests our survey has touched on, most of the sample size formulae derivable from the power equations we’ve seen share a fairly consistent structure which we can summarize by:

- Where CCV is the confidence level critical value, PCV is the power critical value, NHSE is the standard error under the null hypothesis and AHSE is the standard error under the alternative hypothesis threshold.

## Hierarchical Design and Sample Size

- Our survey so far has assumed that in each case, the sampling unit was the individual subject. By contrast, a hierarchical or nested design results if the sampling process involves two or more levels. A two-level hierarchical design would result if, for example, a sample of treatment sites were randomly chosen from a population of such sites, and then individual subjects were sampled randomly within those sites.
- In general, a hierarchical design will make a higher demand on sample size than the corresponding single-level design which ignores all structure above that of the individual subject. It does so essentially by inflating the standard error components of the power and sample size formulae by a multiplicative factor of
- (1+[Site Size –1]*[Intrasite Correlation]).

## Other Approaches to Sample Size Calculation

- Bayesian approaches
- Simulation

## In Conclusion

- In calculating sample size it is essential
- to know what statistical test you are sizing up for
- to know specifically what the null and alternative hypotheses are for the test
- to have values or estimates ready for the inputs to the calculation for the particular test at issue (these always include confidence level, power and detectible difference, but may also include variances, actual parameter values for the alternative hypothesis threshold etc.)