**Sample Size Calculations for the Modular Grant Application Process
Outcome Evaluation Study**

One purpose of this paper is to describe and illustrate the methods used
to determine the sample sizes for the Modular Grant Application Process
(MGAP) Outcome Evaluation Study. The sampling strategy for this study
involves selecting a simple random sample without replacement from four
of the six study populations. (A census was used for the remaining two
populations.) This paper also serves as a primer on the factors affect
sample size, how sample size is calculated, and the use of the finite
population correction (FPC) factor when drawing a simple random sample.

The paper is divided into four sections and uses one of the six study
populations[^1] as an example throughout the paper. The first section
outlines factors affecting the sample size when using a simple random
sampling plan without replacement. The second part illustrates the
sample size formula. The third section explains the purpose of the
Finite Population Correction (FPC) Factor and depicts its use. The last
section describes how we obtained the number of respondents to survey
from each of the populations. More specifically, we explain how our
expected response rates and anticipated returned emails (called "bounce
backs") are taken into consideration when calculating the final number
of respondents to survey. Appendix A shows the sample sizes for all
study populations and Appendix B provides web addresses of sample size
calculators easily accessible via the Internet. The population used in
the examples throughout the paper consists of all principal
investigators (PIs) who have applied for and received at least one
modular grant (N=16,450). This group is referred to as PIs~1~.

**Factors Affecting Sample Size.**

Three factors are used in the sample size calculation and thus,
determine the sample size for simple random samples. These factors
are: 1) the *margin of error*, 2) the *confidence level*, and 3) the
*proportion (or percentage) of the sample that will chose a given
answer* to a survey question. Each one of these will be discussed below.

The *margin of error* (also referred to as the confidence interval)
measures the precision with which an estimate from a single sample
approximates the population value. For example, in a national voting
poll the margin of error might be + or -- 3%. This means that if 60% of
the people [in a sample]{.underline} favor Mr. Smith, you could
confident[^2] that, if you surveyed the entire population, between 57%
(60-3) and 63% (60+3) [of the population]{.underline} would favor Mr.
Smith. The margin of error in social science research generally ranges
from 3% to 7% and is closely related to sample size. A margin of error
will get narrower as the sample size increases. The margin of error
selected depends on the precision needed to make population estimates
from a sample. If it's acceptable to have an interval of + or -7% around
a given estimate, then the sample size needed will be smaller than if an
interval of + or -3% is the largest acceptable interval. For all samples
used in the MGAP Outcome Evaluation, the margin of error is + or - 5%.

The *confidence level* is the estimated probability that a population
estimate lies within a given margin of error. Using the example above, a
confidence level of 95% tells you that you can be 95% confident that
between 57% and 63% of the population favors Mr. Smith. Common
confidence levels in social science research include 90%, 95%, and 99%.
Confidence levels are also closely related to sample size. As the
confidence level increases, so too does the sample size. A researcher
that chooses a confidence level of 90% will need a smaller sample than a
researcher who is required to be 99% confident that the population
estimate lies within the margin of error. Looking at it another way,
with a confidence level of 95%, there is a 5% chance that an estimate
derived from a sample will fall outside the confidence interval of 57%
to 63%. Researchers will chose a higher confidence level in order to
reduce the chance of making a wrong conclusion about the population from
the sample estimate. For all samples used in the MGAP Outcome
Evaluation, the confidence level is 95%.

Most of the time, the *proportion (or percentage) of a sample that will
choose a given answer* to a survey question is unknown, but it's
necessary to estimate this number since it is required for calculating
the sample size. Most researchers will use a proportion (or percentage)
that is considered the most conservative estimate -- that is, that 50%
of the sample will provide a given response to a survey question. This
is considered the most conservative estimate because it is associated
with the largest sample size. Smaller sample sizes are needed if the
proportion of a sample that will choose a given answer to a question is
estimated at 60% (or 40%) while an even smaller sample size is needed if
the estimated proportion of responses is either 70% (or 30%), 80% (or
20%), or 90% (or 10%). Thus, when determining the sample size needed for
a given level of accuracy (i.e., given confidence level and margin of
error), the most conservative estimate of 50% should be used because it
is associated with the largest sample size. For the MGAP evaluation
study, the proportion (or percentage) of respondents that will choose a
given answer to a survey question is unknown. Therefore, we have
estimated this percentage as 50%.

**Sample Size Formula.**

The formula for calculating the sample size for a simple random sample
without replacement is as follows:

> $n = \left( \frac{z}{m} \right)^{2}p(1 - p)$

where,

> *z* is the z value (e.g., 1.645 for 90% confidence level, 1.96 for 95%
> confidence level, and 2.575 for 99% confidence level);
>
> *m* is the margin of error (e.g., .07 = + or -- 7%, .05 = + or -- 5%,
> and .03 = + or -- 3%); and
>
> *p* is the estimated value for the proportion of a sample that will
> respond a given way to a survey question (e.g., .50 for 50%).

Using our factors for the principal investigator population, PIs~1~, and
solving for the sample size equation, we find:

$n = \left( \frac{1\text{.}\text{96}}{\text{.}\text{05}} \right)^{2}\text{.}5\left( 1 - \text{.}5 \right)$
**=** (39.2)^2^(.25) **=** 1536.64(.25) = 384

Thus, without using the finite population correction factor (explained
below), the sample for PIs~1~ = 384.

**Finite Population Correction (FPC) Factor.**

The Finite Population Correction (FPC) factor is routinely used in
calculating sample sizes for simple random samples. In fact, many sample
size formulas for simple random samples include the FPC as part of the
formula. It has very little effect on the sample size when the sample is
small relative to the population but it is important to apply the FPC
when the sample is large (10% or more) relative to the population.

The sample size equation solving for $n^{'}$(new sample size) when
taking the FPC into account is:

> $$n^{'} = \frac{n}{1 + \frac{n}{N}}$$

where,

*n* is the sample size based on the calculations above, and

*N* is population size.

Calculating the new sample size for PIs~1~ using the formula above, we
find:

$n^{'} = \frac{\text{384}}{1 + \frac{\text{384}}{\text{16450}}}$ =
$\frac{\text{384}}{1\text{.}\text{023}}$ = 375.37

Thus, the new sample size using the FPC = 375. As can be observed,
applying the FPC affects the sample size very little in this case since
the original sample size of 384 is small relative to the population
(N=16,450). The new sample size of 375 is the number of respondents
needed to make sample estimates in which we are 95% confident that the
value obtained is within + or -- 5% of the true population value. For
example, if 75% of PIs~1~ in the sample report that they prefer the
modular grant limit be higher than \$250,000, you would be 95% confident
that between 70% and 80% of all PIs~1~ in the population prefer the
modular grant limit be higher than \$250,000. Put another way, using a
sample of 375 PIs~1~, there is only a 5% chance that a sample estimate
will fall outside the confidence interval of 70% to 80%.

**Taking Into Account Response Rate and Email "Bounce Backs".**

Since we expect a response rate of 60%, we accounted for this by
dividing the sample size (n=375) by .60 to get a new sample size of 625.
Thus, 60% of 625 = 375. Our experience with email addresses obtained
from a list has also taught us to expect approximately 15% of all email
addresses to bounce back. Therefore, we have accounted for this by
decreasing our sample expectations by another 15% \[375/(.60 - .15) =
375/.45 = 833\].

To sum, the final number of PIs~1~ we will email a survey to is 833 with
the expectation of receiving 375 surveys back from this sample. To help
achieve our desired number of completed surveys (375), we sent a
pre-notification letter, a survey notification email, two follow-up
emails, and conducted telephone prompting.

**References**

Cochran, W. (1977). Sampling Techniques. New York: John Wiley & Sons.

Kalton, G. (1983). Introduction to Survey Sampling. Thousand Oaks, CA:
Sage Publications.

Kish, L. (1965, 1995). Survey Sampling. New York: John Wiley & Sons.

Moore, D. and McCabe, G. (1999). Introduction to the Practice of
Statistics (3rd Edition). New York:

W.H. Freeman and Company**. \[Note: This is the easiest of the four
references to read.\]**

**Appendix A**

**Sample Sizes for All Populations**

+--------------------------+---------+----------+----------+-----------+
| Population Name          | Pop     | Sample   | Sample   | Sample    |
|                          | ulation | Size     | Size     | Size      |
|                          | Size    | with     | Ac       | A         |
|                          |         | FPC^\*^  | counting | ccounting |
|                          |         |          | for 60%  | for 15%   |
|                          |         |          | Response | Email     |
|                          |         |          | Rate     | "Bounce   |
|                          |         |          |          |           |
|                          |         |          |          | Bac       |
|                          |         |          |          | ks"^\*\*^ |
+--------------------------+---------+----------+----------+-----------+
| 1.  Principal            | 16,450  | 376      | 627      | 836       |
|     Investigators who    |         |          |          |           |
|     have received at     |         |          |          |           |
|     least one modular    |         |          |          |           |
|     grant (PIs~1~)       |         |          |          |           |
+--------------------------+---------+----------+----------+-----------+
| 2.  Principal            | 14,730  | 375      | 625      | 833       |
|     Investigators who    |         |          |          |           |
|     have never received  |         |          |          |           |
|     a modular grant      |         |          |          |           |
|     (PIs~2~)             |         |          |          |           |
+--------------------------+---------+----------+----------+-----------+
| 3.  Peer Reviewers (PRs) | 2,836   | 339      | 565      | 753       |
+--------------------------+---------+----------+----------+-----------+
| 4.  NIH Scientific       | 372     | 189      | 315      | N/        |
|     Review               |         |          |          | A^\*\*\*^ |
|     Administrators       |         |          |          |           |
|     (SRAs)               |         |          |          |           |
+--------------------------+---------+----------+----------+-----------+
| 5.  NIH Program and      | 1,213   | 292      | 487      | N/A       |
|     Grants Management    |         |          |          |           |
|     Staff (PGMs)         |         |          |          |           |
+--------------------------+---------+----------+----------+-----------+
| 6.  Institutional        | 327     | 177      | 295      | 393^      |
|     Officials (IOs)      |         |          |          | \*\*\*\*^ |
+--------------------------+---------+----------+----------+-----------+

^\*^The number of surveys needed to make population estimates with a
confidence level of 95%

> and a margin of error of + or - 5%.

^\*\*^The number of respondents that were emailed a web link to the
survey.

> ^\*\*\*^We assumed the email lists provided by NIH of their employees
> will be fairly accurate and accounting
>
> for 15% bounce back emails is not necessary; For the SRA population,
> we used a census, meaning we sent all 372 SRAs a web link to the
> survey.
>
> ^\*\*\*\*^When we accounted for the additional 15% for email "bounce
> backs," the sample size exceeded the
>
> population size. Therefore, we used a census of institutional
> officials for the study, meaning we sent all
>
> institutional officials in the population were emailed a web link to
> the survey.

**Appendix B**

**Examples of Sample Size Calculators on the Internet**

<http://www.raosoft.com/samplesize.html>

<http://surveysystem.com/sscalc.htm#ssneeded>

<http://calculators.stat.ucla.edu/>

[^1]: The six populations for this study are: 1) all principal
    investigators who have applied for and received at least one modular
    grant (N=16,450), 2) all principal investigators who have applied
    for but never received a modular grant (N= 14,730), 3) all current
    NIH peer reviewers in addition to those who have completed a
    rotation within the last 2 years (N=2836), 4) all NIH scientific
    review administrators (SRAs) (N= 372), 5) all NIH program and grants
    management staff (N=1213), and 6) institutional officials from
    institutions that applied for 10 or more modular grants, excluding
    foreign (but including Canadian) institutions (N=327).

[^2]: How much confidence you would have depends on the *confidence
    level*, which is discussed in the next paragraph.