National C

U.S. DEPARTMENT

OF HEALTH AND

HUMAN SERVICES

National Institutes 

of Health

ANALYTIC

METHODS

TO EXAMINE

CHANGES

ACROSS YEARS

USING HINTS

2003 & 2005

DATA

Lou Rizzo, Ph.D.

Richard P. Moser, Ph.D.

William Waldron, B.S.

Zhuoqiao Wang, M.S.

William W. Davis, Ph.D.

ANALYTIC

METHODS

TO EXAMINE

CHANGES

ACROSS YEARS

USING HINTS

2003 & 2005

DATA

Lou Rizzo, Ph.D.1

Richard P. Moser, Ph.D.2

William Waldron, B.S.3

Zhuoqiao Wang, M.S. 3

William W. Davis, Ph.D. 2

1 Westat Inc.;

2 Division of Cancer Cont

Sciences, National Cance

3 Information Managemen
2

deviations between the replicate estimates 

‘full-sample’ estimate, with appropriate a

provides consistent estimators of the varia

example, suppose θˆ is an estimator (a perc

within a subgroup, for example) using the 

sample’ weights. We generate replicate es

in parallel, doing the calculation in the sa

using each set of replicate weights in plac

original full-sample weights. The jackkni

estimator of θˆ is

Final methodology reports are available f

HINTS 2003 and HINTS 2005 and are ac

online at no cost on http://hints.cancer.gov

reports provide details of the sampling an

for the respective surveys. This methodolo

is closely based on a similar methodology 

(Lee, et al. 2007) for the California Healt

Information Survey (CHIS).

2

is sponsored and directed by the National Cancer

Institute’s Division of Cancer Control and Population

Sciences. The baseline year is 2003, and data from

the first follow-up sample in 2005 are also available

(see http://hints.cancer.gov). A second follow-up

sample (for 2007) is currently being implemented.

Each biennial sample is drawn using a random-digitdial (RDD) sample design to produce a representative

sample of telephone households in the country.

Exchanges with high percentages of Blacks and

Hispanics were oversampled in 2003, in order to

provide a larger yield of these important subgroups.

In a second stage of selection, one adult was randomly

selected among all adults living in the sampled

household. This adult was recruited to complete

the main survey instrument by telephone interview4

.

Weights are assigned to account for all of the stages

of selection (from the RDD sampling frame and

within the household), and for attrition from

noncontacts, screener nonresponse, and interview

nonresponse. These weights are designed to provide

approximately unbiased estimators of population

totals using a modified Horvitz-Thompson estimator

(see for example Cochran 1977, Section 9A.7)5

.

Replicate weights are also provided to allow for

consistent variance estimation. The replicate weights

for all of the biennial HINTS surveys are based on the

jackknife replication method, with R = 50 replicate

weights for each survey year. The replicate weights

4

In HINTS 2005, a small number of persons completed interviews via the Internet, as part of an experimental study nested within the main H

5

Nonresponse is viewed as a further ‘pseudo’ stage of sampling, in which it is assumed that respondents are a simple random sample from all s

within carefully defined response cells (the ‘pseudo-randomization paradigm’: see for example Oh and Scheuren 1983).
sociodemographic subgroups of general interest, as

well as standard errors (the square roots of the

jackknife variance estimates).

inferences from multiple crosssectional survey

2003 2005

SUBGROUP Weighted % Standard Error Weighted % Standard Error

ALL 19.7% 0.6% 28.3% 0.7%

AGE

18–34 23.5% 1.3% 32.6% 1.5%

35–49 23.3% 1.2% 32.5% 1.6%

50–64 20.6% 1.2% 30.0% 1.4%

65+ 4.2% 0.5% 9.6% 0.8%

EDUCATION LEVEL

Less than high school 6.5% 1.4% 6.4% 1.1%

High school graduate 12.0% 0.9% 19.9% 1.6%

Some college 23.9% 1.3% 34.7% 1.9%

College graduate or more 36.0% 1.3% 46.5% 1.6%

RACE

Non-Hispanic White 23.1% 0.8% 33.3% 1.1%

Non-Hispanic Black 13.6% 1.7% 23.3% 3.4%

Hispanic 7.2% 1.0% 11.2% 2.0%

Non-Hispanic other 22.1% 2.4% 28.2% 3.7%

GENDER

Male 16.7% 0.8% 25.3% 1.4%

Female 22.4% 0.9% 31.0% 0.9%

ANNUAL INCOME

Less than $25,000 10.1% 0.9% 18.0% 1.5%

$25,000 to $49,999 16.6% 1.2% 25.6% 1.9%

$50,000 to $74,999 27.3% 1.6% 30.4% 2.0%

$75,000 or more 36.3% 1.8% 44.6% 2.1%

Table 2 Estimates of percentages of adults who have ever looked for cancer information online.

6

The exact derivation of the example percentage from the HINTS questionnaire items is given in Appendix A.
4

g

the estimate of the change?

• Goal 2: Estimating a change in a characteristic

while controlling for covariates (across and

within subgroups):

– Example 1: Has the percentage of persons who

have ever looked for cancer information online

changed in the last two years, after controlling

for age, education level, and gender?

Note that Goals 1 and 2 are relevant to te

differences or change in responses to surv

that are identical (or comparable) across y

Goal 3 would be used to combine across 

obtain one larger sample size.
deviation) in year s. We label the true value in year s

as θs, the estimated value as θˆ s, and the estimated

variance (the square of the standard error) as v(θˆ s).

The true change between years is Δ = θ2 -θ1, with

consistent estimator Δ = ˆ θˆ 2 -θˆ 1. Because the samples

are independent, the variance is the sum of the two

variances, and a consistent variance estimator is

Table 3-1 provides a summary of this information.

A hypothesis test for the null hypothesis of no change

(θ1 = θ2) can be tested against a one-sided (θ1 < θ2) or

two-sided (θ1 = θ2) alternative. The one-sided

alternative may be more appropriate when any change

that occurs is expected to be positive change (such as

in the degree of Internet usage). The test statistic is 

freedom will be the most conservative (i.e., giv

widest confidence intervals), thereby reducing 

likelihood of committing a Type I error. Using

Welch’s method, the number of degrees of free

will be something between 49 and 98. It should 

noted that all of these t-distributions are close t

other, and close to the standard normal distribu

(i.e., the corresponding percentiles are nearly e

For most applications for HINTS, the Welch

approximation assuming 49 degrees of freedom 

each year will be reasonable. The degrees of fr

for the chi-square distribution can be no larger 

the set of independent nonzero squares that und

the variance estimator. Suppose for example th

particular estimate is restricted to a limited sub

of the sample, so that many of the replicate squ

deviations are negligibly close to zero (see the

equation for vr(θˆ) at the end of Section 1). In t

case, a smaller number of degrees of freedom s

be used7

. SAS/SUDAAN does allow the user to

specify degrees of freedom if the user wishes t

overrule the software’s choice. It should be not

without manual specification the SAS/SUDAA

program uses as degrees of freedom the total n

of replicates, and the STATA software uses as d

of freedom: the total number of replicates minu

respectively. STATA doesn’t appear to allow fo

re-specification of degrees of freedom. These d

of freedom are ‘liberal’ (just beyond the high e

the ‘acceptable’ range as per the Welch method

Table 3-2 on the next page presents one-sided a

two-sided p-values for the null hypothesis of n

change between 2003 and 2005 in percentages 

adults who had ever looked for cancer informa

Table 3-1 Summary of Estimating Changes Using

Two Independent Surveys.

7

A procedure recommended here is to consider as ‘negligible’ any replicate square in the set of replicate squares that is less than 1% of the median

square, which will eliminate spurious ‘essentially nonzero’ squares.The software packages do not currently do this or anything similar to it, so the

interested user will need to do this in a ‘manual’ way.

1 θ1 θˆ 1 v(θˆ 1)

2 θ2 θˆ 2 v(θˆ 2)

Change Δ=θ2 -θ1 Δ=ˆ θˆ 2 -θˆ 1 v(Δˆ ) = v(θˆ 1)-v(θˆ 2)

Year True Value Estimated

Value

Variance of Estimate
6

Table 3-2 Estimates of differences of percentages of adults who have ever looked for cancer inform

between 2003 and 2005.

SUBGROUP

ALL 19.7% 0.6% 28.3% 0.7% 8.6% 0.9% 0.0000% 

AGE

18–34 23.5% 1.3% 32.6% 1.5% 9.1% 2.0% 0.0013% 

35–49 23.3% 1.2% 32.5% 1.6% 9.3% 2.0% 0.0014% 

50–64 20.6% 1.2% 30.0% 1.4% 9.4% 1.8% 0.0001% 

65+ 4.2% 0.5% 9.6% 0.8% 5.4% 0.9% 0.0000% 

EDUCATION LEVEL

Less than high school 6.5% 1.4% 6.4% 1.1% -0.1% 1.7% 96.77% 

High school graduate 12.0% 0.9% 19.9% 1.6% 8.0% 1.8% 0.0033% 

Some college 23.9% 1.3% 34.7% 1.9% 10.7% 2.4% 0.0014% 

College graduate or more 36.0% 1.3% 46.5% 1.6% 10.5% 2.1% 0.0002% 

RACE

Non-Hispanic White 23.1% 0.8% 33.3% 1.1% 10.1% 1.3% 0.0000% 

Non-Hispanic Black 13.6% 1.7% 23.3% 3.4% 9.6% 3.8% 1.22% 

Hispanic 7.2% 1.0% 11.2% 2.0% 4.1% 2.2% 7.36% 

Non-Hispanic other 22.1% 2.4% 28.2% 3.7% 6.1% 4.4% 16.58% 

GENDER

Male 16.7% 0.8% 25.3% 1.4% 8.6% 1.7% 0.0001% 

Female 22.4% 0.9% 31.0% 0.9% 8.6% 1.2% 0.0000% 

ANNUAL INCOME

Less than $25,000 10.1% 0.9% 18.0% 1.5% 7.9% 1.8% 0.0021% 

$25,000 to $49,999 16.6% 1.2% 25.6% 1.9% 9.0% 2.2% 0.0101% 

$50,000 to $74,999 27.3% 1.6% 30.4% 2.0% 3.1% 2.5% 22.85% 

$75,000 or more 36.3% 1.8% 44.6% 2.1% 8.3% 2.8% 0.34% 

The table shows that for all but four groups (less than

high school, Hispanic, non-Hispanic other, and

$50,000–$74,999) we would reject the two-sided test

of no change at the 5% significance level. Note that

viewed as significant results. There are m

multiple comparisons tests that are less co

than the Bonferroni approach; these are av

current versions of both SAS and STATA 

8

Note that these are percentages: .0021% is .000021, 5.2% is .052.This allows for greater clarity (more significant digits).

2003

Weighted

Estimate

θˆ 1

Standard

Error

√v(θˆ 1)

2005

Weighted

Estimate

θˆ 2

Standard

Error

√v(θˆ 2)

Estimate of

2003 to

2005

Change

Δˆ

Standard

Error

√v(Δ)ˆ

Two-sid

p-value 

Test of N

Change
using the corresponding t-distribution. Table 3-3

confidence intervals include zero for these 

SUBGROUP

ALL 8.6% 0.9% 6.8% 10.4%

AGE

18–34 9.1% 2.0% 5.2% 13.1%

35–49 9.3% 2.0% 5.3% 13.3%

50–64 9.4% 1.8% 5.8% 13.1%

65+ 5.4% 0.9% 3.6% 7.2%

EDUCATION LEVEL

Less than high school -0.1% 1.7% -3.5% 3.4%

High school graduate 8.0% 1.8% 4.3% 11.6%

Some college 10.7% 2.4% 6.1% 15.4%

College graduate or more 10.5% 2.1% 6.4% 14.5%

RACE

Non-Hispanic White 10.1% 1.3% 7.6% 12.7%

Non-Hispanic Black 9.6% 3.8% 2.1% 17.1%

Hispanic 4.1% 2.2% -0.4% 8.5%

Non-Hispanic other 6.1% 4.4% -2.6% 14.8%

GENDER

Male 8.6% 1.7% 5.3% 11.8%

Female 8.6% 1.2% 6.1% 11.1%

ANNUAL INCOME

Less than $25,000 7.9% 1.8% 4.4% 11.5%

$25,000 to $49,999 9.0% 2.2% 4.6% 13.3%

$50,000 to $74,999 3.1% 2.5% -2.0% 8.1%

$75,000 or more 8.3% 2.8% 2.8% 13.7%

Table 3-3 Confidence intervals for differences in percentages of adults who have ever looked for

cancer information online, between 2003 and 2005.

Standard

Error

√v(Δ)ˆ

Estimate of

2003 to

2005

Change

Δˆ

Lower

Bound

95% C.I.

Upper

Bound

95% C.I.
8

that if the data files are combined properly, the

analyses of Goal 1 can also be easily reproduced

using the combined data set.

The main purpose of Goal 3 is to allow an augmented

sample size: both years can be combined, virtually

doubling the sample size. This will considerably

improve precision for those characteristics which do

not change much between the years. 

To create the combined data file, one can concatenate

the 2003 and 2005 public use files so that the number

of respondents in the combined data file is the sum of

the respondents from the two individual data files.

Two main tasks are required to combine the data files.

First, variables used in the analyses should have the

same name and values or categories in both data files.

Section A of the Appendix describes how variables

are redefined for the tasks in this document. Second,

create a set of new statistical weights as shown in

Table 4. There will be 101 weights in the combined

FWGT50 from the sample persons from t

2003 survey, and we use the final weight 

all 50 replicates) for sample persons from 

2005 survey. Replicate weights equal to th

weight essentially result in zero sums of s

contributed to the variance estimator from 

replicates. For the first 50 replicate weigh

HINTS 2003 survey contributes variance. 

remaining 50 replicate weights (NFWGT

NFWGT100), we use replicate weights F

FWGT50 from the sample persons from t

2005 survey, and we use the final weight 

all 50 replicates) for sample persons from 

2003 survey. For replicate weights 51 thro

only the HINTS 2005 survey contributes 

When the sums of squares for all 100 rep

put together, the result is a sum of HINTS 

HINTS 2005 variance, as desired (as the 

in fact independent).

It is also necessary to define a YEAR fiel

2003 (or 1) for HINTS 2003 sample mem

equal to 2005 (or 2) for HINTS 2005 sam

members. The Goal 1 Δ = ˆ θˆ 2 -θˆ 1, with corr

standard errors, test statistics, and confide

intervals, can be easily (and correctly) est

this combined data set using a contrast wi

YEAR field (+1 for HINTS 2005 records 

HINTS 2003 records). Appendix A provid

syntax for computing the new replicate w

SUDAAN syntax for calculating the estim

difference10. Appendix B provides corresp

STATA code11.

9

Under the title “Adjust replicate weights for the combined dataset”.

10Under the title “Test for differences across years using combined dataset.”.

11Under the titles “Create the replicate weights for the combined data” and “Test for differences across years using combined data. . . . ”.

Table 4 Construction of statistical weights for the

combined data file.

Final Sample Replicate Replicate

Weights Weights 1–50 Weights 51–100

Hints 2003 2003 Final 2003 Replicate 2003 Final Weight Weight Weights (FWGT)

(FWGT) (FWGT1–FWGT50)

Hints 2005 2005 Final 2005 Final 2005 Replicate Weight (FWGT) Weight (FWGT) Weights

(FWGT1–FWGT50)

Combined Final Weight Final Replicate Final Replicate

Data (NFWGT) Weights Weights

(NFWGT1–NFWGT50) (NFWGT51–NFWGT100)
population (all groups within themselves had no

change in Internet usage). In general, analysts want to

be able to distinguish these compositional changes

from actual trends in the characteristic of interest.

In this section, we explore how to conduct analyses

that search for ‘true’ non compositional changes in

HINTS responses between 2003 and 2005. For

example, Table 5-1 presents results from checking for

2003 to 2005 differences using logistic regression

(with the binary dependent variable equal to 1 if ever

Internet searched, and 0 otherwise). The beta

coefficients represent effects on a log-odds12 scale: the

estimated odds ratios are also given (the transformed

Table 5-1 Changes in percentages of adults who have ever looked for cancer information online between 

and 2005 controlling for age, education level, and gender.

SUBGROUP

INTERCEPT -1.74 0.11 0.17 0.14 0.2

SURVEY YEAR

2003 0.00 0.00 1.00 1.00 1.0

2005 0.51 0.06 1.66 1.48 1.8

AGE

18–34 1.57 0.10 4.78 3.93 5.8

35–49 1.45 0.09 4.27 3.57 5.1

50–64 1.32 0.10 3.75 3.06 4.6

65+ 0.00 0.00 1.00 1.00 1.0

EDUCATION LEVEL

Less than high school -2.24 0.16 0.11 0.08 0.1

High school graduate -1.31 0.09 0.27 0.23 0.3

Some college -0.59 0.08 0.55 0.47 0.6

College graduate or more 0.00 0.00 1.00 1.00 1.0

GENDER

Male -0.36 0.07 0.70 0.60 0.8

Female 0.00 0.00 1.00 1.00 1.0

higher of ever having used the Internet to searc

cancer information in 2005 as compared to 200

(with a 95% confidence interval ranging from 4

87% higher). Since the 95% confidence interva

the odds ratio does not include 1, we would rej

hypothesis of no change for Goal 2 example 1. 

table shows higher odds ratios for the younger 

categories compared to the oldest category (65

lower odds ratios for the lower education group

compared to the highest education level group

(‘college graduate or more’). The SAS/SUDAA

STATA code for carrying out this calculation is 

in Appendices A and B respectively.

12The odds of an event is the probability of an event divided by the complement of that probability, or p / (1-p): e.g., an event probability of 1/2

corresponds to the event occurring with odds 1; an event probability of 2/3 corresponds to the event occurring with odds 2. An odds ratio of 1.6

between Events A and B means the following. Suppose Event A has an event probability of 1/3 (an odds ratio of 1/2).Then Event B will have an odd

times higher, or 0.8, which corresponds to an event probability of 44.5%. If Event A has an event probability of 1/2 (odds of 1), then Event B will have

odds of 1.6 (1.6 times 1), which corresponds to an event probability of 61.5%. Note also that the probability p can be computed from the odds O a

= O / (1 + O).The log-odds is the logarithm of the odds (putting the naturally multiplicative odds scale onto an additive scale).

Beta

Coefficient

Standard Error

Beta

Coefficient

Odds Ratio

Lower Bound

95% CI

Odds Ratio

Upper 

95% 

Odds 

vent probability of 1/2 correspon
10

education level is interacted with year. The ‘Education

Level 2003’ parameters represent differences between

each education level and the baseline education level

(‘college graduate or more’) for the baseline year

2003. These would be the estimates for the main

effects for education level in a traditionally structured

table (see for example Korn and Graubard [1999],

both SAS/SUDAAN and STATA. 

If the ‘Education Level 2003’ beta coeffic

estimates and the ‘Education Level 2005 

coefficient estimates are added together, t

summations for each education level are e

that education level (as against the baselin

level) for the year 2005.

Table 5-2 Changes in percentages of adults who have ever looked for cancer information online be

and 2005 controlling for age, education level, and gender, with a year vs. education level in

SUBGROUP

INTERCEPT -1.73 0.10 0.18 0.15 

SURVEY YEAR

2003 0.00 0.00 1.00 1.00 

2005 0.47 0.09 1.60 1.34 

AGE

18–34 1.57 0.10 4.80 3.94 

35–49 1.46 0.09 4.29 3.58 

50–64 1.32 0.10 3.75 3.06 

65+ 0.00 0.00 1.00 1.00 

GENDER

Male -0.36 0.07 0.70 0.60 

Female 0.00 0.00 1.00 1.00 

EDUCATION LEVEL 2003

Less than high school -1.97 0.25 0.14 0.09 

High school graduate -1.40 0.11 0.25 0.20 

Some college -0.64 0.09 0.53 0.44 

College graduate or more 0.00 0.00 1.00 1.00 

EDUCATION LEVEL 2005 VS 2003

Less than high school -0.52 0.32 0.60 0.32 

High school graduate 0.16 0.17 1.17 0.83 

Some college 0.08 0.15 1.09 0.81 

College graduate or more 0.00 0.00 1.00 1.00 

Beta

Coefficient

Standard Error

Beta

Coefficient

Odds Ratio

Lower Bound

95% CI

Odds Ratio
SUBGROUP

INTERCEPT -4.41 0.44 0.01 0.01 0.0

SURVEY YEAR

2003 0.00 0.00 1.00 1.00 1.0

2005 -0.07 0.30 0.93 0.51 1.6

AGE

18–34 2.53 0.48 12.61 4.91 32.4

35–49 1.76 0.50 5.84 2.17 15.7

50–64 1.33 0.57 3.78 1.22 11.7

65+ 0.00 0.00 1.00 1.00 1.0

GENDER

Male -0.08 0.33 0.92 0.48 1.7

Female 0.00 0.00 1.00 1.00 1.0

Beta

Coefficient

Standard Error

Beta

Coefficient

Odds Ratio

Lower Bound

95% CI

Odds Ratio

Upper 

95% 

Odds 

Table 5-3-2 Changes in percentages of adults who have ever looked for cancer information online betwe

2003 and 2005 controlling for age and gender, subsetted to the education level subgroup

‘high school graduate’.

SUBGROUP

INTERCEPT -3.43 0.24 0.03 0.02 0.0

SURVEY YEAR

2003 0.00 0.00 1.00 1.00 1.0

2005 0.64 0.14 1.90 1.45 2.4

AGE

18–34 1.97 0.23 7.15 4.55 11.2

35–49 1.91 0.22 6.76 4.33 10.5

50–64 1.62 0.25 5.03 3.04 8.3

65+ 0.00 0.00 1.00 1.00 1.0

GENDER

Male -0.55 0.18 0.58 0.40 0.8

Female 0.00 0.00 1.00 1.00 1.0

Beta

Coefficient

Standard Error

Beta

Coefficient

Odds Ratio

Lower Bound

95% CI

Odds Ratio

Upper 

95% 

Odds 
12

18–34 1.49 0.17 4.44 3.15 

35–49 1.46 0.17 4.33 3.09 

50–64 1.31 0.18 3.72 2.61 

65+ 0.00 0.00 1.00 1.00 

GENDER

Male -0.50 0.13 0.61 0.47 

Female 0.00 0.00 1.00 1.00 

Table 5-3-4 Changes in percentages of adults who have ever looked for cancer information online b

2003 and 2005 controlling for age and gender, subsetted to the education level subgrou

‘college graduate or more’.

SUBGROUP

INTERCEPT -1.54 0.13 0.21 0.17 

SURVEY YEAR

2003 0.00 0.00 1.00 1.00 

2005 0.46 0.09 1.58 1.33 

AGE

18–34 1.24 0.15 3.45 2.56 

35–49 1.12 0.14 3.08 2.33 

50–64 1.13 0.15 3.10 2.32 

65+ 0.00 0.00 1.00 1.00 

GENDER

Male -0.18 0.08 0.84 0.71 

Female 0.00 0.00 1.00 1.00 

Beta

Coefficient

Standard Error

Beta

Coefficient

Odds Ratio

Lower Bound

95% CI

Odds Ratio
indicated by table number).
14

distinct values or a single value. But in those cases

when estimates from the two years do not differ

much, then combining the data sets will certainly

allow a considerable increase in precision (twice as

large a sample size). This may be very useful for

population subgroups in which the one-year sample

sizes are not very large. 

The average of two survey years may be estimated by

using one of two easy steps: 1) using two separate

data files, and 2) using the combined data file. In the

first approach, we use the mean value θm = 0.5 *

(θ1 + θ2) as the parameter of interest. Table 6-1 shows

how we would compute the mean and its variance.

weighted mean reduces to the unweighted 

Over a short period of time, the populatio

most groups would change very little so t

parameters should be similar; however, th

subgroups increasing or decreasing in size 

immigration. One advantage of using the 

data set with the new weights is that it tak

account change in population size. 

Table 6-2 presents averages of the separat

estimates13 for the percentage of adults wh

looked for cancer information online (θm)

noted in the computation of the confidenc

Table 6-2 uses a symmetric t-distribution 

degrees of freedom14.

Table 6-1 Summary of estimating changes using

two independent surveys.

Year True Value Estimated Value Variance of Estimate

1 θ1 θˆ 1 v(θˆ 1)

2 θ2 θˆ 2 v(θˆ 2)

Average θm = 0.5 ∗ (θ2 -θ1) θˆ m = 0.5 ∗ (θˆ 1 + θˆ 2) v(Δˆ ) = 0.25 ∗ (v(θˆ 1) + v(θˆ 2))

13These separate-year estimates were computed using SAS/SUDAAN and STATA (both programs giving the same answer).The averaging was 

14These t confidence intervals were computed using Excel.
EDUCATION LEVEL

Less than high school 6.5% 1.4% 6.4% 1.1% 6.4% 0.9% 4.7% 

High school graduate 12.0% 0.9% 19.9% 1.6% 16.0% 0.9% 14.2% 

Some college 23.9% 1.3% 34.7% 1.9% 29.3% 1.2% 27.0% 

College grad or more 36.0% 1.3% 46.5% 1.6% 41.2% 1.0% 39.2% 

RACE

Non-Hispanic White 23.1% 0.8% 33.3% 1.1% 28.2% 0.6% 26.9% 

Non-Hispanic Black 13.6% 1.7% 23.3% 3.4% 18.4% 1.9% 14.7% 

Hispanic 7.2% 1.0% 11.2% 2.0% 9.2% 1.1% 7.0% 

Non-Hispanic other 22.1% 2.4% 28.2% 3.7% 25.2% 2.2% 20.8% 

GENDER

Male 16.7% 0.8% 25.3% 1.4% 21.0% 0.8% 19.3% 

Female 22.4% 0.9% 31.0% 0.9% 26.7% 0.6% 25.5% 

ANNUAL INCOME

Less than $25,000 10.1% 0.9% 18.0% 1.5% 14.0% 0.9% 12.3% 

$25,000 to $49,999 16.6% 1.2% 25.6% 1.9% 21.1% 1.1% 18.9% 

$50,000 to $74,999 27.3% 1.6% 30.4% 2.0% 28.9% 1.3% 26.3% 

$75,000 or more 36.3% 1.8% 44.6% 2.1% 40.5% 1.4% 37.7% 
16

15For example, for dichotomous response variables, if one uses the svy: mean or svy: proportion command then the confidence interval will be 

If one uses the svy: tabulate command the confidence interval will be asymmetric (it uses the logit transform).

Table 6-3 Percentages of adults who have ever looked for cancer information online using the com

2003/2005 data file.

SUBGROUP

ALL 19.7% 0.6% 28.3% 0.7% 24.0% 23.1% 

AGE

18–34 23.5% 1.3% 32.6% 1.5% 28.1% 26.2% 

35–49 23.3% 1.2% 32.5% 1.6% 27.9% 26.0% 

50–64 20.6% 1.2% 30.0% 1.4% 25.5% 23.7% 

65+ 4.2% 0.5% 9.6% 0.8% 7.0% 6.1% 

EDUCATION LEVEL

Less than high school 6.5% 1.4% 6.4% 1.1% 6.4% 4.9% 

High school graduate 12.0% 0.9% 19.9% 1.6% 15.9% 14.2% 

Some college 23.9% 1.3% 34.7% 1.9% 29.9% 27.5% 

College grad or more 36.0% 1.3% 46.5% 1.6% 41.2% 39.2% 

RACE

Non-Hispanic White 23.1% 0.8% 33.3% 1.1% 28.2% 26.9% 

Non-Hispanic Black 13.6% 1.7% 23.3% 3.4% 18.4% 15.0% 

Hispanic 7.2% 1.0% 11.2% 2.0% 9.3% 7.2% 

Non-Hispanic other 22.1% 2.4% 28.2% 3.7% 25.5% 21.2% 

GENDER

Male 16.7% 0.8% 25.3% 1.4% 21.1% 19.5% 

Female 22.4% 0.9% 31.0% 0.9% 26.8% 25.6% 

ANNUAL INCOME

Less than $25,000 10.1% 0.9% 18.0% 1.5% 13.7% 12.1% 

$25,000 to $49,999 16.6% 1.2% 25.6% 1.9% 20.5% 18.5% 

$50,000 to $74,999 27.3% 1.6% 30.4% 2.0% 29.0% 26.5% 

$75,000 or more 36.3% 1.8% 44.6% 2.1% 40.8% 38.1% 

2003

Weighted

Estimate

Standard

Error

2005

Weighted

Estimate

Standard

Error

2003 to

2005 θw

Estimate

Lower

Bound

95% CI
estimated total number of the population (or a

subgroup) who had ever looked for cancer

information using the Internet. The total number of

users can be expressed as the product of the

prevalence and the population size. Thus, the

programs that were used to estimate prevalence can

also be used to estimate the total by modification of

the option statements in the program; for example, we

could obtain estimates of the total in SAS/SUDAAN

users guide can easily be extended to ordinal lo

regression and linear regression models. In SU

the appropriate command for ordinal/nominal

multinomial logistic regression is PROC MULT

In STATA, the corresponding command for ord

logistic regression is SVY:OLOGIT. REGRESS

(SVY:REGRESS) is the proper command for li

regression in SAS/SUDAAN (STATA).
18

and averages using combined crosssectional survey data from multiple years. Available as a method

on www.chis.ucla.edu. 

Oh, H. L., and Scheuren, F. S. (1983). Weighting adjustments for unit nonresponse, in Incomplete Da

Surveys, Vol. II: Theory and Annotated Bibliography (W. G. Madow, I. Olkin, and D. B. Rubin, eds.

Academic Press. 

Research Triangle Institute (2004). SUDAAN Example Manual: Release 9.0. Research Triangle Park, 

Research Triangle Institute.

StataCorp. 2007. Stata Statistical Software: Release 10. College Station, TX: StataCorp LP. 
p; ;

proc format; %include forms2;

proc format;

value yearf

1='2003'

2='2005'

;

value agef

1='18-34'

2='35-49'

3='50-64'

4='65+'

;

value racef

1='NH White'

2='NH Black'

3='Hispanic'

4='NH Other'

;

value educf

1='Less than High School Grad'

2='High School Grad'

3='Some College'

4='College Grad'

;

value sexf

1='Male'

2='Female'

;

value incomef

1='<$25K'

2='$25K-<$50K'

3='$50K-<$75K'

4='$75K+'

;

value yesno

0='No'

1='Yes'

;

run;
20

/*Demographic Characteristics*/

label sex='Gender';

sex=spgender;

format sex sexf.;

label age='Age Group';

if 18<=spage<=34 then age=1;**18-34;

else if 35<=spage<=49 then age=2;**35-49;

else if 50<=spage<=64 then age=3;**50-64;

else if 65<=spage<=96 then age=4;**65+;

format age agef.;

label race='Race/Ethnicity';

if raceEthn=1 then race=3;**Hispanic;

else if raceEthn=2 then race=1;**NH White;

else if raceEthn=3 then race=2;**NH Black;

else if 4<=raceEthn <=7 then race=4;**NH Other;

format race racef.;

label income='Household Income';

if HHIncB=1 then income=1;**<$25K;

else if HHIncB in (2,3) then income=2;**$25K-<$50K;

else if HHIncB in (4,5) then income=HHIncB-1;**$50K-<$75K/$75K+;

format income incomef.;

label educ="Education";

if educA in (1,2,3,4) then educ=EducA;

format educ educf.;
/*Respondents whose last search for cancer information was online*/

if CA12WhereLookCancerInfo=7 then InternetForCancer=1;**Yes;

/*Respondents who never looked for health information online*/

else if CA08SeekCancerInfo=2 or GA1UseInternet=2 then InternetForCancer=0;**No;

/*Respondents who have used the internet for general health information*/

else if CA15InternetForCancer in (1,2) 

then InternetForCancer=mod(CA15InternetForCancer,2);**Yes/No;

end;

format InternetForCancer yesno.;

/*Adjust Replicate Weights for the combined dataset*/

array origwgts[50] fwgt1-fwgt50;

array newwgts[100] nfwgt1-nfwgt100;

nfwgt=fwgt;

do i = 1 to 50;

if srvyYear=1 then do;***2003;

newwgts[i] = origwgts[i];

newwgts[i+50] = fwgt;

end;

else if srvyYear=2 then do;***2005;

newwgts[i] = fwgt;

newwgts[i+50] = origwgts[i];

end;

end;

run;
22

proc descript data=combined design=jackknife ddf=99;

weight nfwgt;

jackwgts nfwgt1-nfwgt100 / adjjack=.98;

class srvyYear InternetForCancer _one_ /nofreq;

var InternetForCancer;

catlevel 1;

tables _one_;

diffvar srvyYear=(2 1) / name="Change from 2003 to 2005";

print nsum percent sepercent lowpct uppct t_pct p_pct / style=nchs;

run;

/*View percentages by specified years using combined dataset.*/

/*This code recreates the yearly percentages and standard errors in columns 1–4 of Table 3-2.*/

proc descript data= combined design=jackknife ddf=99;

weight nfwgt;

jackwgts nfwgt1-nfwgt100 / adjjack=.98;

class srvyYear age educ sex income race InternetForCancer/nofreq;

var InternetForCancer;

catlevel 1;

table srvyYear*(age educ sex income race);

print nsum percent sepercent lowpct uppct / style=nchs;

run;

/*Test for differences across years for a subset of demographic va riables using combined datase

/*This code recreates the difference estimates and two-sided p-values in columns 5–7 of Table 3-

It also regenerates the 95% confidence intervals provided in Table 3-3. P-values will va ry based 

of denominator degrees of freedom.*/

proc descript data= combined design=jackknife ddf=99;

weight nfwgt;

jackwgts nfwgt1-nfwgt100 / adjjack=.98;

class srvyYear age educ sex income race InternetForCancer/nofreq;

var InternetForCancer;

catlevel 1;

diffvar srvyYear=(2 1) / name="Change from 2003 to 2005";

tables age educ sex income race;

print nsum percent sepercent lowpct uppct t_pct p_pct / style=nchs;

run;
/*Assess differences across years while controlling for cova riates—education, age, and gender—using 

combined dataset. Includes an interaction term to test for differential change by levels of education. 

Table 5-2.*/

proc rlogist data= combined design=jackknife ddf=99;

weight nfwgt;

jackwgts nfwgt1-nfwgt100 / adjjack=.98;

class srvyYear educ age sex/nofreq;

model InternetForCancer=srvyYear educ age sex srvyYear*educ;

effects srvyYear / name="SRVYYEAR";**Wald F values for SRVYYEAR;

effects educ / name="EDUC";**Wald F values for EDUC;

reflev srvyYear=1;

run;

/*Assess differences across years for each level of education while controlling for age and gender.*/

/*Note subpopn statement that delineates testing at each level of education.*/

/*See Tables 5-3-1 through 5-3-4*/

proc rlogist data= combined design=jackknife ddf=99;

weight nfwgt;

jackwgts nfwgt1-nfwgt100 / adjjack=.98;

subpopn educ=1 / name="Less than High School Only";

class srvyYear age sex / nofreq;

model InternetForCancer=srvyYear age sex;

reflev srvyYear=1;

run;

proc rlogist data= combined design=jackknife ddf=99;

weight nfwgt;

jackwgts nfwgt1-nfwgt100 / adjjack=.98;

subpopn educ=2 / name="High School Graduates Only";

class srvyYear age sex / nofreq;

model InternetForCancer=srvyYear age sex;

reflev srvyYear=1;

run;
24

proc rlogist data= combined design=jackknife ddf=99;

weight nfwgt;

jackwgts nfwgt1-nfwgt100 / adjjack=.98;

subpopn educ=4 / name="College Graduates Only";

class srvyYear age sex / nofreq;

model InternetForCancer=srvyYear age sex;

reflev srvyYear=1;

run;

GOAL 3—Estimating Averages by Combining 2003 and 2005 Data. (See section 6.)

/*Obtain weighted percentages by demographic subgroup using combined dataset. See Table 6-

proc descript data= combined design=jackknife ddf=99;

weight nfwgt;

jackwgts nfwgt1-nfwgt100 / adjjack=.98;

class srvyYear age educ sex income race InternetForCancer/nofreq;

var InternetForCancer;

catlevel 1;

tables age educ sex income race;

print nsum percent sepercent lowpct uppct / style=nchs;

run;
generate srvyyear = 1

label variable srvyyear "Survey Year"

*** Create the demographic variables

recode spgender (1=1 "Male") (2=2 "Female"), generate(sex)

label variable sex "Gender"

recode spage (18/34=1 "18-34") (35/49=2 "35-49") (50/64=3 "50-64") (65/96=4 "65 +") (nonmissing=.),

generate(age)

label variable age "Age Group"

recode raceethn (1=3 "Hispanic") (2=1 "NH White") (3=2 "NH Black") (4/7=4 "NH Other") (nonmissing=

generate(race)

label variable race "Race/Ethnicity"

recode hhincb (1=1 "<$25K") (2 3=2 "$25K-<$50K") (4=3 "$50K-<$75K") (5=4 "$75K +") (nonmissing=

generate(income)

label variable income "Household Income"

recode educa (1=1 "Less than High School Grad") (2=2 "High School Grad") (3=3 "Some College") (4=4

"College Grad") (nonmissing=.), generate(educ)

label variable educ "Education"

* Create the variable internetforcancer

generate internetforcancer = .

* Respondents who never looked for health information online

replace internetforcancer = 0 if hc9seekcancerinfo == 2

replace internetforcancer = 0 if hc20useinternet == 2

replace internetforcancer = 0 if hc27lastonlinehealth == 5

* Respondents who have used the internet for general health information

replace internetforcancer = 0 if hc29internetforcancer == 2

replace internetforcancer = 1 if missing(internetforcancer) & hc29internetforcancer == 1

label variable internetforcancer "Have you ever specifically looked for cancer info online?"

* Create the replicate weights for the combined data

generate nfwgt = fwgt

foreach i of numlist 1/50 {

generate nfwgt`i' = fwgt`i'

}
26

ga1useinternet ca15internetforcancer

generate srvyyear = 2

* Create the demographic variables

recode spgender (1=1 "Male") (2=2 "Female"), generate(sex)

recode spage (18/34=1 "18-34") (35/49=2 "35-49") (50/64=3 "50-64") (65/96=4 "65 +") (nonmissing

generate(age)

recode raceethn (1=3 "Hispanic") (2=1 "NH White") (3=2 "NH Black") (4/7=4 "NH Other") (nonmis

generate(race)

recode hhincb (1=1 "<$25K") (2 3=2 "$25K-<$50K") (4=3 "$50K-<$75K") (5=4 "$75K +") (nonmis

generate(income)

recode educa (1=1 "Less than High School Grad") (2=2 "High School Grad") (3=3 "Some College") 

"College Grad") (nonmissing=.), generate(educ)

* Create the variable internetforcancer

generate internetforcancer = .

* Respondents whose last search for cancer information was online

replace internetforcancer = 1 if ca12wherelookcancerinfo == 7

* Respondents who never looked for health information online

replace internetforcancer = 0 if missing(internetforcancer) & (ca08seekcancerinfo == 2 | ga1useintern

* Respondents who have used the internet for general health information

replace internetforcancer = 2 - ca15internetforcancer if missing(internetforcancer) & (ca15internetfor

ca15internetforcancer == 2)

* Create the replicate weights for the combined data

generate nfwgt = fwgt

foreach i of numlist 1/50 {

generate nfwgt`i' = fwgt

}

foreach i of numlist 51/100 {

local j = `i' - 50

generate nfwgt`i' = fwgt`j'

}
svyset [pw=nfwgt], jkrw(nfwgt1-nfwgt100, multiplier(0.98)) vce(jack) mse

GOAL 1—Estimating Changes Without Controlling for Other Factors. (See section 3.)

***The following codes recreate the yearly percentages, differences, standard errors, and two-sided p-

***values in columns 1-7 of Table 3-2. Also included are the 95% C.I.’s provided in Table 3-3.

* Test for differences across years using combined data -- overall

svy: mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

* Test for differences across years using combined data -- by age group

* among non-missing age

generate selectedgroup = (age ~= .)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among those with age 18-34

generate selectedgroup = (age == 1) if !missing(age)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among those with age 35-49

generate selectedgroup = (age == 2) if !missing(age)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among those with age 50-64

generate selectedgroup = (age == 3) if !missing(age)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among those with age 65+

generate selectedgroup = (age == 4) if !missing(age)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup
28

drop selectedgroup

* among those high school graduate

generate selectedgroup = (educ == 2) if !missing(educ)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among those some college

generate selectedgroup = (educ == 3) if !missing(educ)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among those college graduate

generate selectedgroup = (educ == 4) if !missing(educ)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* Test for differences across years using combined data -- by sex

* among non-missing sex

generate selectedgroup = (sex ~= .)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among males

generate selectedgroup = (sex == 1) if !missing(sex)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among females

generate selectedgroup = (sex == 2) if !missing(sex)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup
drop selectedgroup

* among those $25K -< $50K

generate selectedgroup = (income == 2) if !missing(income)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among those $50K -< $75K

generate selectedgroup = (income == 3) if !missing(income)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among those $75K+

generate selectedgroup = (income == 4) if !missing(income)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* Test for differences across years using combined data -- by race group

* among non-missing race

generate selectedgroup = (race ~= .)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among NH white

generate selectedgroup = (race == 1) if !missing(race)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup

* among NH black

generate selectedgroup = (race == 2) if !missing(race)

svy, subpop(selectedgroup): mean internetforcancer, over(srvyyear)

lincom [internetforcancer]2 - [internetforcancer]1

drop selectedgroup
30

GOAL 2—Estimating Changes Controlling for Other Factors. (See section 5.)

*** Logistic Regression – adjusted by education, age and sex. (Table 5-1)

char srvyyear [omit] 1

char educ [omit] 4

char age [omit] 4

char sex [omit] 2

xi: svy: logit internetforcancer i.srvyyear i.educ i.age i.sex

test _Isrvyyear_2 _Ieduc_1 _Ieduc_2 _Ieduc_3 _Iage_1 _Iage_2 _Iage_3 _Isex_1 _cons, nosvyadjust

test _Isrvyyear_2 _Ieduc_1 _Ieduc_2 _Ieduc_3 _Iage_1 _Iage_2 _Iage_3 _Isex_1, nosvyadjust

test _Isrvyyear_2, nosvyadjust

test _Ieduc_1 _Ieduc_2 _Ieduc_3, nosvyadjust

test _Iage_1 _Iage_2 _Iage_3, nosvyadjust

test _Isex_1, nosvyadjust

xi: svy, or: logit internetforcancer i.srvyyear i.educ i.age i.sex

*** Logistic Regression -- adjusted by education, age, sex and i.srvyyear*i.educ. (Table 5-2)

xi: svy: logit internetforcancer i.srvyyear i.educ i.age i.sex i.srvyyear*i.educ

test _Isrvyyear_2 _Ieduc_1 _Ieduc_2 _Ieduc_3 _Iage_1 _Iage_2 _Iage_3 _Isex_1 _IsrvXedu_2_1 _Is

_IsrvXedu_2_3 _cons, nosvyadjust

test _Isrvyyear_2 _Ieduc_1 _Ieduc_2 _Ieduc_3 _Iage_1 _Iage_2 _Iage_3 _Isex_1 _IsrvXedu_2_1 _Is

_IsrvXedu_2_3, nosvyadjust

test _Isrvyyear_2, nosvyadjust

test _Ieduc_1 _Ieduc_2 _Ieduc_3, nosvyadjust

test _Iage_1 _Iage_2 _Iage_3, nosvyadjust

test _Isex_1, nosvyadjust

test _IsrvXedu_2_1 _IsrvXedu_2_2 _IsrvXedu_2_3, nosvyadjust

xi: svy, or: logit internetforcancer i.srvyyear i.educ i.age i.sex i.srvyyear*i.educ

*** Logistic Regression – adjusted by age and sex, stratified by education. (Table 5-3)

* among those less than high school

generate selectedgroup = (educ == 1) if !missing(educ)

xi: svy, subpop(selectedgroup): logit internetforcancer i.srvyyear i.age i.sex

test _Isrvyyear_2 _Iage_1 _Iage_2 _Iage_3 _Isex_1 _cons, nosvyadjust

test _Isrvyyear_2 _Iage_1 _Iage_2 _Iage_3 _Isex_1, nosvyadjust

test _Isrvyyear_2, nosvyadjust

test _Iage_1 _Iage_2 _Iage_3, nosvyadjust

test _Isex_1, nosvyadjust
y, pp(gp)gyyg

drop selectedgroup

* among those some college

generate selectedgroup = (educ == 3) if !missing(educ)

xi: svy, subpop(selectedgroup): logit internetforcancer i.srvyyear i.age i.sex

test _Isrvyyear_2 _Iage_1 _Iage_2 _Iage_3 _Isex_1 _cons, nosvyadjust

test _Isrvyyear_2 _Iage_1 _Iage_2 _Iage_3 _Isex_1, nosvyadjust

test _Isrvyyear_2, nosvyadjust

test _Iage_1 _Iage_2 _Iage_3, nosvyadjust

test _Isex_1, nosvyadjust

xi: svy, or subpop(selectedgroup): logit internetforcancer i.srvyyear i.age i.sex

drop selectedgroup

* among those college graduate

generate selectedgroup = (educ == 4) if !missing(educ)

xi: svy, subpop(selectedgroup): logit internetforcancer i.srvyyear i.age i.sex

test _Isrvyyear_2 _Iage_1 _Iage_2 _Iage_3 _Isex_1 _cons, nosvyadjust

test _Isrvyyear_2 _Iage_1 _Iage_2 _Iage_3 _Isex_1, nosvyadjust

test _Isrvyyear_2, nosvyadjust

test _Iage_1 _Iage_2 _Iage_3, nosvyadjust

test _Isex_1, nosvyadjust

xi: svy, or subpop(selectedgroup): logit internetforcancer i.srvyyear i.age i.sex

drop selectedgroup

GOAL 3—Estimating Averages by Combining 2003 and 2005 Data. (See section 6.)

*** Obtain weighted percentages using combined dataset. (Table 6-3)

* Estimate using the combined data by age group

svy: tabulate age internetforcancer, row se ci format(%8.5f) percent

* Estimate using the combined data by education group

svy: tabulate educ internetforcancer, row se ci format(%8.5f) percent

* Estimate using the combined data by sex group

svy: tabulate sex internetforcancer, row se ci format(%8.5f) percent

* Estimate using the combined data by income group

svy: tabulate income internetforcancer, row se ci format(%8.5f) percent

* Estimate using the combined data by race group

svy: tabulate race internetforcancer, row se ci format(%8.5f) percent
to the tdistribution in this case is . The method for computing the degrees of freedom 

difference of normally-distributed simple random sample estimators with unequal variances from inde

surveys is taken from Bickel and Doksum (1977). Section 6.4C recommends the Welch approximatio

computes as the degrees of freedom k for the estimating equation 

where . 

In our application n1 and n2 are both 50. If v(θˆ 1) and v(θˆ 2) are also both equal, then c = 1/2

and 

That is the maximum value of k. If v(θˆ 1) is much smaller, or much larger, than v(θˆ 2),

then 

.

49 is the minimum value of k. Thus 49 is the ‘conservative’ approximation for the degrees of freedom

widest confidence intervals (using the t distribution on 49 degrees of freedom). If v(θˆ 1) and v(θˆ 2) are un

both in the same order of magnitude, then Welch’s approximation value can be used to generate an ap

which will be in the range [49,98].

32

49

49

1 1 2

⎥ =

⎦

⎤

⎢

⎣

⎡ ≈

−

k

16The pseudo-values may not necessarily have a normal distribution: it is good practice to check this assumption and make sure there is not ku

reduce the effective degrees of freedom.

17It actually may be smaller if for a particular statistic many of the replicates are effectively equal to the full-sample estimator. 49 is actually the ma
pgp

Phone: 1-800-4-CANCER (1-800-422-6237) 

TDD: 1-800-332-8615

Other NCI or DHHS Sources of Cancer Information 

National Cancer Institute: www.cancer.gov

Office of Education and Special Initiatives (OESI): www.cancer.gov/aboutnci/oesi

Office of Communications: www.cancer.gov/aboutnci/office-of-communications/page3 

American Cancer Society (ACS)

www.cancer.org/docroot/home/index.asp 

F E D E R A L LY-SPONSORED PROGRAM PLANNING RESOURCE

Cancer Control P.L.A.N.E.T.

http://cancercontrolplanet.cancer.gov/

Research-tested Intervention Programs (RTIPs) 

http://rtips.cancer.gov/rtips/index.do 

Guide to Community Preventive Services

www.thecommunityguide.org/ 

RESEARCH TOOLS AND RESOURCES

Behavioral Risk Factor Surveillance System (BRFSS)

www.cdc.gov/brfss/ 

National Health Interview Survey (NHIS)

www.cdc.gov/nchs/nhis.htm

Current Population Survey (CPS)

www.census.gov/cps/ 

Surveillance, Epidemiology, and End Results (SEER)

http://seer.cancer.gov/

Pew Internet and American Life Project

http://www.pewinternet.org/data.asp
hints.cancer.gov

NIH Publication No. 08-6435

Printed June 2008