PROBABILITY BASED INTERNET SURVEYS:

 A SYNOPSIS OF EARLY METHODS 

AND SURVEY RESEARCH RESULTS1

 Vicki Huggins, Knowledge Networks and Joe Eyerman, Research Triangle Institute 

Abstract 

Successfully targeting a nationally representative panel sample over the Internet has been intractable, primarily 

because a large proportion of U.S. households do not have Internet access. This paper presents a new methodology 

created and implemented at Knowledge Networks that overcomes this inherent shortcoming. The new methodology 

begins with selection of a stratified, random sample of households using RDD telephone methods. By phone, the 

sampled households are asked to participate in the Knowledge Networks research panel sample. Once recruited, the 

households are then equipped with simple Internet access devices attached to their televisions that are used to field 

multi-media based surveys. To improve the efficiency of sampling, panel members are sent profile surveys that 

collect information on their demographic, economic, political and social characteristics. Once panel members 

complete the core profile survey, they are available for assignment to specific surveys according to specified 

sampling criteria. This paper will briefly discuss the sample design and methods of this new survey mode and will 

then focus largely on the survey research results to date that identify and measure sampling and nonsampling error. 

Application of this methodology is about two years old and we now have considerable information on response 

rates, coverage and nonresponse bias, and overall measures of data quality to share. 

I. Introduction 

We start with the goal of selecting survey research samples that can statistically support 

inferences about the total population of households in the United States, and/or subsets within 

the population. Nationally representative Internet based sampling frames do not currently exist 

because every household in the U.S. cannot be accessed via the Internet. This results in serious 

undercoverage that can significantly affect outcome study variables. Statistics from the Current 

Population Survey for August 2000 indicate that 51% of the U.S. households have computers 

and 41.5% of households have access to the Internet. [Newburger, 2001] Any serious attempt to 

do national household surveys needs to take into account this undercoverage. 

The Knowledge Networks solution is to utilize standard Random Digit Dial (RDD) sampling to 

obtain a representative sample of U.S. telephone households and then equip those households 

with a WebTV unit for survey administration. This approach greatly reduces the inherent 

problem of non-Internet coverage in a pure random sample of households. Similar to other panel

samples, adjustments are made to sample design weights to reduce bias due to noncoverage of 

nontelephone households, WebTV noncoverage and nonresponse. The result of this survey 

design approach leads to a representative sample of U.S. households that is basically comparable 

to RDD sample selection methods. 

Since use of this new Internet based survey mode for collecting nationally representative data is 

still in the early stages, it is critical to analyze and document the methodologies applied and the 

resulting effects on reported data. To that end, there is an established and ongoing methods 

1

 This paper was prepared for presentation at the November 14-16, 2001 Research Conference of the Federal 

Committee on Statistical Methodology. 
 

research program for evaluating and improving the survey methods and quality of studies 

conducted using the Knowledge Networks panel. This research has been conducted by 

Knowledge Networks, the Research Triangle Institute and supported by academic researchers.2 

Research results thus far suggest that use of the Knowledge Networks sampling methodology is a 

viable approach for conducting representative sample surveys. Internet data collection supports 

in-home data collection, multi-media tools for administering surveys, which can be very helpful 

for studies with sensitive topics and studies requiring video or audio components. And, Internet 

data collection offers very fast turnaround even with multi-media enhancements. 

Following a more detailed description of the sample design for the Knowledge Networks Panel 

in section II and a summary of cooperation rates in section III, survey methods research results to 

date are summarized and presented in section IV. Section V presents sample weighting methods 

applied to the panel and individual surveys and Section VI closes with a summary of plans for 

future research. 

II. Sample Design for a Web-based National Probability Sample Panel:

 A Proposed Solution: 

I n t r o d u c t o r y p a c k a g e R e v e r s e A d d r e s s RDD Sample Match 

File 

Telephone

Recruitment 

Installation and 

s e r v i c e s u p p o r t 

Household is survey ready P r o f i l e i n f o r m a t i o n 

collected 

The solution proposed by Knowledge Networks and implemented now for almost two years for 

establishing a nationally representative probability based panel sample involves a multi-stage 

process. It begins with an Random Digit Dialing (RDD) sample of households, followed by a 

reverse address match, and mailing of an introductory letter and incentive to every household for 

which we are able to obtain an address match. Households (both addressed matched and nonaddressed matched) are then recruited by telephone resulting in a cooperation rate of about 56%. 

Once a household agrees to participate, KN delivers a WebTV unit that essentially transforms 

the television in the household into a monitor for survey administration. All household members 

are recruited and all adults (18 and over) are given a welcome survey to familiarize them with 

use of the WebTV. Then a profile questionnaire is assigned to each household to collect basic 

2 Knowledge Networks and RTI are in an Alliance to competitively bid Federal research projects that involve a webenabled panel. Knowledge Networks works independently with academic researchers on their own investigatorinitiated research projects. For more information on the Alliance, go to the following Internet link: 

www.knowledgenetworks.ganp 
demographic information about the household and its members. Once we have received the 

profile data, the household is considered ready to receive regular surveys. The cost of the unit 

and monthly connection fees are borne by Knowledge Networks. In return, the household 

members agree to complete an average of four surveys a month for the duration of their tenure 

on the panel. The weekly surveys are usually 5-15 minutes long, although we have fielded 

longer ones, particularly for our government-sponsored surveys, in which case we have either 

offered incentives or have excused the member from doing a survey the following week. 

Teenagers, aged 13-17, are profiled and surveyed only with parental consent, which can be 

unconditional or conditional, with conditional consent meaning that the parent needs to see and 

review the questionnaire that is sent to the teenager. We do not send surveys directly to children 

under 13 years of age but we have requested parents to conduct a survey with their children. 

We have not yet determined the optimal tenure on the panel for members. This is under 

discussion and is being researched although early indications suggest that a period of about 2-3 

years strikes a good balance between risk of fatigue and the need to recover the initial 

investment. 

As stated above, the selection of the panel starts with an RDD sample of households. In fact, the 

entire recruitment process is based on known telephone survey methodology. [Lepkowski, 1988] 

The novel challenge posed by this new mode is the coverage of households by WebTV, an 

Internet Service Provider (ISP), which raises issues of telephone connectivity between individual 

households and the ISP. 

To sign on to this service without incurring long distance telephone toll costs, each household 

must dial a local Point of Presence (POP) of which WebTV has about 3,000 scattered throughout 

the country. Unfortunately, coverage is not universal and a 6% of households do not have access 

to one of WebTV's POPs. Most of these households are located in hard-to-reach rural areas. 

Prior to July 2001, only a few households were recruited out of the Web TV covered areas due to 

the cost of going through other providers. This led to bias in the panel recruitment due to the 

undercoverage of rural areas. 

However, as of July 2001, a subsample of households outside the WebTV universe is being 

included in the Knowledge Networks panel using other ISPs for the Internet connection. It is a 

subsample relative to the sampling rate for households in areas covered by Web-TV service. The 

size of the sample for the non-Web TV covered areas can grow if the demand for more reliable 

data from this segment of the population increases. 

Drawing samples from the panel for individual surveys is an important part of the process. 

There were two key objectives in designing the sampling system: 

• Only one survey can be assigned per member per week 

• Selection of members will be random within sampling criteria 

To ensure appropriate representation, panel post-stratification weights are updated after each 

sample selection such that the weighted panel distributions match benchmarks as determined 

from the most recent monthly CPS. We use a 42-stratum cell weighting approach where the 
strata are defined using the following variables: age, gender, region, race, ethnicity, and 

education. 

Samples are drawn consecutively throughout the week with probabilities proportional to the 

panel weights using systematic sampling applied to the sorted panel members. The distributions 

for the panel samples are consistent with the national population distributions for the abovementioned variables. 

After every sample selection, the panel weights for remaining members are adjusted in 

preparation for the next sample. 

III. Cooperation/Response Rates 

Ensuring high response and cooperation rates is one of the most challenging aspects of this 

methodology. Clearly in the industry, response rates through the telephone mode of data 

collection are more and more difficult to maintain. As mentioned earlier, the overall historical 

cooperation rate at the recruitment stage is approximately 50%. There remain at least three more 

stages before the member becomes fully profiled, active, and ready for weekly surveys. And at 

each stage there is some attrition. Thus, the final overall cumulative response rate ranges from 

25% to 50% depending on the level of efforts expended for individual projects. We consider this 

to be one of our major challenges and are carrying out extensive research to maximize the 

cooperation rate at each stage. 

The current and cumulative response rates for fielding an Internet survey from the Knowledge 

Networks panel are found in Table 1. 

We have given considerable thought to strategies for improving these rates. These strategies 

include in-person recruitment, relaxed requirement to complete only one survey per month, 

additional incentives for all potential recruits or incentives to convert reluctant or soft refusals, 

followup of a sample of reluctant or soft refusals with incentives, and closer monitoring of panel 

health. Several of these are being tested such as additional phone contacts with members at 

different stages of participation, an incentive program to maintain panel health, and testing of 

alternative services for advance letter mailing. 

The survey completion rates (i.e., specific to the study) are highly correlated with the length of 

the interview, the interest level of respondents in the survey, the use of advance letters, the use of 

incentives, the complexity of the interview, etc. A straightforward 10-minute survey of high 

interest levels with an advance letter and some followup for nonresponse and/or incentives can 

achieve an Internet survey completion rate as high as 90%. For example, a recent 10 minute 

survey to estimate purchase intent of a particular phone card resulted in a 94% completion rate 

after an 18 day field period. E-mail reminders were sent to nonrespondents. 

We have initiated studies to evaluate the effect of nonresponse on panel and survey estimates. 

We also were a subcontractor to the Research Triangle Institute to evaluate the effects of 

nonresponse on key outcome variables. Selected results from these efforts are described in the 

section below. 
IV. Panel Quality 

Now that over 4,000 surveys have been conducted using the Knowledge Networks Panel, we 

have substantial data to begin the on-going process of assessing the quality of the panel and 

individual survey data. No survey is without some level of sampling and nonsampling error – the 

goal in any study should be to minimize survey error, quantify remaining errors to the extent 

possible and apply survey procedures and methods to mitigate its effect on the outcome 

variables. Below, we discuss several areas where analyses have been initiated to investigate 

sampling and nonsampling error in the conduct of surveys from the Knowledge Networks Panel. 

We will present summary results in the following areas: 

• Coverage Error 

• Benchmarking Analyses 

• Existence of Panel Bias 

• Nonresponse Bias 

• Weighting and Sampling Error 

A. Coverage Error 

There are two key sources of coverage error that can affect the representative nature of the 

Knowledge Networks panel sample: Error arising from noncoverage of nontelephone 

households and error arising from noncoverage of non-WebTV areas. We discuss the magnitude 

of each of these and our planned approaches to reduce biases stemming from them. 

Noncoverage of Nontelephone Households 

According to the June 2001 CPS, approximately 5% of households in the U.S. are without a 

phone at the time of interview. Phone coverage differs by household income (80% for 

households with income less than $5,000 and 92% for households with income $15-20K), state, 

metro status, race, ethnicity, etc. Currently, a post-stratification weighting adjustment is made to 

the Knowledge Networks panel to ensure total population estimates from the RDD based sample 

are consistent with U.S. population estimates for the phone and non-phone population. The 

adjustment is made at the state level, and then further refined through post-stratification (raking) 

using gender, age, race/ethnicity and education level. The complete post-stratification scheme is 

implemented for two purposes: (1) Reduce the bias in the panel due to coverage and 

nonresponse error and (2) Reduce the variance for statistics highly correlated with the

demographic benchmarks. 

We are investigating whether a separate weighting adjustment specifically to account for 

nontelephone coverage error would be more accurate for potential bias at lower levels in the 

sample. Specifically, we are investigating the methodology proposed by Frankel (2000) for 

reducing nontelephone bias in RDD surveys that uses survey data collected on interruption in 

telephone service to identify respondents more like non-telephone households for weighting 

purposes. Our investigation supports further evaluation of the approach for the Knowledge 

Networks panel. 

Table 2 presents comparative estimates of household characteristics of panel members who were 

asked whether they had an interruption in telephone service for 1 week or more in the past year. 

It is quite clear from the table that the group with interruption in telephone service and the group 
without are different. Estimates of the number of children under 18, household size, household 

type, number of computers in the household, and access to the Internet are all statistically 

different between the groups with and without telephone interruption. Approximately 3.6% of 

recruited households reported being without telephone service for 1 week or longer in the 

previous 12 months. 

Frankel et.al. showed that estimates from the group with an interruption in telephone are much 

more like those that had no phone in the population. The estimates examined were: 

• Did not get medical care for cost reasons in past 12 months 

• Looking for work last week 

• Race of person is Black 

• Age of person is less than 5 years 

These are certainly important characteristics for many of the surveys conducted using the 

Knowledge Networks Panel. The next steps are to look at mean square error for selected 

estimates if the weighting approach is administered at the state/msa level. Since we won’t know 

the true bias, we will do some sensitivity analysis on a range for the bias. We will also 

investigate whether other variables such as having access to a computer at home and/or access to 

the Internet or household type might well be good predictors as well. The advantage to using 

these characteristics is that is does not require asking respondents about interruptions in the 

telephone service, which can be interpreted negatively by respondents. 

Noncoverage of NonWebTV Service Areas 

As discussed earlier, the Knowledge Networks panel suffered from noncoverage of households

due to the fact that the Internet Service Provider – WebTV – does not cover all areas of the U.S. 

Using available information at the exchange level, demographic estimates for the phone numbers 

in and outside of WebTV Service Areas were calculated. As expected, non-WebTV covered 

areas are much more rural (78% versus 22% for covered areas), a little more elderly and with a 

lower income distribution. Estimates are highly variable due to significant levels of missing data 

at the exchange level. However, the direction of the coverage error is consistent with other 

derived analyses. 

The good news is that we have begun recruiting households in these non-covered areas using 

different Internet providers so we will definitely be reducing panel bias associated with this 

noncoverage. In addition, we will have the ability to better measure the effects of excluding 

non-WebTV covered areas by using data collected on newly recruited members in non-WebTV 

covered areas. 

B. Benchmarking Analyses 

One method for analyzing the quality and representativeness of a study or sample is to compare a 

variety of estimates from the study or sample to known and/or official benchmark estimates. 

This section presents results of several comparisons of data from the Knowledge Networks panel 

to several other sources including the Current Population Survey, the Behavior Risk Factor 

Surveillance Survey (BRFSS) 2000, and the Ohio State University RDD Survey on Public 

Opinion and Voting Intentions for the 2000 U.S. Presidential Elections. Comparisons across 

demographic estimates and topical estimates are covered. 
Table 3 presents a comparison between the unweighted KN panel and the Current Population 

Survey (CPS) for selected demographics as of June 2001. Column 1 contains estimates for 

active, profiled members after post-stratification to CPS benchmarks using selected 

characteristics. Column 2 contains estimates from the panel using the entire recruited panel 

sample with the associated weight from the initial selection probabilities. Column 3 presents 

June 2001 CPS estimates and the last two columns present calculated differences of the two sets 

of Knowledge Networks estimates from the CPS benchmarks. 

As you can see from table 3, column 2, the Knowledge Networks panel under represents the 

elderly, is skewed towards the upper end of the socioeconomic scale, and under represents the 

African American minority. The differences in the race estimates is an primarily a difference in 

the way Census asks race as compared to Knowledge Networks, with Knowledge Networks 

offering “Other” as a race category. The panel also slightly underrepresents the Hispanic 

population. Column 1 presents estimates after final weighting is applied to the Knowledge 

Networks active and profiled members. Due to the large sample sizes associated with both the 

Knowledge Networks panel and the Current Population Survey, small differences are statistically 

detectable as asterisked in columns 5 and 6 of table 3. In general, none of the average deviations 

are huge, and sample representative ness is never dramatically poor. Approximately 74% of the 

estimates moved closer to the CPS benchmarks as a result of final weighting procedures. 

As mentioned earlier, an anti-rural bias existed in the panel because of the WebTV coverage 

issue. However, with the sampling underway to recruit households in non-WebTV covered 

areas, this coverage error will be greatly mitigated. 

Benchmarking of results from several surveys conducted using the Knowledge Networks panel

has also been conducted. Table 4 presents comparative results from a Study on Smoking to 

comparable estimates from the BRFSS. [Dennis, 2001] Table 4 presents estimates on current 

smoking behavior for Veterans aged 22-80. According to the Knowledge Networks sample, 

26% of Veterans between the ages of 22 and 80 currently smoke. The BRFSS survey of year 

2000 shows that 24% of Veterans between the ages of 22 and 80 currently smoke. 

Table 5 displays the demographic characteristics of male veterans in the U.S. by two data 

sources: the Knowledge Networks Panel and BRFSS Survey 2000. On age, race, and 

education, the Knowledge Networks data are consistent with those of BRFSS. On income there 

are some differences as noted before with the Knowledge Networks income distribution being 

somewhat skewed to the higher end. Other estimates from the two sources were compared with 

the small number of differences identified primarily attributable to different question concepts 

and question wording. 

Table 6 presents a demographic comparison between Knowledge Networks Panel data for the 

population with Internet access to the August 2000 CPS data from the Computer Usage

Supplement. Generally, the results are very comparable. Estimates from the Knowledge 

Networks panel on education level, gender, marital status, employment and most importantly 

broadband use are consistent with the CPS, even though the large sample sizes generally detect 

statistically significant differences between the two sources. There are some substantive 

differences between the sources on presence of children, income and race/ethnicity. 
In an independent study conducted by Jon Krosnick and LinChiat Chang at Ohio State 

University [Krosnick, 2001], Knowledge Networks survey results were compared to results from 

both a random digit dial study conducted by the Ohio State University Center for Survey 

Research and the Harris Interactive Internet opt-in panel. The same questionnaire to gauge 

public opinion and voting intentions for the 2000 U.S. Presidential Election was administered 

under each of the survey modes and standard data collection methods. Krosnick and Chang 

compared: 

Demographic characteristics 

Distributions of survey responses 

Reliability of individual questions 

Survey satisficing 

Predictive validity 

Krosnick and Chang concluded that Internet based data collection represents a viable approach to 

conducting Random Digit Dialing surveys. And the Knowledge Networks methodology resulted 

in a more representative sample than the opt-in panel sample utilized by Harris Interactive. 

Results also suggest that Internet data collection improves the accuracy of the reports 

respondents provide over accuracy obtained through telephone interviews. 

In summary, benchmark comparisons of Knowledge Networks estimates to the CPS, BRFSS 

2000, the U.S. Census and other sources show reasonable consistency considering what we know 

about potential coverage and nonresponse levels. 

C. Preliminary Research on the Existence of Panel Bias 

Research panels may be susceptible to two types of panel effects. The first type is the possibility 

of conditioning research subjects in a panel sample, turning them into “professional respondents” 

whose attitudes and behaviors are changed by panel participation. The second type of effect that 

panels are potentially vulnerable to is selection bias, which can make successive samples less 

representative. Preliminary research, using data from a variety of different studies, has not 

detected serious levels of panel effects. The discussion below present results that illustrate these 

findings. More detail can be found in Dennis (2001). 

Attitudes toward new products. The question arises whether more experienced panelists have 

the same orientation toward new products and new technology as less experienced panelists. In a 

survey of more than 6,000 panelists about hybrid electric cars, responses are not related to panel 

tenure. As shown in table 7, the future of hybrid electric cars appears equally bleak across the 

tenure groups. There are also no significant differences across responses when grouped by levels 

of survey participation. 

Personal finance 

The area of personal finance relates directly to panelists’ demographics (wealth), orientation 

toward risk (ownership of individual stocks), and inclination to use computers and the Internet to 

increase personal productivity (online banking). A personal finance survey of about 6,500 

panelists in January 2001 showed that less and more experienced panelists have similar 

behaviors – table 8. While not a statistically significant finding, the most experienced panelists 

show indications of using the Internet more for investing than less experienced panelists. 
There are also no significant differences across responses when grouped by levels of survey 

participation. For instance, panelists who had completed fewer than 15 surveys and those who 

had completed more than 35 surveys use online banking at the same rate (12%). 

Sensitive questions 

Panel members with more tenure might be expected to be more comfortable with the survey 

environment and be less affected by the impulse to give socially desirable answers. Although the 

surveys are taken in a self-administered setting, some newer panelists might feel an urge to be 

more positive and conforming. However, the data from a survey of approximately 6,000 

panelists provides limited support for this hypothesis. 

When asked about their comfort level with a shop owner with AIDS, newer panel members were 

more likely to provide the socially pleasing response of comfortable – see table 9. We should not 

read too much into this finding because most of these small-scale effects evident in the other 

questions disappeared or were diminished when controlling for panelists’ demographic 

characteristics within each tenure group. Overall, the effects are small and are almost certainly 

less serious than the social desirability effects well documented in telephone and face-to-face 

interviewing. 

Survey participation is significantly related to attitudes in only three of the 30 pair-wise 

comparisons, a not surprising result given the large number of significance tests performed. 

An earlier study to evaluate panel effects found similar results. [Clinton, 2000] Five groups of 

respondents, each with different panel tenure, were assigned an identical instrument dealing with 

politics, views of the economy, media consumption, and Internet usage. Very few significant 

differences were found between the responses of the five tenure groups. The behavioral 

differences that were detected appear to reflect an increase in news consumption and Internet 

usage during the early stage of panel recruitment. However, behavior appears to return to 

normal afterwards. 

Results of Selection Bias 

While the unreachable ideal is to observe no panel attrition, the second-best goal is for attrition 

to be evenly distributed across key demographic dimensions and to replace dropped-off panelists 

with demographically representative individuals and households. In this circumstance, selection 

bias is minimized for any follow-up studies using a panel. 

Table 10 presents selected panelists’ demographic characteristics for groups defined by length of 

panel tenure. If panel attrition is evenly distributed across demographic groups, then the statistics 

should be constant across the table. For instance, the proportion of the currently active panel is 

between 50% and 51% female across the tenure groups, showing that the participation rate for 

males and females is independent of length of panel tenure. 

One way to gauge the relevance of the table is to conceptualize that random survey samples 

drawn from any of the tenure groups will resemble each other on key demographic dimensions. 

This is an indication that the effects of panel attrition do not meaningfully increase selection bias. 

The slight fluctuations in the exhibit are not statistically significant (p < 0.05). 
D. Nonresponse Bias 

As described in section III above, nonresponse or cooperation bias can creep in at several 

different stages, from RDD recruitment, WebTV installation, profiling of members, and 

completion of project-specific surveys. Different levels and detail for data are available on 

nonrespondents at the different stages. For example, in evaluating the differences in 

nonresponders and responders to completion of the household and member profile surveys, we 

have information about the household from the recruiting interview as well as geographic 

information associated with the household phone number. But for evaluation of responders and 

nonresponders from RDD recruitiment, we only have the aggregate demographic and geographic 

information associated with sampling the telephone number. 

Currently, a weighting adjustment to reduce nonresponse bias from panel recruitment through 

profiling implemented with the use of post-stratification to CPS population totals prior to sample 

selection of weekly surveys. Then, after a survey is fielded, a separate nonresponse adjustment 

to reduce nonresponse bias for individual surveys is applied. The variables used include age, 

race, sex, ethnicity, income, education, computer usage, access to the Internet, and metro status. 

The number of variables and cross-classification structure are dependent on the survey needs and 

sample size. The construction of survey specific nonresponse adjustments has been implemented 

on a very ad-hoc basis. There is a definite need for more consistency and a better understanding 

of the effect on the MSE of estimates generated. 

Our goal is to identify the best combination of weighting adjustments to account for nonresponse 

bias from all stages of panel activation. We need to determine whether separate adjustments for 

nonresponse are needed at each stage or whether more global adjustments suffice. There is a 

trade-off in evaluating the MSE of key estimates from making multiple weighting adjustments as 

well as keeping the weighting methodology as simple as possible since time required for weekly 

preparation of the panel sample and many profile components has to be minimized. 

The steps we are taking in investigating enhanced nonresponse weighing include: 

1. Evaluate differences in nonresponders and responders at each stage of panel construction 

and survey implementation. 

2. Conduct nonresponse studies to better measure differences and evaluate the effect on key 

outcome variables. 

3. Identify and test adjustments at each stage as well as combinations of adjustments to 

minimize the MSE for key outcome variables. 

As part of step 1, we have been able to compare selected characteristics of responders and 

nonresponders at the point where recruited households are asked to complete the household 

profile questionnaire. The recruiting interview collects information about household decision 

maker, use of a computer, access to the Internet, and household composition. Table 11 presents 

5 variables from the recruiting interview by whether or not the household completed the

household core profile survey. Table 11 shows statistical differences between responders and 

nonresponders about whether a computer exists in the home (78.7% yes for nonresponders, 

69.6% for responders) and whether the computer is connected to the Internet (86.9% and 78.7% 

respectively for nonresponders and responders). Also, there is a slight skewness for households 

with a smaller number of members completing the household core profile versus not completing 
it. These variables can be considered for use in a nonresponse adjustment for profiled 

households to better adjust for non-profiled households. 

We are continuing to look at differences in nonresponders and responders at the other stages of 

panel recruitment and fielding of surveys. 

The Research Triangle Institute sponsored a formal study of the effects of nonresponse on key 

outcome variables in the recent Survey on Health and Aging. [Wiebe, 2001] The methodology 

included re-sampling of nonrespondents, fielding the core survey to the nonrespondents, and 

weighting the nonrespondent completes using the resampling design. Implemented in 2000, 

telephone interviews were conducted with samples from the following nonresponse groups:

• RDD Panel Recruitment Refusers (n=71 completes) 

• RDD Acceptors: Agreed to participate in the Web-enabled panel but had not yet hooked 

up Web TV (n=129 completes) 

• Telephone prompting encouraged Panel Nonrespondents to complete the survey on the 

Internet Appliance (n=238 completes) 

Data collection from the resample of nonrespondents was conducted using both telephone and 

web assisted methods. Where possible, nonrespondents were contacted by phone and asked to 

complete the Survey of Health and Aging (SHA) on the web device. If this was not possible, 

they were asked to complete the survey over the phone. 

The weighted response rate for the study increased from 25% to 43% as a result of nonresponse 

followup by phone. Different participation groups appear to report different answers in the 

survey with no clear pattern in the responses. 

The primary question that motivated the study was whether the followup would change the key 

study estimates. The conclusion made by the researchers was that the nonresponse follow-up did 

not make any significant changes in the overall representativeness of the sample. The 

representativeness of the sample was actually achieved through the standard procedures used by 

Knowledge Networks to select the sample from the full panel. Inclusion of additional 

recruitment groups did not affect the estimates. When all components of the nonresponse 

followup are included with the intial WebTV based estimate, no significant changes in the 

outcome estimates resulted. 

Table 12 presents the estimates for the question on coping with serious injury split out by 

response and nonresponse stages as well as estimates using data combining the nonresponse 

sample results with the original Survey of Health and Aging Results. We can see by examining 

the cumulative results that the additional weighted responses from NRFUS had little impact on 

the overall prevalence estimates 

The graph below presents one set of results that illustrate the findings. Respondents were asked 

about how concerned they were with having adequate health insurance, coping with serious 

injury or illness, keeping a job, job hunting, or changing careers, and paying for children’s 
college education. Figure 1 shows the responses provided on these questions for the four types 

of respondents: 

• Those who completed during the initial study 

• Those who refused the RDD recruitment but completed the nonresponse follow-up 

survey (NRFUS) 

• Those who failed to install the WebTV device but completed the NRFUS 

• Those who refused the initial survey but completed the NRFUS. 

The results indicate that the persons who refused the RDD recruitment and those who failed to 

install the device provided significantly different responses on topical questions than did those 

who cooperated with the initial survey. This suggests that the respondents and nonrespondents 

may be different and should be carefully evaluated in future studies.

 Figure 1: Effect of Nonresponse on Substantive Estimates? 

Major Concern - Weighted % - 95% CI 

20% 

30% 

40% 

50% 

60% 

70% 

80% 

90% 

Health Ins. Serious Inj. Keep/Find 

Job 

Pay College 

Initial RDD Install SHA 

V. Weighting and Sampling Errors 

Whereas the sample design is an equal probability design that is self-weighting, in fact there are 

several known deviations from this guiding principle. We address these sources of survey error 

globally through the poststratification weights which we describe below. 

Sample Design Weights 

The five sources of deviation from an equal probability design are: 

1. Half-sampling of telephone numbers for which we could not find an address, 

2. RDD sampling rates proportional to the number of phone lines in the household, 

3. Minor oversampling of Chicago and Los Angeles due to early pilot surveys in those 

two cities, 

4. Short-term double-sampling the four largest states (CA, NY, FL, and TX) and central

region states, and 

5. Selection of one adult per household. 
A few words about each feature: 

1. Once the telephone numbers have been purged and screened, we address match as 

many of these numbers as possible. The success rate so far has been in the 50-60% 

range. The telephone numbers with addresses are sent a letter. The remaining, 

unmatched numbers are half-sampled in order to reduce costs. Based on previous 

research we suspect that the reduced field costs resulting from this allocation strategy 

will more than offset increases in the design effect due to the increased variance 

among the weights. We are currently quantifying these balancing features. 

2. As part of the field data collection operation, we collect information on the number of 

separate phone lines in the selected households. We correspondingly downweight 

households with multiple phone lines. 

3. Two pilot surveys carried out in Chicago and Los Angeles increased the relative size 

of the sample from these two cities. The impact of this feature is disappearing as the 

panel grows, but we still include it as part of our correction process. 

4. Since we anticipated additional surveying in the four largest states, we doublesampled these states during January-October 2000. Similarly, the Central region 

states were oversampled for a brief period. 

5. Finally, for most of our surveys, we select panel members across the board, regardless 

of household affiliation. For some surveys, however, we select members in two 

stages: households in the first stage and one adult per household in the second stage. 

We correct for this feature by multiplying the probabilities of selection by 1/ai where 

ai represents the number of adults (18 and over) in the household. 

The final sample weights are scaled to sum to the final sample size representing the total number 

of completed surveys. 

Once the samples are drawn, assigned, and the data returned, we currently subject the final 

respondent data to a poststratification process to adjust for variable nonresponse and 

noncoverage. Once the individual surveys are completed in the field, a nonresponse adjustment 

to reduce the effects of differential nonresponse for the individual survey are applied as 

discussed earlier. Depending on the sample size of the survey, noninterview cells are collapsed. 

Post-stratification is then applied to the sampling weights (after noninterview adjustment) to 

bring the survey estimates in line with CPS benchmarks by age, race, sex, ethnicity, census 

region, and education. 

Currently, design effects are almost always less than 1.5 and the average design effect for most 

study estimates is 1.3. The effect of differential sampling for non-Web TV covered areas will be 

assessed on the sampling error for key characteristics. 
VI. Future Research 

The innovative Internet survey methodology launched by Knowledge Networks has been 

underway for almost two years during which time we have learned a great deal about this new 

mode of survey research, its strengths as well as its weaknesses. Survey researchers should 

consider this mode of data collection as one more tool in the kit for collecting data national, 

subnational and subpopulation data about the nation. Its key advantages include a rich panel, 

quick-turnaround capability, video and audio capabilities, and a panel selected and maintained 

using probability methods. It also provides a rich base for identifying and surveying low 

incidence populations and supports longitudinal analyses. Finally, the panel is an excellent 

resource for basic methods research on web-enabled panels and classical problems in general 

survey research. 

Research thus far has indicated that survey results for a wide variety of estimates calculated from 

the Knowledge Networks panel are not critically affected by nonsampling error such as noncoverage, nonresponse, and panel bias. This statement is made based on current needs and uses 

of the panel data. Even so, we will continue to dedicate resources and methods to reduce current 

levels of nonsampling error and measure potential effects on other survey results. Also, some 

future studies may require more stringent levels of reliability and preciseness. The goal is to 

improve data quality and continue to implement sound statistical methods that meet customer 

requirements. 

The methodological issues presented in this paper will continue to be investigated. These 

include teasing out panel effects, mode effects, nonresponse and noncoverage bias, and response 

bias. We also shall address instrument design issues on the Internet raised by Mick Couper 

[Couper, 2000] and others. More topical benchmarking is needed as well. Knowledge Networks 

and the Research Triangle Institute will jointly conduct basic research on the panel, 

experimenting with the use of incentives, assessing panel bias as the panel ages, and expanding 

nonresponse studies. As new surveys and new research related to this new survey mode for large 

scale panel data collection continues, we will continue to clarify the problems and pose potential 

solutions. 

References 

Dennis, Michael J., “A Study of Panel Effects”, May 2001. 

Dennis, Michael J. “Knowledge Networks Profile Data Compared to BRFSS 2000: Smoking 

Prevalence and Demographic Characteristics of U.S. Veterans”, Knowledge Networks Report, 

May 22, 2001. 

Clinton, Joshua, “InterSurvey Panel Effects Study – June 2000”, Knowledge Networks Research 

Note, June 2000. 

Couper, Mick P., “Web Surveys: a Review of Issues and Approaches”, Public Opinion 

Quarterly, 64:464-494, 2000. 
Elizabeth F. Wiebe, Joe Eyerman, and John Loft, “Evaluating Nonresponse in a Web-Enabled 

Survey on Health and Aging”, Presented at the 2001 Meeting of the American Association for 

Public Opinion Research, Montreal, Quebec May 17 - 20, 2001. 

Frankel, Martin R. and Michael P. Battaglia, David C. Hoaglin, Robert A. Wright, and Philip J. 

Smith, “Reducing Nontelephone Bias in RDD Surveys”, Presented at the Meeting of the 

American Assocication for Public Opinion Research, 2000. 

Krosnick, Jon A. and LinChiat Chang, “A Comparison of the Random Digit Dialing Telephone 

Survey Methodology with Internet Survey Methodology as Implemented by Knowledge 

networks and Harris Interactive”, Presented at the 2001 Meeting of the American Association for 

Public Opinion Research, Montreal, Quebec May 17-20, 2001. 

Lepkowski, J. “Telephone Sampling Methods in the United States”, Chapter 5 in Telephone 

Survey Methodology, Wiley and Sons, 1988. 

Newburger, Eric. Home Computers and Internet Use in the United States: August 2000, Current 

Population Reports, U.S. Census Bureau, P-23-207, September, 2001. 

Table 1: Knowledge Networks Cooperation/Response Rates 

Component of Overall 

Response Rate

 Cumulative Response Rate 

Panel Recruitment Cooperation 56% 56% 

WebTV Installation 80% 45% 

First-survey Profile Completion 88% 39% 

Internet Survey Response 85% 34% 

* Varies according to design choices between 75% and 90%. 

Table 2. Characteristics by Interruption in Telephone Service 

Characteristic Interruption: Yes Interruption: No 

# of Children <18 .79* .61 

# of Computers in the 

household 

1.76* 2.03 

HH Type 

detached 

– Single, 45%* 67% 

Tenure - Owner 45%* 73% 

Have Internet 66%* 75% 

* Indicates statistical significance for p<.05 (2-sided) 
Table 3. Knowledge Networks Panel and Current Population Survey (CPS) 

Demographics: June 2001 

Characteristic 

Knowledge 

Networks 

Active Panel 

(Note 1) 

Knowledge 

Networks 

Entire Panel 

(Note 2) 

Adult U.S. 

Population 

CPS 

Difference 

(Active 

Panel and 

U.S. Pop.) 

Difference 

(Entire 

Panel and 

U.S. Pop.) 

Gender 

Male 47.3% 49.4% 47.9% -0.6% 1.5% 

Female 52.7% 50.6% 52.1% 0.6% -1.5% 

Age 

18-24 11.0% 12.5% 13.2% -2.2%* -0.7%* 

25-34 20.0% 21.8% 18.3% 1.7%* 3.5%* 

35-44 22.1% 25.9% 21.9% 0.2% 4.0%* 

45-54 20.3% 20.9% 18.7% 1.6%* 2.2%* 

55-64 13.1% 10.3% 11.8% 1.3%* -1.5%* 

65 or over 13.4% 8.6% 16.1% -2.7%* -7.5%* 

Race 

White 79.4% 79.3% 83.2% -3.8%* -3.9%* 

Black/African-American 12.0% 10.5% 11.9% 0.1% -1.4%* 

American Indian/Alaska Native 1.7% 2.0% 0.9% 0.8%* 1.1%* 

Asian/Pacific Islander 1.9% 3.0% 4.0% -2.1%* -1.0%* 

Other 5.0% 5.2% n/a n/a n/a 

Hispanic Ethnicity 

Hispanic 10.9% 6.4% 10.7% 0.2% -4.3%* 

Non-Hispanic 89.1% 93.5% 89.3% -0.2% 4.2% 

Employment Status 

In the Labor Force 72.1% 76.8% 66.1% 6.0%* 10.7%*

 Working full-time 58.7% 62.8% 56.2% 2.5%* 6.6%*

 Working part-time 13.4% 14.0% 9.9% 3.5%* 4.1%* 

Not in the Labor Force 29.9% 23.2% 33.9% -4.0%* -10.7%* 

Marital Status 

Married 61.3% 61.6% 57.5% 3.8%* 4.1%* 

Not married 38.7% 38.4% 42.5% -3.8% -4.1% 

Level of Education 

Less than High School Diploma 9.0% 7.4% 17.1% -8.1%* -9.7%* 

High School Diploma or 

Equiv./Some College 59.6% 55.7% 51.6% -5.0%* -4.1%* 

Associate Degree 5.5% 7.0% 7.6% -2.1%* -0.6%* 

Bachelor’s Degree or Beyond 25.8% 29.8% 23.8% 2.0%* 6.0%* 

Household Income 

Under $10,000 4.2% 3.7% 7.3% -3.1%* -3.6%* 

$10,000-$24,999 15.4% 12.5% 18.4% -3.0%* -5.9%* 

$25,000-$49,999 35.7% 33.0% 29.7% 6.0%* 3.3%* 

$50,000-$74,999 24.4% 26.0% 20.0% 4.4%* 6.0%* 

$75,000 or more 20.3% 24.8% 24.6% -4.3%* 0.2%* 
Census Region 

Northeast 19.0% 18.3% 19.1% -0.1% -0.8%* 

Midwest 22.5% 23.7% 22.9% -0.4% 0.8%* 

South 36.0% 35.5% 35.6% 0.4% -0.1% 

West 22.5% 22.5% 22.4% 0.1% 0.1% 

Note 1: Estimates calculated using the post-stratified weight for active, profiled members. 

Note 2: Estimates calculated using the base sampling weight for the entire recruited Knowledge Networks 

panel. 

* Indicates statistical significance for p<.05 (2-sided) 

Table 4. Current Smoking Prevalence Rates on Knowledge Networks 

Profile Data and BRFSS 2000: Males Age 22-80 Years 

Smoking Status 

Veteran Status 

Yes No 

KN BRFSS KN BRFSS 

Currently Smoke 26% 24% 28% 24% 

No, Do Not Smoke 74% 76% 72% 76% 

Total 100% 100% 100% 100% 

*p-value < .05 (two-sided) 

Table 5: Demographic Characteristics of Veterans: 

Knowledge Networks and BRFSS 2000 

 *p-value < .05 (two-sided) 

Characteristic BRFSS Knowledge Networks 

Age

 18-34 9% 8%

 35-54 33% 35%

 55-74 44% 44%

 75+ 14% 13% 

Race

 White 88% 87%

 Black 8% 9%

 Asian/Pacific Islander 1% 1%

 American Indian, Alaska 1% 2%

 Other 2% 2% 

Education

 Less than high school 9% 8%

 High school graduate 31% 34%

 Some college 30% 32%

 College graduate or 30% 26% 

more 

Household Income

 Less than $25,000 24% 15%

 $25,000 to $34,999 16% 12%

 $35,000 to $49,999 22% 24%

 $50,000 to $74,999 19% 28%

 $75,000 or more 20% 21% 
Table 6. Demographics of the Population with Internet Access† 

Characteristics 

US Internet Population US Internet Population 

 Knowledge Networks Panel 

 

Age 

August 2000: CPS 

Age 

13-17 18-45 Total 13-17 18-45 Total 

Presence of kids < 18 in HH 

Yes 

No 

Gender 

Male 

Female 

Marital Status 

Married 

Widowed 

Divorced 

Separated 

Never Married 

Education 

HS & Less than HS 

Some College 

Bachelor or Higher 

Employment Status 

Employed 

Unemployed 

Retired, Not in labor force 

HH Income (4 category) 

<$10,000 

$10-49,999 

$50-74,999 

$75,000+ 

Ethnicity (Hispanic vs. Not) 

Yes 

No 

Race 

White 

Black/African-American 

American Indian or Alaska Native 

Asian/Pacific Islander 

Other 

Region (4 Census) 

Northeast 

Midwest 

NA 

NA 

50.21% 

49.79% 

0.13% 

0.03% 

0.38% 

0.24% 

99.22% 

99.60% 

0.25% 

0.15% 

NA 

NA 

NA 

1.67% 

34.11% 

25.46% 

38.77% 

6.90% 

93.10% 

86.51% 

8.06% 

0.50% 

4.93% 

19.55% 

24.79% 

49% 

51% 

48.60% 

51.40% 

59.06% 

0.43% 

6.44% 

1.38% 

32.69% 

30.37% 

34.52% 

35.11% 

82.32% 

17.51% 

0.17% 

2.55% 

35.70% 

25.80% 

35.86% 

6.94% 

93.06% 

86.07% 

7.55% 

0.58% 

5.80% 

19.89% 

23.06% 

NA 

NA 

49.38% 

50.62% 

NA 

NA 

NA 

NA 

NA 

99.45% 

0.55% 

NA 

NA 

NA 

NA 

2.12% 

38.64% 

31.32% 

27.92% 

8.74% 

91.26% 

82.51% 

13.17% 

2.62% 

1.71% 

16.66% 

29.12% 

55.97% 

44.03% 

49.68%

50.32% 

58.70% 

0.34% 

7.01% 

1.63% 

32.32% 

33.54% 

34.18% 

32.28% 

84.22% 

14.49% 

1.29% 

2.84% 

41.87% 

28.84% 

26.45% 

13.06% 

86.94% 

83.85% 

11.76% 

1.68% 

2.71% 

16.19% 

30.15% 
 

 

 

 

 

 

 

South 

West 

Broadband Access 

Teens 13-17 

31.82% 

23.84% 

10% 

90% 

33.14% 32.30% 

23.91% 21.93% 

5.51% 

94.49% 

12% 

88% 

11% 

32.85% 

20.81% 

11.28% 

88.72% 

8.89% 

With broadband 

Without broadband 

Young adults 18-25 

With broadband 

Without broadband 

Adults 26-45 

With broadband 

Without broadband 89% 91.11% 

Total broadband age13-45 11% 9.19% 

Total not broadband age13-45 89% 90.81% 

† Most, if not all comparisons of KN

 estimates will be statistically different

 from CPS estimates due to large 

sample sizes. 

Table 7. Attitudes toward hybrid electric cars 

Question 

Panel Tenure (months) 

2-3 4-6 7-9 10-12 

(n=721) (n=2,316) (n=1,646) (n=1,117) 

How does a hybrid car compare to a standard 

car on price (% worse)? 

64 66 68 67 

How does a hybrid car compare to a standard 42 39 38 38 

car on maintenance costs 

(% worse)? 

Plan to purchase or lease a new car in next two 

years (5 Yes) 

39 37 38 37 

How likely to consider a hybrid electric car for 

purchase (% likely)? 

10 9 10 9 

*p-value < .05 (two-sided) 

Table 8. Investments and financial services (%) 

Question 

Panel Tenure (months) 

0-6 7–9 10–12 > 12 

(n=1,016) (n=2,245) (n=1,853) (n=1,471) 

Owns $50,000 or more in investment assets 14 15 14 16 

Owns individual stocks 22 19 19 22 

Invests online 4 4 4 6 

Banks using personal computer 13 12 12 11 

Use of online bill payment 7 5 6 5 

*p-value < .05 (two-sided) 
Table 9. Attitudes on sensitive questions 

Question 

Panel Tenure (months) 

<3 4-6 7-9 10-12 

(n=667) (n=2,137) (n=1,515) (n=1,003) 

People with AIDS deserve it (% agree) 15 18 19 21 

How likely to get AIDS from sharing same 

drink glass (% likely)? 25 25 23 24 

How likely to get AIDS from someone 

coughing or sneezing (% likely)? 22 23 20 22 

Plan to purchase or lease a new car in next two 

years (% yes)? 60** 54* 52* 53* 

Is there currently a cure for AIDS (% yes)? 

19 19 18 18 

*p-value < .05 (two-sided) 

Table 10. Panelists’ demographics by length of panel tenure (%) 

Demographics Panel Tenure (months) 

< 7 7–9 10–12 > 12 Total 

Female 50 51 51 51 51 

Age 18 – 34 years 36 34 33 32 33 

Age 35 – 64 years 48 47 46 47 47 

Age 65 and over 16 19 21 21 19 

High school graduate or equiv. 33 30 30 34 32 

BA degree or more 29 29 33 32 31 

Household income less than $40,000 38 34 35 33 35 

Household income $40,000 - $74,999 38 40 39 39 39 

Household income $75,000 or more 24 26 26 28 26 

*p-value < .05 (two-sided) 

Table 11: Comparison of Respondents and Nonrespondents - Recruiting Interview Data 

Question 

Completed the HH Profile Survey? 

No Yes 

Heard about the World Wide Web 

or Internet before being recruited? Yes 77.4% 79.1% 

No 22.6% 20.9% 

Has respondent or anyone else in the household, 

ever used a computer, either at home, school or at work? Yes 91.5% 91.0% 

No 8.5% 9.0% 

Is there a computer in your home? 

Yes 79.8% 71.5% 

No 20.2% 28.5% 

Is your home computer connected to the World Wide 

Web and/or the Internet? Yes 87.0% 78.9% 

No 13.0% 21.1% 

Number of members in the household? 

1 8.3% 9.4% 
2 29.6% 34.3% 

3 21.0% 21.3% 

4 22.6% 20.3% 

5 12.3% 9.5% 

6 4.2% 3.4% 

7 1.2% 0.9% 

8 0.4% 0.4% 

9 0.2% 0.1% 

10 0.1% 0.1% 

11 0.0% 0.0% 

12 0.0% 0.0% 

14 0.0% 0.0% 

Table 12. How concerned are you with coping with serious injury or illness? Recruitment 

Group with Weighted Percentages 

Recruitment Group 

A B C D Cumulative 

Web 

(standard) 

Web with 

Phone 

Prompt 

Phone -

Refused Web 

Panel 

Recruitment 

Phone -

Failed to 

Install 

Web 

Device A + B A+B+C Total 

Not at all a concern 4.1% 7.1% 4.2% 5.3% 4.2% 4.2% 4.2% 

2.00 6.0% 8.4% 2.8% 6.1% 6.1% 6.1% 6.1% 

3.00 16.9% 14.6% 15.5% 16.0% 16.8% 16.8% 16.8% 

4.00 19.7% 22.8% 19.7% 18.4% 19.9% 19.9% 19.8% 

Major concern 52.8% 45.5% 54.9% 54.1% 52.5% 52.5% 52.5% 

Did Not Answer .5% 1.6% 2.8% .6% .6% .6% 

Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%