Monitoring Response to a Multi-Wave Medical Establishment Survey: 

How Different are Responders from Nonresponders? 

Jessica E. Graber 

NORC – MidWest Clinicians’ Network – University of Chicago Partnership1

NORC at the University of Chicago 

graber-jessica@norc.org

Summary 

In an effort to improve chronic disease management and health outcomes among underserved populations, the 

Bureau of Primary Health Care (BPHC) now requires all federally-funded community health centers to implement 

the Health Disparities Collaborative model, a quality improvement program launched by the BPHC in 1998 that 

aims to enhance the treatment of chronic illness through evidence-based practices, clinical decision support systems, 

and patient self-management. NORC, in partnership with the University of Chicago and the MidWest Clinicians’ 

Network, evaluated the impact of the Collaborative at more than 150 health centers, collecting data in two waves to 

better assess the long term integration of key Collaborative components. 

During the initial wave of data collection in 2004, self-administered questionnaires were mailed to 1,504 medical 

providers, administrators and other staff at health centers that had participated in the Health Disparities 

Collaborative for at least one year. Nonrespondents were subject to extensive follow-up – receiving additional 

questionnaires, letter of support by BPHC officials, and telephone prompting – producing an overall response rate of 

68.1%. In 2005, health centers were re-contacted and 1,456 respondents were asked to complete a similar, but 

shorter questionnaire. Despite significantly less follow-up effort, the final response rate from the second wave was 

68.7%. 

While the two cross-sectional samples were not identical, over half (53.7%) of Wave II respondents were also 

surveyed in the Wave I effort, with more than half of those eligible completing surveys at both points in time. This 

overlap in sample affords us the opportunity to monitor survey response over time. In this paper we explore 

variation in individual and health center-level characteristics of responders by survey participation levels. 

Introduction 

The Health Resources and Services Administration’s Bureau of Primary Health Care (BPHC) funds and oversees over 1,000 

community health centers that provide care for approximately 15 million medically underserved people at 5,000 sites 

(National Association of Community Health Centers 2006). The mission of the community health centers is to serve as a 

safety net for individuals who face major barriers to accessing quality health care, including the medically isolated, 

uninsured, and vulnerable. In keeping with its commitment to quality health care and in attempt to improve health care for 

all while eliminating disparities, in 1998 the BPHC began the Health Disparities Collaborative on a national basis. The 

Health Disparities Collaborative is a sweeping quality improvement initiative that uses the MacColl Institute’s Chronic Care 

Model and the Model for Improvement developed by Associates in Learning (Wagner, Austin, and Von Korff 1996; Langley 

G. et al. 1996). Initially 88 health centers participated in a collaborative that targeted improvements in diabetes care and 

outcomes. Subsequent collaboratives have focused on asthma, depression, cardiovascular disease, cancer screening and 

planned care, finance, and clinical systems redesign. As of April 2007, 1009 health centers were participating in the 

Collaborative [personal communication: Health Resources and Services Administration, April 19, 2007]. Participation in the 

 

1 Grant Support: This project was supported by grant number 1 U01 HS13635 from the Agency for Healthcare Research and Quality, with 

support from the Health Resources and Services Administration. Additional support came from the Agency for Healthcare Research and 

Quality (R01 HS10479), and the National Institute of Diabetes and Digestive and Kidney Diseases Diabetes Research and Training Center 

(P60 DK20595). 
Collaborative requires commitment of resources to attend learning sessions, establish and maintain information systems, and 

introduce and monitor the impact of systems changes. 

NORC, in partnership with the University of Chicago and the MidWest Clinicians’ Network, evaluated the impact of the 

Collaborative at more than 150 health centers, collecting data in two waves to better assess the long term integration of key 

Collaborative components. While the two cross-sectional samples were not identical, over half (53.7%) of Wave II 

respondents were also surveyed in the Wave I effort, more than half of those eligible completing surveys at both points in 

time. This overlap in sample affords us the opportunity to evaluate survey nonresponse. 

Limiting unit nonresponse is critical for any survey as it is a potential source of bias and can affect the generalizability of the 

survey findings (Rogelberg & Luong 1998). Much attention is given to preventing nonresponse by adjusting the type and 

quality of respondent contacts, including the number of follow up attempts, the use of advance letters, and the length and 

format of the survey instrument, with many of these strategies having been shown to improve response rates. For example, 

delivering mailed surveys via Fed Ex has produced higher rates of response than traditional mailings (Hager et al. 2003) and 

enclosing a return envelope with stamped postage produced higher response than those including business reply envelopes 

(Fox, Crask, & Kim 1988). Others have found that enclosing letters of support from relevant organizations increases the 

likelihood of response (Bartholomew & Smith 2006). Demographic comparisons of respondents and nonrespondents has 

also produced some significant associations, with research showing that women (Green 1996; Spitzmuller et al. 2006; 

Rogelberg & Luong 1998), whites (Jackson & Ivanoff 1999), and older persons (Green 1996; Moum 1994) are more likely to 

respond to surveys that others. These same characteristics are associated with decreased attrition in longitudinal studies 

(Goodman & Blum 1996). 

However, demographic characteristics may produce lesser associations when examining nonresponse in organizational 

surveys. Guadagnoli & Cunningham (1989) found no variation in response propensity among physicians across demographic 

variables or practice characteristics and Spitzmuller et al. (2006) found no association between the number of hours worked 

per week and propensity to respond. Response to organizational surveys was, however, found to be associated with higher 

levels of organizational commitment and satisfaction (Rogelberg et al. 2003). These outcomes may be partially explained by 

leverage-salience theory (Groves, Singer, & Corning 2000) as research has shown that respondents with demonstrated 

interest in a survey topic participate at higher rates than do others (Goyder 1987; Groves et al 2004; McDaniel, Madden, and

Verille 1987). While it is the goal of survey researchers to increase response rates by developing questionnaires that are of 

interest to respondents, differential response based on topical salience has the potential to lead to unintended nonresponse 

error (Rubin 1987). As such, our multi-wave evaluation provides an opportunity to examine nonresponse based on variation 

in respondent participation in the Collaborative, impressions of Collaborative outcomes, as well as demographic 

characteristics. 

Methods 

As wide variation exists among health center staff with regards to knowledge of and participation in the Health Disparities 

Collaborative, we developed five unique versions of the survey instrument, each tailored to the respondent’s specific role at 

the health center and in the collaborative effort. Specifically, questionnaire versions were designed for (1) the health center’s 

Chief Executive Officer (CEO) or Executive Director; (2) the Medical Director; (3) the Collaborative Team Leader; (4) 

Collaborative Team Members; and (5) health center staff not participating in the Collaborative effort. Most survey items 

included a 5-point Likert response scale, and focused on attitudes regarding the quality improvement effort including: 

perceptions of clinical outcomes and costs; support from staff and leadership, resource availability, and employee 

satisfaction. 

Eligibility was restricted to centers located in the Midwest and West Central regions of the U.S. who had participated in the 

Collaborative for at least one year. Of the 173 health centers identified, 95 percent provided complete staff listings including 

individual names, positions at the center, and role, if any, in the Collaborative. Our sample was then selected from this list, 

and included all identified CEOs, Executive Directors, Medical Directors, Team Leaders, and a random sample of up to three 

staff members not participating in the Collaborative effort. A respondent’s position at the health center had no effect on his

or her eligibility, and our sample was not restricted to practitioners. 

Wave I data were collected between March and December 2004 following the standards of Dillman’s (1978) Total Design 

Method, utilizing varying modes of contact. Our initial mailing was sent to more than 1,500 eligible respondents. Next, 
using express delivery we sent out 2 additional mailings of the survey to non-respondents. To further increase response we 

conducted telephone prompting and mailed additional surveys along with letters of support by relevant officials. Our overall 

response rate for Wave I was 68.1 percent, with questionnaire-specific rates ranging from a high of 78.9 percent for Team 

Leaders to 57.6 percent for staff not participating in the Collaborative. 

 

Wave II data collection took place across approximately six months beginning in the fall of 2005. Unlike the first wave, only 

respondents with direct participation in the Collaborative were surveyed. Two instruments were developed, differing only on 

the inclusion or exclusion of financial issues. Wave II questionnaires were substantially shorter than those fielded in Wave I,

taking approximately half the time to complete. Nonresponders were mailed additional copies of the questionnaire via 

express delivery, but limited telephone follow-up was conducted. Our final overall response rate for Wave II was 68.7 

percent. 

Analysis 

We examined outcomes of the 782 respondents asked to participate in both data collection waves. Seventy-two percent of 

those who completed the survey at Wave I also participated at Wave II, as well as an additional 71 respondents who had 

refused at Wave I. Our analyses first compare two types of respondents: those who completed surveys at both waves (N = 

415), and those who participated only in Wave I (N = 576). The two groups were compared against several demographic 

characteristics including gender, race, ethnicity, and age, as well as whether respondents were physicians, their length of 

employment at the health center, and if they were responsible for a panel of patients during Wave I data collection (Time 1). 

We also compared several measures of involvement in the Collaborative effort, including respondents’ role on the 

Collaborative team, length of time in that role, hours per week spent on the Collaborative, and change in participation levels 

between waves. Lastly, we measured opinions on the success of the Collaborative itself, creating an index of the five 

variables evaluating Collaborative outcomes.2

 Specifically, respondents were asked their level of agreement3 to the 

following statements, “Overall, the Collaborative has been a success” and “The Collaborative has been worth the effort,” and 

whether the Collaborative had improved or worsened4

 patient health outcomes, processes of clinical care, and patient 

satisfaction. Univariate statistics are presented in Table 1 below. We then developed two logistic regression models to 

predict response at (1) Time 1 only; and (2) both Times 1 and 2. Stepwise procedures were used to obtain the most 

parsimonious models (Searle et al. 1992; Verbeke et al. 1997) which are presented in tables 2 and 3 below. 

Results 

Table 1 compares characteristics of respondents who participated only at Time 1 or at both Times 1 and 2. Respondents who 

participated in both waves of data collection were more likely to be male and white, with 75 percent of men and 82 percent of 

whites responding at both points, compared to 62 percent and 59 percent respectively at Time 1 only. Respondents who 

participated in both waves were found to be slightly, but significantly, older than Wave I only respondents, and employment 

tenure at the health center was also significantly associated with continued survey participation. Approximately one-quarter 

of “time 1 only” respondents were physicians compared to just 16 percent of those who responded to both questionnaires, and 

we found no significant difference between those who were responsible for a panel of patients and those who were not. 

Level of involvement in the Collaborative varied by response group with 19 percent of Team Leaders responding at both 

times compared to just 3 percent at Time 1 only. Those who responded at both waves worked, on average, three more hours 

per week on the Collaborative than those who only participated at Time 1. Similarly, 28 percent of those participating in 

both waves reported increases in their Collaborative involvement at Time 2, as compared to 7 percent who reported decreased 

involvement. 

Our multivariate models show similar results with odds of responding to both questionnaires greater for whites and males. 

The number of hours per week spent on the Collaborative and tenure at the health center also produced significant, yet very 

small increases in odds. We found a negative association between response only at Time 1 and three independent variables: 

being white, spending more time on the Collaborative effort, and being older. The only measure positively associated with 

response to only the first wave of data collection was the presence of a Collaborative champion during at the health center. 

 

2

 Cronbach’s alpha coefficient = 0.8097. 

3

 Five-point Likert scale with response categories: strongly disagree; disagree; neither agree nor disagree; agree; and strongly agree 

4

 “During calendar year 2003, to what degree did the Collaborative worsen or improve the following at your center?” Response categories 

included: greatly worsened; somewhat worsened; no change; somewhat improved; greatly improved.
Discussion 

Our analyses add support to the association between topical interest and survey response. Employees who worked more 

hours per week on Collaborative tasks and those with longer tenure at the health center were more likely to respond to our 

questionnaires during both data collection waves. We also found a significant association between increased responsibilities 

on the Collaborative and participation in the second wave of data collection. This measure, however, was not found to be 

predictive in our multivariate models. Our experience also confirms the need for strong sample development and sample 

management practices in longitudinal studies, as this component can inform not only locating and tracing efforts, but also any 

tailoring of respondent materials to demonstrate relevancy and promote participation. 

Lastly, while we recognize that topical salience is important in encouraging survey response it also has the potential to create 

nonresponse bias as those less motivated to participate may differ from respondents on key substantive issues. Using our 

index measuring perceived Collaborative outcomes and success we find no significant difference between those who 

participated in only the first wave and those who responded at both points in time. However, a key limitation of this analysis

is the lack of available data collected at both waves that would allow for a more comprehensive examination of substantive 

differences between responders and nonresponders. 
References 

Bartholomew, Susan & Anne D. Smith. 2006. “Improving Survey Response Rates from Chief Executive Officers in Small 

Firms: The Importance of Social Networks.” Entrepreneurship Theory and Practice 83-96. 

Chin MH, Cook S, Drum ML, Jin L, Guillen M, Humikowski CA, Koppert J, Harrison JF, Lippold S, Schaefer CT. 

“Improving diabetes care in Midwest community health centers with the Health Disparities Collaborative.” Diabetes Care

27:2-8, 2004 

Chin MH, Kirchhoff AC, Walters AE, Graber JE, Brown SES, Rimington A, Drum ML, Schaefer CT, Heuer L, Huang ES, 

Shook ME, Tang H, Casalino LP. “Incentives and assistance for improving the Health Disparities Collaborative in 

community health centers.” Medical Care, In review. 

Dillman, Don A. 1978. Mail and Telephone Surveys: The Total Design Method. New York: Wiley. 

Fox, Richard, Melvin R. Crask, & Jonghoon Kim. 1988. “Mail Survey Response Rates: A Meta-Analysis of Selected 

Techniques for Inducing Response.” Public Opinion Quarterly 52:467-491. 

Gannon. M., J. Northern & S. Carroll. 1971. “Characteristics of Non-Respondents Among Workers.” Journal of Applied 

Psychology 55: 586-588. 

Goodman, Jodi S. & Terry C. Blum. 1996. “Assessing the Non-Random Sampling Effects of Subject Attrition in 

Longitudinal Research.” Journal of Management 22(4): 627-652. 

Goyder, John. 1987. The Silent Minority: Nonrespondents on Sample Surveys. Boulder, CO: Westview Press. 

Green, K.E. 1996. “Sociodemographic Factors and Mail Survey Response.” Psychology and Marketing 13(2): 171-184. 

Groves, Robert, Eleanor Singer, and Amy Corning. 2000. “Leverage-Salience Theory of Survey Participation: Description 

and Illustration.” Public Opinion Quarterly 64: 299-308. 

Groves, R.M., S. Presser, and S. Dipko. 2004. “The Role of Topic Interest in Survey Participation Decisions.” Public 

Opinion Quarterly 68: 2-31. 

Guadagnoli, E. and S. Cunningham. 1989. “The Effects of Nonresponse and Late Response on a Survey of Physician 

Attitudes.” Evaluation and the Health Professional 12:318-328. 

Hager, Mark A., Sarah Wilson, Thomas H. Pollak & Patrick Michael Rooney. 2003. “Response Rates for Mail Surveys of 

Nonprofit Organizations: A Review and Empirical Test.” Nonprofit and Voluntary Sector Quarterly 32(2): 252-267. 

Jackson, A.P. & A. Ivanoff. 1999. “Reduction of Low Response Rates in Interview Surveys of Poor African-American 

Families.” Journal of Social Science Research 25(1-2): 41-60. 

Langley G, Nolan KM, Nolan TW, et al. 1996. The Improvement Guide: A Practical Approach to Enhancing 

Organizational Performance. San Francisco: Jossey-Bass. 

McDaniel, S., C.S. Madden & P. Verille. 1987. “Do Topic Differences Affect Survey Non-Response?” Journal of the 

Market Research Society 29: 55-67. 

Moum, T. 1994. “Is Subjective Well-Being a Predictor of Nonresponse in Broad Population Surveys?” Social Indicators 

Research 32(1): 1-20. 

National Association of Community Health Centers. 2006. “About NACHC.” [Accessed January 04, 2007]. Available at: 

http://www.nachc.com/about/ 
Rogelberg, S.G., J.M. Conway, M.E. Sederberg, C. Spitzmuller, S. Aziz & W.E. Knight. 2003. “Profiling Active and 

Passive Nonrespondents to an Organizational Survey.” Journal of Applied Psychology 88(6): 1104-1114. 

Rogelberg, S.G. and A. Luong. 1988. “Nonresponse to Mailed Surveys: A Review and Guide.” Current Directions in 

Psychological Science 7: 60-65. 

Rubin, Donald R. 1987. Multiple Imputation for Nonresponse in Surveys. New York: Wiley. 

Searle, S. R., G. Casella, and C.E. McCulloch. 1992. Variance Components. New York: John Wiley & Sons, Inc. 

Smitzmuller, Christiane, Dana Glenn, Christopher Barr, Steven Rogelberg & Patrick Daniel. 2006. “If you treat me right, I 

reciprocate’: Examining the Role of Exchange in Organizational Survey Response.” Journal of Organizational Behavior 27: 

19-35. 

Wagner, E.H., B.T. Austin, and M. Von Korff. 1996. “Organizing care for patients with chronic illness.” Milbank Quarterly 

74 (4): 511-544. 
Table 1. Characteristics of Participating Respondents 

Wave 1 Both 

Waves 1 & 2 

N=576 N=415 

Demographic Characteristics 

Male** 63.0% 75.0% 

White** 58.7% 82.2% 

Hispanic or Latino 15.2% 14.1% 

Age (mean in years)+47.4 49.0 

Physician 22.8% 16.5% 

Responsible for a Panel of Patients 40.2% 40.3% 

Tenure at health center at Time 1 (mean in months)** 83.0 95.1 

Experience with the Health Disparities Collaborative 

Role in Collaborative at Time 1 

Team Leader 3.3% 19.2% 

Team Member 58.7% 54.8% 

Not on the Collaborative Team 38.0% 26.0% 

Tenure in Collaborative role at Time 1 (mean in months) + 42.0 46.4 

No Change in involvement at Time 2** — 61.9% 

Involvement in Collaborative decreased at Time 2** — 7.5% 

Involvement in Collaborative increased at Time 2** — 28.0% 

Mean Hours per week on Collaborative – Time 1+4.2 6.9 

Mean Hours per week on Collaborative – Time 2 — 6.4 

Champion of Collaborative present at health center at Time 1 80.4% 78.2% 

Index of Collaborative outcomes at Time 1 20.2 20.3 

p values for t-test of equal variances: + p<.10; * p<.05; ** p<.01 

Table 2. Multivariate Logistic Regression Estimates of Likelihood of Response to Wave I Only (N = 481) 

Variable 

Regression 

Coefficient 

(SE) 

P-Value Odds Ratio 

Respondent is white -0.6827 

(0.2895) 0.0184 0.505 

Hours per week on Collaborative at Time 1 -0.0561 

(0.0222) 0.0113 0.945 

Center has Collaborative Champion 0.6868 

(0.3861) 0.0753 1.987 

Age -0.0237 

(0.0132) 0.0720 0.977 
Table 3. Multivariate Logistic Regression Estimates of Likelihood of Response to Both Questionnaire 

Waves (N = 493) 

Variable 

Regression 

Coefficient 

(SE) 

P-Value Odds Ratio 

Respondent is white 1.2922 

(0.2330) <.0001 3.641 

Respondent is male 0.4471 

(0.2346) 0.0567 1.564 

Hours per week on Collaborative at Time 1 0.0345 

(0.0144) 0.0165 1.035 

Tenure at Health Center 0.00478 

(0.00161) 0.0031 1.005