The Use of Non-Probability Samples to 

Characterize Rare Conditions 

John M. Boyle, Ph.D. ICF International 

Federal Committee on Statistical Methodology 

November 2013 

icfi.com | 1 
Rare Diseases 

テ Rare Diseases are generally defined by population prevalence, which differ 

somewhat across countries 

テ United States: a rare disease is one that affects less than 200,000 persons 

(1 in 1,500 ) 

テ There are an estimated 5,000 to 8,000 rare diseases 

テ The majority of these conditions are serious and potentially lifethreatening 

テ An estimated 10% of the U.S. population is affected by one or more rare 

diseases 

テ In summary, rare diseases are important to public health due to the 

number of persons affected in the aggregate and the impact of the 

conditions on the affected individuals and the health care system 

icfi.com | 2 
Evidenced Based Medicine 

テ Evidence based medicine (EBM) is currently treated as the best tool or 

gold standard to validate clinical decisions about the care of patients, 

individually or as a whole 

テ Evidence based medicine is the conscientious, explicit and judicious use 

of current best evidence in making decisions about the care off 

individual patients 

テ The practice of evidence based medicine means integrating individual 

clinical expertise with the best available external clinical evidence from 

systematic research. 

テ <┗ideミIe#Hased#マediIiミe#is#さdeヴi┗ed#fヴoマ#high#ケuality#ヴeseaヴIh#oミ#

population samples to inform clinical decision-マakiミgざ#

テ Evidence based medicine evolved from clinical epidemiology --- using 

population health sciences to inform clinical practice 

icfi.com | 3 
 

Evaluating Evidence in EBM 

テ Evidence quality can be assessed based on the source type from metaanalyses and systematic reviews of triple-blind randomized clinical trials 

down to conventional wisdom at the bottom 

テ Good quality, reliable evidence does not always have to come from clinical 

trials 

テ High quality evidence has good internal and external validity 

テ Critical appraisal of evidence for internal validity that can be broken down 

into aspects regarding: 

テ Systematic errors as a result of selection bias, information bias and confounding 

テ Quantitative aspects of diagnosis and treatment 

テ The effect size and aspects regarding its precision 

テ Clinical importance of results 

テ External validity or generalizability 

icfi.com | 4 
 

Primary Immune Deficiency Diseases 

テ Primary immune deficiency diseases (PIDD) is a class of approximately 

150 specific disorders 

テ Immune deficiency diseases both individually and as a class of disorders 

are recognized as rare diseases (NIH Office of Rare Diseases) 

テ ^iHヴaヴy#of#dediIiミe#aHstヴaIt:#さケ-Linked Agamma-globulinemia Marks the 

Spot: Rare Diseases in Evidenced-1ased#wヴaItiIeざ#

テ Uniquely, the Immune Deficiency Foundation commissioned a national 

probability sample of 10,000 households sampled by random digit dialing 

and screened by telephone in 2005 

テ Identified 18 households and 23 persons with a PIDD diagnosis 

テ Estimated prevalence for the class would be 1/1,200 persons 

icfi.com | 5 
Evidence-Based Medicine and Treatment 

テ EMB is usually used in the context of treatment decisions (treat 

or not) 

テ Random control trials are gold standard for assessment of 

efficacy and safety 

テ Rare diseases are a challenge for RCT in approval of new 

therapies 

テ BUT what about assessing treatment and treatment outcomes 

after FDA approval? 

テ The majority of patients with PIDD have antibody disorders for 

which immunoglobulin (IgG) therapy is the standard of care 

icfi.com | 6 
icfi.com |

 

 

Current IgG Use: IDF Patient Surveys and 

National Prevalence Survey 

70% 67%

74%

22%

0%

20%

40%

60%

80%

100%

1996 IDF Patient 2002 IDF Patient 2008 IDF Patient 2005 Prevalence

Sources: IDF Primary Immune Deficiency Diseases in America: 1996, 2002 & 2008, 

2005 IDF National Prevalence Survey 

icfi.com | 7 
2010 National PIDD Treatment Survey 

テ Characterize treatment with immunoglobulin 

replacement therapy among a national communitybased sample of PIDD patients 

テ Obtain a sample of at least 100 diagnosed PID 

patients to provide more stable estimates of 

immunoglobulin use 

テ Cost of probability survey would be prohibitive 

テ Member based survey may be biased toward treatment 

テ Large web panel of the general public to screen for a 

nationally representative sample of persons 

diagnosed with primary immune deficiency diseases 

may offer a less biased, non-probability source of 

information on treatment 

icfi.com | 8 
Web Survey Study Procedures 

テ Internet panel company sends invitations to samples of web panel 

members to participate in a survey 

テ Oミteヴested#paミel#マeマHeヴs#go#to#a#さヴeケuiヴeマeミts#pageざ#┘heヴe#they#selfidentify their eligibility for the survey 

テ さ<ligiHleざ#ヴespoミdeミts#aヴe#gi┗eミ#the#liミk#to#the#O8F suヴ┗ey#┘eHsite#to#

conduct the treatment interview 

テ These respondents must answer a more detailed set of questions to 

determine whether they are eligible for the survey 

テ Respondents must have a member of their household or an immediate 

family member living outside of their household to be eligible for the 

survey 

テ The household or immediate family member must have a specific 

appropriate diagnoses of primary immune deficiency diseases 

テ Those respondents with eligible household or immediate family members 

are interviewed on-liミe#IoミIeヴミiミg#the#wO8#patieミt’s#disease#aミd#its#

treatment 

icfi.com | 9 
icfi.com icfi.com | 10 

E-mail Invitation to participate in new survey opportunity N=859,379 

Went to web screener = 13% N=114,934 Did not respond= 87% N=744,445 

No One Qualified in Household or 

Family = 97%

 N=111,447 

Anyone in household or immediate family diagnosed 

with XLA, CVID, IgA def., IgG def., SCID or other 

immune deficiency? Yes=3% N=3,487 

Did not go to Main Questionnaire 

No=51% N=1785 

Went to main questionnaire Yes=49% N=1,702 

Has anyone in your household/immediate family ever been 

diagnosed with a primary immune deficiency disease? What 

specific types of primary immune deficiency has the (AGE) 

been diagnosed with? 

Good PIDD Diagnosis=9% N=144 

No answer to screen 10% 

No PIDD in HH/Family 47% 

No Diagnosis given 6% 

Bad Diagnosis given 28% 
PIDD Web Survey 

テ Goal: Obtain a sample of at least 100 diagnosed PID patients 

to provide more stable estimates of immunoglobulin use 

テ Result: Web panel yielded completed interviews with 147 

eligible respondents reporting on 160 PIDD 

テ Result: Web panel yielded interviews with119 patients with 

antibody disorders for which IgG treatment is recommended 

テ Result: Web panel found relatively low levels of IgG treatment 

in a national community based sample 

テ Problem: How do we evaluate the quality of the evidence 

from a non-probability sample such as a web panel? 

icfi.com | 11 
External Validity 

テ Geographic distribution --- compared to Census population distribution 

テ Prevalence --- compared to 2005 National Prevalence Survey (RDD) 

テ Treatment --- compared to 2005 National Prevalence Survey (RDD) 

テ Health --- compared to large, IDF member surveys (non-probability, but not 

convenience sample) 

icfi.com | 12 
 

 

External Validity: Distribution of Population 

and Patients by Region 

5%

15%

17%

7%

18%

6%

11%

6%

16%

6%

16%

8%

17%

6% 6%

17%

16%

6%

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

New

England

MidAtlantic E North

Central

W North

Central

S Atlantic E South

Central

W South

Central

Mountain Pacific

US Population Patients web

Patients in 2007 based on state of birth. Patients in web survey based on state of residence from 

sample file. Base: One or more eligible PIDD and completed interview N=144 (3 missing data) 

icfi.com | 13 
 

 

External Validity: 

Household Prevalence of PIDD 

テ 144 respondents have a qualified PIDD diagnosis in their 

household in web survey 

テ 114,934 panelists saw the screening questionnaire so 

1 out of 798 households prevalence in web survey 

テ If the distribution of eligible diagnoses was the same for those 

who qualified on the refinement question and did not go to 

the web survey, we would expect 295 eligible respondents for 

a household prevalence rate of 

1 in 390 households estimated prevalence in web survey 

テ Estimated household rate for diagnosed PIDD was 

1 in 555 households in RDD probability survey 

icfi.com | 14 
 

 

Compare with a Probability Survey: 

Past and Current IG Treatment 

44%

36%

29%

22%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

IVIG 2005 RDD N=23 IVIG 2010 Web N=160

Ever Current 

Q8a. Has the (AGE) ever been treated for a month or longer with the following? 

Q8c. Is the (AGE) currently being treated with any of the following? All living eligible PIDD N=160 

icfi.com | 15 
 

 

Compare with Other Non-Probability Survey: 

Current Health Status 

7% 8%

33%

37%

12%

4%

8%

21%

30%

28%

10%

2%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Excellent Very good Good Fair Poor Very poor

2010 Web 2002 Member

Q23. Would you describe the (AGE)’s current health status as ...?

Base: Selected PIDD patient N=147 

icfi.com | 16 
 

 

 

Compare to Other Non-Probability Survey: 

Limitations due to Health 

17%

33% 34%

16%

30% 29%

25%

14%

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

No limitation Slight

limitation

Moderate

limitation

Severe

limitation

2010 Web 2002 Member

Q24. How much, if any, is the (AGE) limited in work, play or normal physical activity as 

a result of his/her health? Base: Selected PIDD patient N=147 

icfi.com | 17 
 

 

Compare with Other Non-Probability Survey: 

Acute Conditions in Previous 12 Months 

45%

17%

34%

25%

16%

3%

8%

3%

17%

2%

17%

36%

14%

37%

24%

11%

5%

6%

3%

20%

5%

25%

0% 10% 20% 30% 40% 50%

Bronchitis

Candida

Repeated Diarrhea

Repated Ear Infections

Eye Infections

Lymphopenia

Malabsorbtion

Neutropenia

Pneumonia

Sepsis

Urinary Infection

2010 Web Survey

2002 Member Survey

Q25. Which of the following conditions, if any, has the (AGE) had during the past 12 

months? Base: Selected PIDD patient N=147 

icfi.com | 18 
Internal Validity 

テ Disease specific eligibility 

テ Correct diagnoses by clinical expectations 

テ Incorrect diagnoses by type 

テ Location of treatment by treatment level 

テ Contact with patient organization by treatment level 

icfi.com | 19 
Disease Specific Eligibility in Web: 

Minimizing Fraud and Error 

テ Must report household or immediate family member with

primary immune deficiency disease 

テ Must select one or more legitimate diagnosis of primary immune 

deficiency disease from pre-coded list of 18 legitimate and 13 

non-legitiマate#Ioミditioミs#plus#oミe#さotheヴざ#Iategoヴy#oミ#t┘o#

screens 

テ Must report one legitimate condition, and 

テ Not more than 5 persons in household (or immediate family) 

with PIDD conditions 

テ Not three or more PIDD conditions for any person 

テ Not two rare, non-PIDD conditions placed at beginning of list 

テ Not combinations of PIDD conditions that were improbable 

icfi.com | 20 
 

 

 

Internal Validity: Specific PIDD Diagnoses 

5%

1%

3%

13%

27%

3%

1%

1%

2%

12%

14%

6%

3%

5%

1%

1%

2%

0% 5% 10% 15% 20% 25% 30%

Agamma

Ataxia Tel

CGD

Combined

CVID

Complement

DiGeorge

Angiodema

Hyper IgM

IgG subclass

Selective IgA

SCID

SCN

SAD

W-A

XLP

Mixed

Q5a/b. What specific types of primary immune deficiency has the (AGE) been diagnosed with? 

Base: Selected PIDD patient N=147 (Conditions using IgG in blue) 

icfi.com | 21 
 

 

 

-

I 

I 

I 

I 

I 

. 

I 

I 

I 

I 

I 

I 

I 

I I I I 

Internal Validity: Non PIDD Diagnoses Reported as PIDD 

21%

18%

18%

11%

9%

9%

5%

5%

5%

5%

4%

3%

2%

2%

2%

2%

23%

2%

0% 5% 10% 15% 20% 25%

Fibromyalgia

Lupus

Rheumatoid Arthritis

Crohn or IBS

MS

Diabetes

AIDS/HIV

Sjogrens

Hashimoto's

Cancer/Leukemia

Chronic Fatigue Syndrome

ITP

Sarcodosis

CIDP

Hepatitis C

Autoimmune Hemolytic Anemia

Other

Not sure

Q5a/b. What specific types of primary immune deficiency has the (AGE) been diagnosed with? 

Base: PIDD Reported but Only Non-PIDD Diagnosis Given N=244 Auto-immune in Green 

icfi.com | 22 
 

 

Ever and Current Treatment with 

Immunoglobulin by Immunologist Location 

67% 69%

61%

25%

0%

60%

44%

20%

0%

61%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

University

hospital

Non-university

hospital

Private

practice/HMO

None Not sure

Ever Current 

Q8a. Has the (AGE) ever been treated for a month or longer with the following? Q8c. Is the 

(AGE) currently being treated with any of the following? Base: Selected PIDD patient N=147 

icfi.com | 23 
 

 

 

Internal Validity: Contact with IDF 

5%

9%

22%

14%

8%

2%

2%

6%

1%

59%

0% 10% 20% 30% 40% 50% 60% 70%

Called them

Get newsletter

Visited website

Seen patient information

Told about by others

Other

Only this survey

No contact

Not sure

Never/not sure heard of them 

Q41b. What kinds of contact, if any, have you had with the Immune Deficiency Foundation? 

Base: Selected PIDD patient N=147 

icfi.com | 24 
 

 

 

Internal Validity: Treatment with 

Immunoglobulin by IDF Contact 

100% 92% 92% Ever Current 

90%

80%

70% 62%

60%

49% 51%

50% 41%

40%

30%

20%

10%

0%

Receive Newsletter Heard of IDF/No Never head of IDF

newsletter

Q8a. Has the (AGE) ever been treated for a month or longer with the following? Q8c. Is the 

(AGE) currently being treated with any of the following? Base: Selected PIDD patient N=147 

icfi.com | 25 
Evaluating a Non-Probability Sample for Data Quality 

テ Patients distributed nationally similar to Census 

テ Household prevalence of diagnosed PIDD in web panel 

equivalent to RDD telephone survey 

テ Treatment rates similar in web and RDD surveys 

テ Correct diagnoses distributed to clinical expectations 

テ Wrong diagnoses primarily auto-immune diseases and similar 

to probability survey (not presented here) 

テ Disease characteristics of web panel patients similar to 

previous member surveys 

テ Treatment rates vary by immunologist location 

テ Only a minority have heard of IDF and less than one in ten 

would be in the IDF database 

テ Treatment rates of patients in web survey who receive IDF 

newsletter similar to IDF member surveys 

icfi.com | 26 
Conclusions 

テ Incidence of rare diseases make population estimates of treatment and 

treatment outcomes infeasible using probability samples 

テ National web population panel was able to generate a national sample of a 

rare disease of usable size 

テ Web panel sample was compared to Census, probability surveys, other nonprobability surveys for external validity 

テ Web panel sample was analyzed for internal validity for indications of fraud, 

error and internal consistency 

テ Findings from this web survey of a rare disease appear to be internally and 

externally consistent 

テ These estimates confirm a potential problem in treatment in the general 

community that was suggested by probability survey 

テ Carefully constructed and conducted surveys using an appropriate web 

panel can provide credible measures of the treatment of a rare disease in 

the community 

テ Can we develop a set of procedures for evaluating the quality of nonprobability samples for rare diseases that would permit their use in 

evidence-based medicine? 

icfi.com | 27 
 

Thank You 

John M. Boyle, Ph.D. 

Lead, Survey Research Line of Business 

ICF International 

john.boyle@icfi.com 

icfi.com | 28