# Presentation: 413441

## Methods to Detect and Control Bias in Comparative Analyses of Disability Statistics

**Methods to Detect and Control Bias in Comparative Analyses of Disability Statistics**

**5****th****  Annual Meeting of the Washington Group on Disability Statistics**** ****Rio de Janeiro, Sept 21-23 2005**

- Bedirhan Üstün & Nenad Kostanjsek
- Classifications Assessment Terminology
**Measurement and Health Information Systems**
_**www.who.int/classifications/icf**_

## Slide 2

## Temperature: Hot, Mild or Cold ???

_**Measured  Real temperature 20 C**_**: **
** ****	****Iceland ****	****-  ****	****Hot **
** ****	****Germany-****	****Mild **
** ****	****Brazil****	****- ****	****Cold**** **

## Big Mac Index

- University Professor in Turkey gets 	      	 3000 USD.
- University Professor in the Switzerland gets    8000 USD.
- Big Mac in TR = 1.5 USD
- Big Mac in CH = 4 USD
- Salary = 2000 Big Macs
-  in both

##  Health & Disability Survey Data

**prove useful to complement National Health & Disability Information Systems**
**more representative of entire populations**
**direct information from individuals on their own experiences rather than professionals or surveillance information**
**can be tailored to specific management and evaluation purposes in mind:**
**p****rovide empirical data for system performance assessment**
** ****More like  to be standard and accurate **
**gathered more carefully**

## Numerous 
National and International Health & Disability Surveys exist and they differ in:
Content
Questions
Response scales
Samples
Timing 

**National and International Health & Disability Surveys exist and they differ in:**
- Content
- Questions
- Response scales
- Samples
- Timing

** ****DOWNSIDE of Surveys**

## Problem of Comparability in International Health Data

- Mortality statistics  		3 x
- Health Survey data 		6 – 8 x
- Disability Survey data 	30 – 60 x
- Health Interventions	      40 – 300 x

**Notes:**
a statistical measure of the spread or variation of a group of numbers in a sample, equal to the square of the standard deviation. Other measures of the spread are the ratio of the squared standard deviation to the sample size population variance, and the ratio of the squared standard deviation to the sample size minus one sample variance 

## Methods for Comparability1. Same Concepts

**Methods for Comparability****1. Same Concepts**

**Examples:**** **
**Disability = Impairment, Activity limitation & Participation restriction *****(ICF definition)***
**Disability = Impairments & simple activities **
**Hearing = body function**
**Communicating = activity and participation domain**
**The self-care category includes the sub-categories: washing, dressing, toileting.... **

## Methods for Comparability2. Same Domains & Questions

## Methods for Comparability2. Same Domains & Questions (cont.)

## Methods for Comparability3. Same response scale (cont.)

- Different response categories for *severity scale*
- none, mild, moderate, severe...
- Numerical scales (0,1,2,3,4)  (1,2,3,4,)
- Likert  visual
- Percentage visual scale
-  Different response categories for *frequency scales*
- always, usually, sometimes, never or all the time,
- once, a few times, sometimes,  many times

## Methods for Comparability3. Same response scale (cont.)

## Methods for Comparability4. Translation protocols

- Translation/interpretation of **questions**
- Translation/interpretation of **response scales**
- The meaning of the **underlying construct** may vary
- The **relation** between the questions and the underlying construct may vary

## Methods for Comparability4. Translation protocols (cont.)

- The item is difficult or impossible to translate because there is _**no equivalent term in the local language. **_
- The item, when translated, has a _**meaning that is more narrow**_** **than the original term. Only part of the connotations of the original term carries over into the local language.
- The item, when translated, has a _**meaning that is expanded**__ _beyond the meaning found in the original. The translated items has additional meanings that would change the interpretation of the item.
- The item is translated into exactly the _**same word that has been used for another item in the questionnaire**_. Words that are different concepts in the questionnaire become merged into the same concept in the local language.
- The item can be translated, but there is a _**cultural applicability problem**_ with the item, or with the definition, or with the examples used for the item.

## Methods for Comparability5. Same application mode – same stimulus

**Respondent**

**Input**

**Output**

## Methods for Comparability6. Measuring and adjusting for shifts in cut points “Different Rulers”  Do response categories mean the same in different populations?

## Systematically inconsistent reporting in different surveys 

- men and women
- age groups
- education groups
- economic level
- education levels
- cultural and ethnic groups
- occupations
- levels of health insurance.

## Methods for adjusting for shifts in cut points Calibration Tests: Vision

**used extensively in international studies**
_**Distant Vision**_
**standard Snellen’s chart**
_**Near vision**_** **
**standard card for use in illiterate populations**

## Methods for adjusting for shifts in cut points Calibration Tests: Cognition

**Verbal fluency**
- names of animals in one minute
- record correct number & errors
**Simple Memory****  **
- Immediate recall
- delayed recall (20 min.) of word list
**Colour cancellation test****  **
- time taken
- number of correct and wrong cancellations

**selected from WHO international studies on **
**Culture-fair Cognitive Battery**

## Methods for adjusting for shifts in cut points Calibration Tests: Mobility

**Posturo-Locomotor-Manual test**
**Time to stand up**
**Time taken to complete task**
**3 trials**
**stand up from sitting position**
**walk one step**
**lift up weight of 2 kgs**
**walk 6 metres**
**place weight at shoulder weight**
**lift up weight**
**walk back towards chair**
**place weight back **
**return to chair and sit down**

## Methods for adjusting for shifts in cut points Vignettes

**Mobility ****	**
**	****[Mary] has no problems with walking, running or using her hands, arms and legs. She jogs 4 kilometres twice a week.**
**	****[Anton] does not exercise. He cannot climb stairs or do other physical activities because he is obese. He is able to carry the groceries and do some light household work. ****	**
**	****[David] is paralyzed from the neck down. He is unable to move his arms and legs or to shift body position. He is confined to bed.**

## “bias”Differential Item Functioning

- a distortion *(possibly  systematic and non-random)* of the results
- due to a *measurement error* affects all measurements in the same way
- violates the fair comparison of results across different groups
- substantive implications?
- statistically significant?
- random or systematic?

## Methods for Cross-Population Comparability

**Same concepts **
**Same domains & questions**
**Same response scales**
**Translation protocols**
**Same application mode – same stimulus**
**Correction for confounders (education, income...) using known calibrators**
**Use modern Item Response Theory (IRT) approaches**