# Presentation: 690601

## Household Survey Comparison

- Melissa Heinen, M.P.H.
- Margaret Warner, Ph.D.
- Lois Fingerhut, M.A.
**Montreal 2002**

**Notes:**
There is increased interest in comparing household injury surveys across countries and or developing a comprehensive method for collecting injury survey information.  Lois Fingerhut recently attending a meeting with the Injury Prevention Program of the European Commission and was asked what the single – best injury question would be.  

There wasn’t an answer to this question.  

This renewed our interest in comparing injury surveys.  

This presentation will discuss our most recent comparison of the 6 injury surveys available on the ICE Website.  

## Importance of Household Surveys

- Not limited to medical records
- Health professionals description
- More detail about the event
- Costs and resources

**Notes:**
Household surveys main strength is that one is not limited to what is available in medical records.

## National Household Injury Surveys

**Surveys reviewed**
- Canada – Canadian Community Health Survey (CCHS) and the National Population Health Survey (NPHS)
- New Zealand – National Health Survey
- Scotland – Scottish Health Survey
- South Africa – South African Demographic and Health Survey
- United States – National Health Interview Survey

- SOURCE: CDC/NCHS

**Notes:**

The review process involved examining surveys that were available on the internet and comparing various important survey components.  
A follow-up e-mail was sent to survey contacts for clarification.  

I have put together summary tables based on my review.  But is was hard to fit some of the items into boxes.  If you notice something about your countries survey that does not seem to be correct, please feel free to make me aware of this either during my talk or after.  

Our first country listed here, Canada, has two separate surveys that include injuries. 
New Zealand
Scotland 
South Africa
The United States

## National Household Injury Surveys

**Items compared**
- Recall Period
- Severity Threshold
- Survey Methodology
- Unit of Analysis
- Injury Definitions
- Survey Frequency
- Probing / Screening Question

- SOURCE: CDC/NCHS

**Notes:**

Next we will address with each of these items compared and discuss some important items to consider.  I will then present what each of the countries we reviewed practice regarding these factors.

Now things to consider are not limited to these listed, but these are the only ones actually compared.  At the end I will list other ones that would be beneficial to assess.

## Recall Period

**Items to consider**
- Memory decay
- Telescoping
- Heaping
- Sample size / relative standard error

- SOURCE: CDC/NCHS

**Notes:**

The longer individuals remember back to their injury, the less likely they are to remember the incident. Especially the less severe episodes.  

Also, the longer a recall period is, an increase in heaping and telescoping can occur.  Heaping is when a prefererance for a date 

If you try to shorten your recall period, there is cost of decreasing the size of your sample.  

One needs to find a compromise between these composing parts and come up with a recall period they are comfortable with.

## Annual estimates of injury and poisoning conditions using different recall periods, NHIS 1997

## Recall Period

- SOURCE: CDC/NCHS

| Canada | 12-months |
| --- | --- |
| New Zealand | 12-months |
| Scotland | 12-months |
| South Africa | 1-month |
| United States | 3-months |

**Notes:**
Scotland, New Zealand, and Canada all had a recall period of a year.  This would allow for a larger sample size, but could have the offset of possibly a smaller proportion of less severe injuries reported.  

The US historically collected injuries with a recall period of 2 weeks.  The sample size was too small for multivariate analysis.  The recall period was increased to 3 months in 1997(?).  This increased the sample size, allowing for more detailed analysis of variables.  However there may have been possible loss of the less severe injuries with a longer recall period.

South Africa used a recall period of one month.  

Is the respondent asked what was the date of the injury?

## Severity Threshold

**Measurements**
- Limitation in daily activities
- Medical advice or treatment
- Hospitalization status

- SOURCE: CDC/NCHS

**Notes:**
The severity threshold allows some measure of impact the injury made to the individual’s life.  The severity measure is very related to the length of recall. The longer the recall you have the higher the level the severity threshold needs to be.

Some of the ways that severity threshold is measured is by a limitation in daily activities, medical treatment or advice, or the individual was hospitalized.  

## Severity Threshold

**Items to consider**
- Health insurance coverage
- Subjective - health seeking behaviors
- Accessibility of health professional (rural vs. urban)
- Emergency department visits

- SOURCE: CDC/NCHS

**Notes:**
If “medical treatment or advice” is chosen as the level of severity, a number of items can effect consistency of measurement.  

An individual’s insurance often determines when a health professional is visited.  

Also, this measurement is very subjective.  Some individual’s are more likely to seek medical attention and while others are more likely to “wait out” an injury, before consulting a health professional.

Also, where a person lives, can effect whether a person sought medical advice.  If a person lives in a more rural setting, medical services may be less available, regardless of medical coverage and health seeking behaviors.  

The amount of waiting the individual will need to do determines whether they will seek medical advice or treatment.  This is try with non-socialized medicine and socialized medicine.
  We recently held cognitive interviews to test some of the questions for the National health interview survey, and one gentleman stated that he did not get medical advice or treatment for his burn, because he did not want to have to wait in the local hospital’s emergency room.  He estimated his wait would have been about 3-6 hours.  So instead, he self treated and medicated his injury and pain.

## Severity Threshold

- SOURCE: CDC/NCHS

| Canada | Limitation of normal activities |
| --- | --- |
| New Zealand | Received medical treatment |
| Scotland | Saw a doctor, nurse, or other health professional |
| South Africa | Treated by a doctor or a nurse |
| United States | Got medical advice or treatment |

**Notes:**
Canada appeared to be the only country that used “limitation of normal activities” as the severity threshold.  

## Survey Methodology

**Items to consider**
- Non-response rates
- Representation
- Sensitive questions and non-reporting
- Cost and resources

- SOURCE: CDC/NCHS

**Notes:**
When determining what method to collect the survey information – the information you are trying to compile and resources available must be considered.  

The method can affect who decides to participate in the study and what kind of information the individual will share.  

Non response rates can very by survey method and by population sub groups.

The US has seen an increase in non-response rates from telephone interview questions.

Telephone interviews cost significantly less than face-to-face interviews.  This needs to be taken into account when determining what method will be used for one’s survey

## Survey Methodology

- SOURCE: CDC/NCHS

| Canada | NPHS -Telephone 
CHHS – Telephone and face-to-face |
| --- | --- |
| New Zealand | Face-to-face |
| Scotland | Face-to-face |
| South Africa | Face-to-face |
| United States | Face-to-face |

## Unit of Analysis

**Items to consider**
- Comparability of numbers
**Events (e.g., car crash)**
**Persons**
**Most severe**
**Most recent**
**Conditions (e.g., fractured leg)**

- SOURCE: CDC/NCHS

**Notes:**
The definition of the unit of analysis is needed to help compare numbers across surveys and define the rate or number.

## Unit of Analysis

- SOURCE: CDC/NCHS

| Canada | Injury events |
| --- | --- |
| New Zealand | Injured persons |
| Scotland | Person based – most recent injury-related event |
| South Africa | Injured persons |
| United States | Injury events |

**Notes:**
Canada and US count injury events.

Scotland counts most recent injury-related event.

New Zealand and South Africa – count injured persons.

## Injury Definitions

**Items to consider**
- Self report vs. coding of self reports (ICD coding)
- Comparability

- SOURCE: CDC/NCHS

**Notes:**
Another item to consider is how the injury is defined.  

What are your inclusion and exclusion criteria? 
Is an ICD code use to “verify” that it is an injury?  If so what ICD code..  
Also who is coding or classifying the injury is it  the interviewer fitting the response into a set of categories or is it a trained nosologist coding from a set of verbatim of how the injury happened?
Or is the injury taken at face value.  If the respondent states that they are injured are they counted as injured or are they compared to a standard measurement?

For instance people respond in our survey that they were injured – they have a back injury from repeated lifting.  This would not be considered an injury according to the ICD-9 codes (since it is repetitive).

## Injury Definitions

- SOURCE: CDC/NCHS

| Canada | ICD coded injuries |
| --- | --- |
| New Zealand | ICD coded injuries |
| Scotland | Self reported accidents |
| South Africa | Self reported injuries |
| United States | ICD coded injuries |

**Notes:**
This slide is the one that I had the hardest time putting together.  I think this is because this topic is hard to categorize by its nature.

Canada, New Zealand, and the US all use ICD codes to define and classify injuries. 

Scotland uses self reported accidents

South Africa uses self reported injuries.

## Survey Frequency

**Items to consider**
- Trends over time
- Assess data
- Sample size

- SOURCE: CDC/NCHS

**Notes:**
The more frequent the survey information is collected the better you are able to:

## Survey Frequency

- SOURCE: CDC/NCHS

| Canada | Conducts surveys annually and reports results every two years |
| --- | --- |
| New Zealand | Conducts survey every three years |
| Scotland | Conducts survey every three years |
| South Africa | Conducts survey every five years |
| United States | Conducts survey and reports results annually |

**Notes:**
Canada collects the information annually, but reports findings every 2 years.

New Zealand and Scotland conducted their survey every 3 years and South Africa every 5.

The US appears to be the only country reviewed that collects and presents information annually as of now.

## Screener Question

**Items to consider**
- Length
- Complexity
- Embedded examples

- SOURCE: CDC/NCHS

**Notes:**
As I mentioned earlier with the cognitive testing that was done with the US National Household Interview Survey, The longer the question and the more complex it is the greater level of confusion and misunderstanding occurred.

Examples appear to be very helpful and decrease the level of confusion or open interpretation on behalf of the interviewer and interviewee.  Examples help define the concept one is trying to measure and allows for a more consistent measurement with each individual.

## Screener Question

- SOURCE: CDC/NCHS

| Canada | “(Not counting repetitive strain injuries), in the past 12 months, that is from ## to yesterday, were you injured?” |
| --- | --- |
| New Zealand | “In the last 12 months, have you had an injury which you received medical treatment?” |
| Scotland | “How many accidents – about which you saw a doctor, nurse, or other health professional – have you had in the last 12 months?” |

## Screener Question

- SOURCE: CDC/NCHS

| South Africa | “Did you have any injury that was treated by a doctor or a nurse in the last 30 days?” |
| --- | --- |
| United States | “During the past 3 months, that is  since ###, were you or anyone in the family injured or poisoned seriously enough that you got medical advice or treatment?” |

## US Survey

- Decrease injury rate
- Cognitive testing -screening question
- Length
- Definitions
- Severity

**Notes:**
In recent years the NHIS injury rate began to decrease – we were confident that this was not a measure of the true trend in injury rates.
We decided to subject the instrument to cognitive testing.  We determined that the screener question was an area that needed great improvement.
The screening question was too long, too complex, and did not have any embedded examples.
People stated that they either missed the word poisoning or injured completely and thus did not report having any injury or poisoning.
One individual mentioned they would forget to mention family member injuries or poisonings.  
Many didn’t know what the word poisoning actually meant or included.  Most only could think of rat poisoning or food poisoning.  
Also, many thought the only type of injury or poisoning we wanted were ones that the individual almost died as a result of the injury or poisoning.  They thought we only wanted the most severe of the severe.  This could have been an artifact of the cognitive lab selection method, or it may reflect a true misunderstanding by the respondent.
Based on these findings we reworked the NHIS question and came up with a proposed change to the survey.

## Screener Question

**Proposed changes for US**
- During the past 3 months, that is since ##, did you have an **injury** where any part of your body was hurt, for example, with a **broken bone, sprain, burn, wound, cut, bruise, or animal or insect bite**?
- During the past 3 months, how many times were you **injured**?
- Did you talk to or see a medical professional about any of these **injuries**?
- Of the # times that you were injured, how many of those times was the** injury** serious enough that you consulted a medical professional?

- SOURCE: CDC/NCHS

**Notes:**

We decided to “unpack” our one question into four questions.  In testing it in the cognitive lab, we found far less confusion.  Respondents appear to use the example list as a cueing for their injury and was able to identify their episode.  

## Screener Question

**Proposed changes for US**
- During the past 3 months, that is since ##, were you **poisoned by swallowing or breathing in a harmful substance such as bleach, carbon monoxide, or too many pills or drugs**?  (Do not include food poisoning, sun poisoning, or poison ivy rashes)
- During the past 3 months, how many different times were **poisoned**?
- Did you talk to or see a medical professional about any of these **poisonings**?
- Of the # times that you were **poisoned**, how many of those times was the **poisoning** serious enough that you consulted a medical professional?

- SOURCE: CDC/NCHS

**Notes:**
As you can see the poisoning questions follow the same flow and questions, but with poisonings as being the focus.  We also found the same results with this question as we did with the injury question.

## National Household Injury Surveys

**Additional items to consider**
- Placement of injury section within a larger survey
- Length of survey
- Sample selection
- Definitions of injuries and poisonings
- Narrative introduction to the section
- Weighting of data
- Respondent vs. Proxy

- SOURCE: CDC/NCHS

**Notes:**
The factors we compared for this presentation is not a complete list.  There are a number of other items that need to be considered.  
Is this only an injury survey or is it part of a more comprehensive survey?  If it is part of a larger survey – what kind?  A health, crime, etc.
Be sure to consider the length of the survey and placement of most important questions near the beginning.  
Sample selection is also an important thing to analyze and compare. 
Defining injury and poisoning.  Having a clear understanding of what you want to measure and make sure this is understood by the interviewers and the interviewees.  Remove as much ambiguity as possible and provide examples as appropriate.  Make it clear that you do not want poison ivy or food poisoning.
Also the introduction to the section needs to be reviewed.  The section should be short, concise, and prepare the individual for the next series of questions.  
How is the data weighted?  What sort of estimates can be made from the data – Annual estimates?  National represented sample?  Is it a convenient sample?
Is the individual only responding for themselves or are they responding for other individual’s experience?

## National Household Injury Surveys

**Generic considerations**
- Continuous funding
- Strong lobbying
- Demonstrating usefulness of data

- SOURCE: CDC/NCHS

**Notes:**
It is important to make sure you have continuous funding in place and this often is best done by strong lobbying and demonstrating the usefulness of the data.

## Website: http://www.cdc.gov/nchs/about/otheract/ice/housesur.htm

## Considerations

- Should our goal be to compare our selves to each other to have better questions and survey methodology?
- Is it a worthwhile goal to standardize questions?
- Cause compatible with ICD?
- Should health behaviors be a priority?
- Is it even worth comparing numbers?