# Presentation: 609857

## Injury Indicators Group(ICEIInG)

*Colin Cryer*
*CHSS, University of Kent, England*
*on behalf of ICEIInG*

## Agenda

- Where we got to at the last meeting
- who joined ICEIInG (page 1)
- Post meeting thoughts
- A proposal of work
- Progress since Washington 2001
- Revisit the validation criteria
- Discussion

## The last meeting....

- Presentation of the work of ICEIInG
- the validation criteria
- Discussion by ICE
- Workshop discussion
- Presentation and further discussion
- Revised validation criteria

## Criteria - slide 1

- (A) The indicator should reflect the occurrence of injury satisfying some case definition of anatomical or physiological damage.
- (B) The indicator should reflect a well defined information objective.
- (C) It should be possible to use existing data systems, or it should be practical to develop new systems, to provide data for computing the indicator.

## Criteria - slide 2

- (D) The probability of a case being ascertained should be independent of social, economic, and demographic factors, as well as health service supply and access factors.
- (E) The indicator should be derived from data that are inclusive or representative of the target population that the indicator aims to reflect.
- [Note: the intention is that (E) also implies not going beyond the bounds of the target population]

## Criteria - slide 3

- (F) The indicator should be based on events that are associated with significantly increased risk of impairment, functional limitation, disability, or death, decreased quality of life, or increased cost.
- (G) The indicator should be fully specified to allow calculation to be consistent at any place and at any time.

**Notes:**
These have been reproduced on the first part of page 2 of the handout.

You will recall that there were differences of opinion over whether or not criterion F should be included. two views were expressed:
1. That different applications of the validation criteria will require different subsets of the criteria to be used (eg. different subsets used when considering indicators for priority setting compared with when considering indicators that are the basis of national targets.
2. That whatever the application, the more criteria that are satisfied, the better the indicator is likely to be.

## Other findings from our discussions: 

- Definition of an indicator needed
- Precise definition of an injury needs to be included
- We need some statement about standardisation / comparability / harmonisation.
- Criteria should be split into >1 dimension, eg.
- quality
- usefulness
- practicality of measurement
- Hierarchy of criteria needed
- how much fuzziness can we tolerate

**Notes:**
Definition of an indicator is needed - I had not found a definition; however for the draft report of our work I produced one based on the dictionary definition of an indicator. Subsequently, I have come across a definition quoted by Rod McClure from an Australian Commonwealth document.
Precise definition of an injury needs to be included - I have addressed this by indicating that any specification of an indicator should include a precise definition.
We need some statement about standardisation / comparability / harmonisation - this was in order to ensure that indicators compared between countries mean something, or to ensure that time series mean something.
Criteria should be split into >1 dimension... Particularly if the number of criteria grows, these need to be grouped under sensible headings to aid use.
Hierarchy of criteria are needed - since it was felt by some that greater weight should be attributed to some criteria over other criteria.

## A walk in the cherry blossoms ...

- Post-meeting thoughts
- there is much work to be done
- much could be done in parallel
- I can take some work further
- Others are better placed to take other work forward
- eg. development of indicators for priority setting
- Mechanism needed:
- Encourage greater participation - all work to our strengths

**Notes:**
During the Washington meeting, we were encouraged to go into central Washington to see the spectacular cherry blossoms. I am glad I di, since not only did I thoroughly enjoy the sights, my walk amongst the cherry blossom allowed me to collect my thoughts from the 2 days. The meeting highlighted how much work there is to do, and I guess this was gnawing at me during this period on my own. The  thoughts that came to me to deal with this were that
much of the work can be done in parallel
I can take some work forward
others are better placed to take other work forward
.....

## Proposal put to ICEIInG

- The work of the group be addressed in discrete projects
- clear achievable objectives
- outputs
- papers (peer review journals / conferences)
- ICEIInG report
- collection of papers, with introduction and linking statements

## Method of working

- Those with interest / expertise - encouraged to take the initiative.
- This PI email a proposal to ICEIInG
- Comments from ICEIInG members
- if major concerns, not embraced by ICE
- Collaborators take the work forward
- keep ICEIInG informed periodically
- Publication
- author list + “...on behalf of the ICEIInG”

**Notes:**
This proposal is captured on the email on page 3 of the handout.

## A proposal of work

- Use the validation criteria to assess national indicators used in setting road safety targets
- Canada
- New Zealand
- United Kingdom
- Indicators and how criteria used
- handout

**Notes:**
First bullet point then...

The particular indicators that were considered in this work are shown at the bottom of page 4 of the handout.

The 6 validation criteria that were used to assess the indicators are shown in the middle of page 4.

The validation criteria were scored on a 0-10 scale by 3 of the team - Susan Mackenzie, John Langley and Steve Jarvis (University of Newcastle, England). On this scale:
0 = validation criterion not satisfied
10= validation criterion completely satisfied
1-9 = validation criterion incompletely satisfied - the magnitude of the score indicating the degree to which the criterion is satisfied.

The results of this part of the work is shown here also.

## Trends in 4 NZ road safety indicators

**Notes:**
Failure to satisfy the validation criteria matters – trends in injury rates depends on the indicator used.
As an illustration, this slide shows trends in 4 New Zealand road safety indicators. The dark blue line is the trend for indicator NZ3: reported injuries resulting from motor vehicle traffic crashes (MVTCs) per 100,000 people. The red line is the trend for NZ4: the number of hospitalised people  for injuries resulting from MVTCs. Neither of these indicators scored well on the validation criteria. The green and the crimson lines show the age-adjusted hospitalisation rates resulting from MVTCs for injuries with maximum AIS severity of 3 or more, and of 4 or more respectively. These latter 2 indicators pretty much satisfy all of our criteria.

It can be seen that the trend lines are different. This matters. If the target was the reduction of non-fatal injury by 10% between 1988 and 1995, the less valid current indicators would suggest that the target had been met, whereas the more valid AIS-based indicators would suggest otherwise. Your conclusions depend on the choice of indicator – and our choice should be valid indicators.

## Discussion of the criteria 

- (1) Case definition - high variability UK1/2
- (2) Serious outcomes - variability
- (3) Ascertainment - OK, but variable UK1
- (4) Inclusive - OK
- (5) Source data - N/a
- (6) Fully specified - N/a

**Notes:**
Another view of this work is to ask how well the criteria have performed.
1. There is a great deal of variation in the ratings for the UK indicators. One rater give high marks on the basis of the existence of a case definition, another rated as 0 on the basis that part of the definition was based on utilisation of service or reports to the police, and the final rater gave mid-range marks to reflect equal weightings to the part of the definition based on diagnosis and the part based on utilisation / reports.
2. There was quite a lot of inconsistency of assessment across all indicators
3. For the third criterion dealing with Ascertainment, the ratings were reasonably consistent with the exception of UK1 - the reason for this is unclear.
4. The fourth criterion dealing with inclusiveness of the target population, again assessments were reasonably consistent across all indicators
5. Consideration of the consistency of assessment did not seem appropriate for criteria (5) and (6). (5) deals with the existence of data to derive the indicator - and this is always  the case for existing indicators. (6) is not applicable since for non of the indicators was a specification available so this should have attracted a low score.
As  a result of the inconsistencies found, it was proposed that documentation be developed and tested with an aim of increasing consistency.

## Proposed additional criteria - for discussion

- (H) The indicator should measure accurately the entity that it aims to reflect. [cf (B)]
- (I) The indicator should be derived from data that is complete and is accurately coded [cf (D)].
- (J) There should have been no substantial changes / differences in coding frames or in coding practice between place or over time (ie. comparability etc.)
- (K) A sound indicator should reflect the occurrence of injuries across personal characteristics, place and time [cf (D), (E)].

**Notes:**
Finally, the following additional validation criteria were proposed to deal with some of the apparent deficiencies in the criteria that were applied to the road safety indicators....
(H)... This arose from consideration of NZ1-NZ3, each of which are based on the same numerators, but have different denominators. They appear to aim to measure different population parameters or entities. Generally, a goal is to define indicators that  measure explicit population parameters or entities.
(I) & (J) come from a recognition that quality and stability of data sources affects the accuracy of indicators.
(K) .... There was a concern when assessing the road safety indicators that non-coding related changes could, for example change over time that effects the performance of the indicator. For example patterns of reporting to the police could changeover time or differ from place to place that would compromise the indicator. This proposed new criteria is related to criteria (D) and (E) and so some rationalisation of these criteria should be necessary. 

## For discussion...?

- Proposed new criteria, or further comments on the old ones.
- Comments / questions on the assessment of road safety indicators
- The need for other country demonstrations?
- Describe related work that you have done.