# Presentation: 495391

## Evaluating Health Information Technology: Putting Theory Into Practice

**Eric Poon, MD MPH**
- Clinical Informatics Research and Development,
- Partners Information Systems
** ****David F. Lobach, MD, PhD, MS**
- Division of Clinical Informatics
- Department of Community and Family Medicine
- Duke University Medical Center, Durham, North Carolina
*AHRQ’s National Resource Center for Health Information Technology*
*Annual Meeting June 2005*

## Outline

- Overview of Evaluating HIT
- Why evaluate?
- General Approach to Evaluation
- Choosing Evaluation Measures
- Study Design Types
- Analytical issues in HIT evaluations
- Evaluation in the ‘real world’
- Duke University Medical Center

## Why Measure Impact of HIT?

- Impact of HIT often hard to predict
- Many “slam dunks” go awry
- You can’t manage/improve what isn’t measured
- Understand how to clear barriers to effective implementation
- Understand what works and what doesn’t
- Invent the wheel only once
- Justify enormous investments
- Return on investment
- Allow other institutions to make tradeoffs intelligently
- Use results to win over late adopters

**Notes:**
Good publicity for organization

## General Approach to Evaluating HIT

- Understand your intervention
- Formulate questions to answer
- Select and define measures
- Pick the study design
- Data analysis

## Getting Started: Get to know your intervention

- What problem(s) is it trying to solve?
- Think about intermediate processes
- Identify potential barriers to successful implementation:
- Managerial barriers
- End-user behavioral barriers
- Understand how your peers around the country are addressing (or not) the same issues.

**Notes:**
Clarify the research question: 
Why did the investigator want to implement the system in the first place?
What important lessons does the investigator think will be learned, that could help the general community
the evaluation plan really should answer the research question for which the study was conducted
It is possible that there are multiple research questions.  If so, consider a primary question and secondary question to narrow the list of primary vs. secondary measures.  This will have an effect on statistical analysis (e.g. Bonferoni corrections)
Specify intermediate processes:
There should by some hypothesis as to how the intervention will affect the environment.  How did the investigator think the HIT would make a difference?  By making these processes explicit, the universe of possible measures will become more clear
Identify associated measures:
Each step of the process above may have one or more associated measures
Some, but perhaps not all, of these measures may help answer the questions of interest
The toolkit may help generate ideas for both processes and measures
Select appropriate methodology
See Eric’s stuff
The details of the HIT to be implemented, the environment where it will be implemented, and the choice of measures will all influence the study design
Define evaluation strategy
Pull everything together at an appropriate level of detail

## Formulating Questions

- Likely questions:
- Does the HIT work?
- What would have made it work better?
- What would the next set of designers/implementors like to know?
- Has this question been fully answered before?
- Don’t reinvent the wheel! (not a big concern)
- What impact would the answer have?
- Peers
- Policy makers

## Array of Measures

- Quality and Safety
- Clinical Outcomes
- Clinical Processes
- Knowledge
- Patient
- Provider
- Satisfaction & Attitudes
- Patient
- Provider

- Resource utilization
- Costs and charges
- LOS
- Employee time/workflow
- Lessons learned

## Choosing Study Measures

- Clinical vs Process Measures
- Clinical outcomes (e.g. mortality) desirable
- Justifiable to measure process outcomes (e.g. door to abx time) if relationship between outcome and process already demonstrated
- Will outcomes be impacted by the intervention?
- Will impact on outcomes be detectable during the study period?
- ? Rare events, e.g. adverse outcomes
- ? Colon cancer screening
- What resources do you have?
- Don’t bit off more than what you can chew.

## Selecting Study Types

- Commonly used study types:
- Optimal design: Randomized Controlled Trials
- Factorial Design
- Before-and-after time series Trials
- Main study design issues:
- Secular Trend: Can a simultaneous control group be established?
- Confounding: Can you randomly assign individuals to study groups?
- Study design often influenced by implementation plan
- Need to respect operational needs, but often there is room for creative designs

## Randomization Nuts and Bolts

- Justifiable to have a control arm as long as benefit not already demonstrated (usual care)
- Want to choose a truly random variable
- Not day of the week
- Consideration: Stratified randomization
- Ensures that intervention and control group are similar on important characteristics (e.g. baseline computer literacy)
- Strongest possible intervention

## Randomization Unit:How to Decide?

- Small units (patients) vs. Large units (practices wards)
- Contamination across randomization units
- If risk of contamination is significant, consider larger units
- Effect contamination-can underestimate impact
- However, if you see a difference, impact is present
- Randomization by patient generally undesirable
- Contamination
- Ethical concern

**Notes:**
Examples of contamination:
A single physician getting alerts for some patients but not others
Practice management meetings
Float nurses

## Randomization Schemes:Simple RCT

- Burn-in period
- Give target population time to get used to new intervention
- Data not used in final analysis

## Randomization schemes: Factorial Design

- May be used to concurrently evaluate more than one intervention:
- Assess interventions independently and in combination
- Loss of statistical power
- Usually not practical for more than 2 interventions

## Randomization Schemes:Staggered Deployment

- Advantages of staggering
- Easier for user education and training
- Can fix IT problems up front
- Need to account for secular trend and baseline differences
- Time variable in regression analysis
- Control for practice characteristics

## Inherent Limitations of RCTs in Informatics

- Blinding is seldom possible
- Effect on documentation vs. clinical action
- People always question generalizability
- Success is highly implementation independent
- Efficacy-effectiveness gap: ‘Invented here’ effect

## Mitigating the Limitations of Before-and-After Study Designs

- Before-and-after trial common in informatics
- Concurrent randomization is hard
- Don’t lose the opportunity to collect baseline data!
- Leave the time gap between before and after trends relatively short
- Look for secular trend in statistical analysis and adjust for it if present

## Common Pitfalls with Data Collection

- Measures you define and collect on your own
- Pilot data collection and refine definition *early*
- Ask yourself *early* whether data your collect measure what you intended to measure.
- Measures others defined but you collect on your own
-  Do you need to adapt other people’s instruments?
- Measures others define and collect for you
- Understand nuisances and limitations, particular with administrative data.

**Notes:**
Appropriate antibiotic prescribing.
	- What is an antibiotic?
	- What is appropriate? 
		Which guidelines are you going to choose?  
		What about local accepted behavior?
	- Where are you going to the data from and how much will you trust it?

Another example:
	Medication administration error study -

## Electronic Data Abstraction: There’s no free lunch!

- Convenient and time-saving, but...
- Some chart review (selected) to get information not available electronically
- Get ready for surprises
- Documentation effect of EMRs

## Data Collection Issue: Baseline Differences

- Randomization schemes often lead to imbalance between intervention and control arms:
-  Need to collect baseline data and adjust for baseline differences
-  Interaction term ( Time * Allocation Arm) gives effect for intervention in regression analysis

## Data Collection Issue: Completeness of Followup

- The higher the better:
- Over 90%
- 80-90%
- Less than 80%
- Intention to treat analysis
- In an RCT, should analyze outcomes according to the original randomization assignment

## A Common Analytical Issue The Clustering Effect

- Occurs when your observations are not independent:
- Example: Each physician treats multiple patients:
- May need to increase sample size to account for loss of power.

**Intervention Group**

**Control Group**

**Physicians**

**Patient -> Outcome assessed**

## Looking at Usage Data

- Great way to tell how well the intervention is going
- Target your trouble-shooting efforts
- In terms of evaluating HIT:
- Correlate usage to implementation/training strategy
- Correlate usage to stakeholder characteristics
- Correlate usage to improved outcome

## Studies on Workflow and Usability

- How to make observations?
- Direct observations
- Stimulated observations
- Random paging method
- Subjects must be motivated and cooperative
- Usability Lab
- What to look for?
- Time to accomplish specific tasks:
- Need to pre-classify activities
- Handheld/Tablet PC tools may be very helpful
- Workflow analysis
- Asking users to ‘think aloud’
- Unintended consequences of HIT

## Cost Benefit Analysis

- Do the benefits of the technology justify the costs?
- Monetary benefits – Monetary costs
- Important in the policy realm
- Need to specify perspective
- Organizational
- Societal
- Cost analysis more straight forward
- Prospective data collection preferred
- Discounting: a dollar spent today worth more than a dollar 10 years from now
- Benefits analysis more controversial
- Cost of illness averted: medical costs, productivity for patient
- What is the cost of suffering due to preventable adverse events?
- What is the cost of a life?

**Notes:**
Cost data:
Most notable pitfall is allocation of indirect costs

Benefit data: think through the process as we did above
Think through the non-technical pieces as well, e.g. process re-engineering (a big driver in PACS implementations)
Consider implications on the revenue cycle: billings, collections, accounts receivable, etc.  May be non-intuitive for clinicians
See toolkit for ideas

Benefit allocation:
Clarification of benefit to third parties may be useful, as the Nation considers forging partnerships, public dollar expenditures, etc

## Using Surveys – Stay Tuned!

- Survey of user believes, attitude and behaviors
- Response rate – responder bias: Aim for response rate > 50-60%
- Keep the survey concise
- Pilot survey for readability and clarity
- Need formal validation if you want plan to develop a scale/summary score

## Qualitative Methodologies – Don’t touch that dial!

- Major techniques
- Direct observations
- Semi-structured interviews
- Focus groups
- Adds richness to the evaluation
- Explains successes and failures. Generate Lessons learned
- Captures the unexpected
- Great for forming hypotheses
- People love to hear stories
- Data analysis
- Goal is to make sense of your observations
- Iterative & interactive

## Concluding Remarks

- Don’t bite off more than what you can chew
- Pick a few study outcomes and study them well.
- It’s a practical world
- Balancing operational and research needs is always a challenge.
- Life (data collection) is like a box of chocolates...
- You don’t know what you’re going to get until you look, so look early!

**Notes:**

While both immediate, individual and retrospective, grouped provision of information have roles in affecting decision-making, the former will be more important

## Thank you

- Eric Poon, MD MPH
- Email: _[epoon@partners.org](mailto:epoon@partners.org)_
- Acknowledgements
- Davis Bu, MD MA
- CITL, Partners Healthcare
- David Bates, MD MSc
- Chief, Div of General Medicine, Brigham and Women’s Hospital