# Presentation: 495394

## Evaluating Health Information Technology:  A Primer

**Eric Poon, MD MPH**
**Clinical Informatics Research and Development,**
**Partners Information Systems**
**Davis Bu, MD MA**
**Center for Information Technology Leadership,**
**Partners Information Systems**
*AHRQ National Resource Center for Health Information Technology*

## Pre-Conference Logistics 

- To Access Slides:
- Go to http://extranet.ahrq.gov/rc
- Login with username and password
- Follow the links to download slides
- Problems? Email ResourceCenter@norc.org
- Q&A Session at the End
- Dial *1 to ask a question
- Please pick up handset (not speakerphone)
- Note that this teleconference is being recorded

## Outline

- Why evaluate?
- General Approach to Evaluation
- Deciding what to Measure
- Study Design Types
- Analytical issues in HIT evaluations
- Some practical advice on specific evaluation techniques

## Why Measure Impact of HIT?

- Impact of HIT often hard to predict
- Many “slam dunks” go awry
- Understand how to clear barriers to effective implementation
- Understand what works and what doesn’t
- Justify enormous investments
- Return on investment
- Allow other institutions to make tradeoffs intelligently
- Use results to win over late adopters
- You can’t manage/improve what isn’t measured
- Good publicity for organization

## General Approach to Evaluating HIT

- Understand your intervention
- Select meaningful measures
- Pick the study design
- Validate data collection methods
- Data analysis

## Getting Started: Get to know your intervention

- Clarify the question:  What problem does it address?
- Think about intermediate processes
- Identify potential barriers to successful implementation
- Identify potential managerial and behavioral process to overcome implementation barriers

**Notes:**
Clarify the research question: 
Why did the investigator want to implement the system in the first place?
What important lessons does the investigator think will be learned, that could help the general community
the evaluation plan really should answer the research question for which the study was conducted
It is possible that there are multiple research questions.  If so, consider a primary question and secondary question to narrow the list of primary vs. secondary measures.  This will have an effect on statistical analysis (e.g. Bonferoni corrections)
Specify intermediate processes:
There should by some hypothesis as to how the intervention will affect the environment.  How did the investigator think the HIT would make a difference?  By making these processes explicit, the universe of possible measures will become more clear
Identify associated measures:
Each step of the process above may have one or more associated measures
Some, but perhaps not all, of these measures may help answer the questions of interest
The toolkit may help generate ideas for both processes and measures
Select appropriate methodology
See Eric’s stuff
The details of the HIT to be implemented, the environment where it will be implemented, and the choice of measures will all influence the study design
Define evaluation strategy
Pull everything together at an appropriate level of detail

## Array of Measures

- Quality and Safety
- Clinical Outcomes
- Clinical Processes
- Knowledge
- Patient knowledge
- Provider knowledge
- Satisfaction
- Patient satisfaction
- Provider satisfaction

- Resource utilization
- Costs and charges
- LOS
- Employee time/workflow

## Introducing the Evaluation Toolkit

- Rough guides on general approach, costs and potential pitfalls
- Major domains:
- Clinical Outcomes
- Clinical Process
- Provider Adoption & Attitudes
- Measure Characteristics:
- IOM Domain
- Data Source
- Relative Cost
- Would love to hear your feedback

- Patient Knowledge & Attitudes
- Workflow Impact
-  Financial Impact
- Potential Pitfalls
- General Notes

## Selecting Evaluation Measures for HIT: Three Examples

## Computerized Provider Order Entry (CPOE) Example

- Clarify the primary question:
- Does CPOE improve quality of care?
- Competing questions:
- Does CPOE save money?
- What are the barriers to physician acceptance?
- Does CPOE introduce new errors?

**Notes:**
Present a couple of high level examples.  This is based on a slice of the work we did, but is intended to be neither comprehensive of CITL work, nor a definition of what is acceptable to AHRQ

In this case, we will look at CPOE. 
Many reasons why CPOE might be implemented
Many lessons to be drawn from this experience
In this hypothetical example, quality of care was the primary area of interest, and how CPOE affects care is our primary question
Many competing questions, such as financial impact.  How important are these other questions? How well can we evaluate these other questions? How would inclusions of these other questions affect our design? Our data analysis (e.g. Bonferoni correction, power calculations)
In this example, the investigators decided to focus on multiple measures of quality, and do a good job at those.  Other questions, while interesting, were deemed out of scope.

## CPOE: How can it affect quality?

- Think about intermediate processes
- Patient data is presented to ordering physician
- ADE alerts may be triggered and presented at the point of care (which alerts?)
- Guideline reminders may be triggered an presented at the point of care (which guidelines?)
- Medication order is entered
- Medication order is executed by pharmacy
- Medication order is executed by nursing staff

**Notes:**
Informed by an understanding of the existing body of research, clarified by an understanding of the specific details of the HIT and institution environment

Think through the process:
Investigators probably had some hypothesis for why CPOE would affect quality
Break down the entire process into manageable and understandable pieces
Walk through the steps in detail; don’t worry so much about whether they are measurable or not... treat that question separately
Having some detail here will help translating processes into measures, e.g. which alerts will be delivered will impact on what can be measured

Understanding the existing body of research would help refine these evaluation planning activities:
Multiple steps in the process of ordering and administering medications have been clarified
Multiple taxonomies of errors exist (e.g. omission vs. comission, “near miss” vs. event with harm vs. event with recovery, etc
Literature exists that has looked at many ways CPOE affects ordering behavior and potential error avoidance

## Does CPOE Improve Quality of Care?

- Identify measures

| Process | Measure |
| --- | --- |
| Data Presentation | (Redundant test ordering) |
| ADE Alert | Alert frequency, ADE frequency |
| Guideline Reminder | Guideline compliance, clinical outcome |
| Order Entry | Ordering errors |
| Pharmacy | (Time to process order) |
| Administration | Time to administration |

**Notes:**
Making the leap from process to measure is helped by understanding the literature:  evidence has shown that presentation of previous results, cost data, etc alters physician ordering behavior
Lots of alerting research and guideline research exists
Framework for medical error analysis exists, which provides insights into measures... many stages of medication ordering and administration

Additional notes:
Lots of additional specificity to be given: how do you define redundant tests, adverse drug events... which guidelines do you use to measure compliance
Some measures may not answer the question of interest, and have varying degrees of interest (e.g. time to process orders may not answer the clinical quality questions)
All measures have varying degrees of difficulty. E.g. cost, how much impact do you think the system will have (informed by research, prior experience, etc) and does the study have the power to detect this difference

## Evaluating CPOE’s Impact on Quality

- Select Appropriate Methodology
- Does existing data exist that can be leveraged? (e.g. ongoing QA activities)
- Does concurrent control exist?
- How will the data be analyzed?

## Electronic Medical Records (EMR) Example

- Clarify the primary question:
- What are the barriers and facilitators to effective EMR implementation?
- Competing questions:
- Do EMRs save money?
- Do EMRs improve quality of care?
- Do EMRs introduce new errors?

**Notes:**
Another example, to look at implementation barriers

Although many research questions exist regarding the nature and value of HIT, many other important lessons to be drawn regarding how to successfully implement HIT and realize value.  These implementation questions should not be overlooked; indeed, they are an important part of the AHRQ grants.

## EMR: Dissecting the EMR Implementation Process

- Identify stakeholders
- Providers, et al.
- Catalogue stakeholder interests and values
- Workflow efficiency
- Clarify stakeholder role in implementation
- Users of system, clinical leaders, administrative leaders
- Clarify impact of Implementation on clinical processes
- User interface optimization, workflow re-engineering
- Define implementation success criteria
- Provider buy-in, provider use and acceptance

**Notes:**
Key point: use common sense, and think through the process carefully

See Atif’s workflow transformation slides for additional ideas

Other resources and existing paradigms exist: e.g. continuous quality improvement, how to identify and measure critical success factors, utilization focused evaluation (an approach to effective evaluation)
Many process improvement methodologies are data driven, and can generate ideas for evaluation

## EMR: Understanding the Barriers and Facilitators to Implementation

- Identify measures

| Process | Measure |
| --- | --- |
| Stakeholder attitudes |  Attitude/Satisfaction surveys
 Readiness Assessment
 Staff Turnover |
| Workflow |  Efficiency metrics |
| Process improvements |  Staffing levels
 Patient flow
 Practice productivity
 Implementation participation from staff |
| Success Criteria |  Usage data
 Training attendance
 Measures listed above |

**Notes:**
Example looking at provider buy-in

Measures exist across the spectrum: stakeholder values; how processes impact on those values; how stakeholders ultimately react to those processes

## EMR: Understanding the Barriers and Facilitator to Implementation

- Select Appropriate Methodology
- Combination of quantitative and qualitative studies
- Example: efficiency measures:
- Time motion studies: how did the system affect provider efficiency?
- Attitude Surveys: How did the system affect provider perception of efficiency?
- Semi-structured interviews: How did the implementation affect stakeholder workflow? Did that effect change over time and why?

**Notes:**
Keep in mind how the results of the evaluation might be used... e.g. the relation of measures to process changes.  
e.g. time motion studies can look at changes in efficiency of each sub-process... surveys can be designed to trace outcome to process changes

I.e. what is the implementation doing -> what impact does the plan have?
     how is the grantee doing the implementation -> how does that impact on the success of implementation?

## Local Health Information Infrastructure (Laboratory)

- Clarify the primary question
- Can LHIIs for labs generate a positive ROI?
- Competing questions:
- Can LHIIs for labs improve quality of care?
- Which architecture is best suited for LHIIs for labs?
- How do LHIIs for labs affect provider and patient perception of the health care system?

**Notes:**
Again, many different questions may be asked.

In this example, the investigators are most concerned about creating a sustainable funding model for LHIIs, and want to understand the financial implications

## LHII (Laboratory): Defining the ROI

- Specify intermediate processes
- Data is pulled from local laboratories
- Previous labs pulled
- Lab order entered
- Lab order transmitted
- Administrative handling
- Lab results reported
- Lab results recorded
- Data is pulled from primary provider
- Authorization and payment is coordinated with payer
- Implementation of LHIO

**Notes:**
Again, understanding the series of steps is important

However, in this example, one higher level of analysis was needed.  LHIIs are all about data transfer, and each type of data transfer may have different processes.  So first, the investigator just mapped out the data transfers that occur

For each type, a set of processes must occur.  In this example, we will look at laboratory transactions that occur between the provider and laboratory facilities

## LHII (Laboratory): Defining the ROI

- Identify associated measures

| Process | Measure |
| --- | --- |
| Provider requests data | Volume of requests |
| Data is pulled | Chart pulls, time for chart pulls, administrative costs |
| Provider interprets data | Amount of missing information |
| Provider Orders test | Volume of redundant tests |

**Notes:**
Again, going from process to measures requires an understanding of the technology, environment and literature.

In this case, there is an additional framework at work: cost=volume * cost per unit

Volume:
Number of external transactions across the LHII
Number of internal transactions within an entity (e.g. chart pulls)
Number of laboratory tests ordered
Number of redundant laboratories ordered

Cost per unit
Time per transaction
Administrative cost per unit time
Administrative cost per transaction
Cost per laboratory test
System cost, both initial and ongoing

## LHII (Laboratory): Evaluating the ROI

- Select Appropriate Methodology
- Does concurrent control exist?
- Are there ongoing trends over time?
- How will the data be analyzed?

## Selecting Outcome Measures: General Comments

- Generally want to pick 1-3 outcomes of primary interest
- If choose more, need to make correction (e.g. Bonferroni)
- Outcome must be sufficiently frequent to be detectable
- Rare events such as adverse events due to errors particularly challenging
- Important enough to provoke interest
- Whether study is positive or negative
- How would the results change policy (local or national)?
- Process vs. outcome
- Legitimate to measure process
- Outcome often takes too long
- In many situations link between process, outcome clear

## Study Types

- Commonly used study types:
- Before-and-after time series Trials
- Randomized Controlled Trials
- Factorial Design
- Study design often influenced by implementation plan

## Time Series vs. Randomized Controlled Trials

- Before-and-after trial common in informatics
- Concurrent randomization is hard
- Don’t lose the opportunity to collect baseline data!
- Off-On-Off trial design possible
- But may not be politically/ethically acceptable to turn off a highly used feature
- RCT preferable if feasible
- Eliminates the issue of secular trend
- Balance of baseline confounding

## Randomization Considerations

- Justifiable to have a control arm as long as benefit not already demonstrated (usual care)
- Want to choose a truly random variable
- Not day of the week
- Legitimate to stratify on baseline variables (e.g. education for pt, computer experience for providers)
- Minimal number of arms
- More arms, less power
- Strongest possible intervention

## Unit of Randomization

- Patients
- Physicians
- Practices/wards

## Randomization Unit:How to Decide?

- Small units (patients) vs. Large units (practices wards)
- Contamination across randomization units
- If risk of contamination is significant, consider larger units
- Effect contamination-can underestimate impact
- However, if you see a difference, impact is present
- Randomization by patient generally undesirable
- Contamination
- Ethical concern

**Notes:**
Examples of contamination:
A single physician getting alerts for some patients but not others
Practice management meetings
Float nurses

## Randomization Schemes:Simple RCT

- Burn-in period
- Give target population time to get used to new intervention
- Data not used in final analysis

## Randomization schemes: Factorial Design

- May be used to concurrently evaluate more than one intervention:
- Assess interventions independently and in combination
- Loss of statistical power
- Usually not practical for more than 2 interventions

## Randomization Schemes:Staggered Deployment

- Advantages
-  Easier for user education and training
-  Can fix IT problems up front
- Need to account for secular trend
-  Time variable in regression analysis

## Randomization Schemes:Multiple Interventions

- Time efficient design
- Every clinic gets something. (Keeps clinics and IRB happy)
- Watch out for cross-arm intervention contamination

## Inherent Limitations of RCTs in Informatics

- Blinding is seldom possible
- Effect on documentation vs. clinical action
- People always question generalizability
- Success is highly implementation independent
- Efficacy-effectiveness gap: ‘Invented here’ effect

## Data Collection

- Electronic data abstraction
- Convenient and time-saving, but...
- Some chart review (selected) to get information not available electronically
- Get ready for nasty surprises
- Pilot your data collection protocol early
- And then pilot some more...

## Data Collection Issue: Baseline Differences

- Randomization schemes often lead to imbalance between intervention and control arms:
-  Need to collect baseline data and adjust for baseline differences
-  Interaction term ( Time * Allocation Arm) gives effect for intervention in regression analysis

## Data Collection Issue: Completeness of Followup

- The higher the better:
- Over 90%
- 80-90%
- Less than 80%
- Intention to treat analysis
- In an RCT, should analyze outcomes according to the original randomization assignment

## A Common Analytical Issue The Clustering Effect

- Occurs when your observations are not independent:
- Example: Each physician treats multiple patients:

**Intervention Group**

**Control Group**

**Physicians**

**Patient -> Outcome assessed**

## Options for Dealing with the Clustering Effect

- Analyze at the level of clinician
- Example: Analyze % of MD’s patients in compliance with guideline, and make MD unit of analysis
- Huge drop in statistical power.
- Not recommended.
- Generalized Estimating Equations
- PROC GENMOD in SAS, or PROC RLOGIST in SUDAAN
- Allows you to randomize at one level (e.g. physician) and then do analysis at another (e.g. patient)
- Accounts for correlation of behaviors within a single physician (i.e. adjusts for the fact that observations across patients are NOT independent)

## A Word About Surveys

- Survey of user believes, attitude and behaviors
- Response rate – responder bias: Aim for response rate > 50-60%
- Keep the survey concise
- Pilot survey for readability and clarity
- Need formal validation if you want plan to develop a scale

## Looking at Usage Data

- Great way to tell how well the intervention is going
- Target your trouble-shooting efforts
- In terms of evaluating HIT:
- Correlate usage to implementation/training strategy
- Correlate usage to stakeholder characteristics
- Correlate usage to improved outcome

## Studies on Workflow and Usability

- How to make observations?
- Direct observations
- Stimulated observations
- Random paging method
- Subjects must be motivated and cooperative
- Usability Lab
- What to look for?
- Time to accomplish specific tasks:
- Need to pre-classify activities
- Handheld/Tablet PC tools may be very helpful
- Workflow analysis
- Asking users to ‘think aloud’
- Unintended consequences of HIT

## Qualitative Methodologies

- Major techniques
- Direct observations
- Semi-structured interviews
- Focus groups
- Adds richness to the evaluation
- Explains successes and failures. Generate Lessons learned
- Captures the unexpected
- Great for forming hypotheses
- People love to hear stories
- Data analysis
- Goal is to make sense of your observations
- Iterative & interactive

## Cost Benefit Analysis

- Cost Data
- Generally available
- Caveat: allocation of indirect costs
- Financial Benefit Data
- Revenue Enhancement
- Cost Avoidance
- Benefit Allocation
- Benefits may accrue to multiple parties
- Are benefits realizable (e.g. labor savings)?
- Calculation of benefits to external parties may be of interest, even if it does not impact on ROI

**Notes:**
Cost data:
Most notable pitfall is allocation of indirect costs

Benefit data: think through the process as we did above
Think through the non-technical pieces as well, e.g. process re-engineering (a big driver in PACS implementations)
Consider implications on the revenue cycle: billings, collections, accounts receivable, etc.  May be non-intuitive for clinicians
See toolkit for ideas

Benefit allocation:
Clarification of benefit to third parties may be useful, as the Nation considers forging partnerships, public dollar expenditures, etc

## Cost Benefit Analysis

- Activity Based Costing Example
- Simply put, a method for assigning costs to particular activities
- Alternate method of assigning indirect costs to the project
- Also, may serve as a framework for capturing cost savings

| Step* | Example |
| --- | --- |
| Identify activities |  Paper chart maintenance |
| Determine cost for each activity |  Cost data for medical records |
| Determine cost drivers |  Number of chart pulls |
| Obtain activity data |  How many charts were pulled |
| Calculate total cost |  Savings from decreased pulls |

- * http://www.pitt.edu/~roztocki/abc/abctutor/

**Notes:**
This should look familiar, because the LHII example is based on this

## Concluding Remarks

- Don’t bite off more than what you can chew
- Pick a few study outcomes and study them well.
- It’s a practical world
- Balancing operational and research needs is always a challenge.
- Life (data collection) is like a box of chocolates...
- You don’t know what you’re going to get until you look, so look early!

**Notes:**

While both immediate, individual and retrospective, grouped provision of information have roles in affecting decision-making, the former will be more important

## Thank you

- Eric Poon, MD MPH
- Email: _[epoon@partners.org](mailto:epoon@partners.org)_
- Davis Bu, MD MA
- Email: _[dbu@partners.org](mailto:dbu@partners.org)__ _

## Give Us Feedback! 

- We are eager to hear your feedback!
- Go to http://extranet.ahrq.gov/rc
- Login with username and password
- Follow the links to provide feedback-thanks!
- Want to hear this teleconference again?
- Dial 1-800-486-4195 to replay until 5/4/05