# Presentation: 799740

## Evaluating California’s Web-based Training for Medical Assistants

**Evaluating California’s ****Web-based Training for ****Medical Assistants**

**Leslie Heyden, MA**
- California Department of Public Health
- Immunization Branch
- Information and Education Section
- February 19, 2009

## Overview

**The purpose of the CDPH IZ Branch evaluation is to assess the use and effectiveness of the first two immunization skills elearning lessons:**** **
*Preparing Vaccines*
*Administering Vaccines*
**The elearning is custom developed, interactive web-based training.**
**The primary target audience is medical assistants in California VFC provider practices.**
**The lessons are available via eziz.org.**

## eziz.org

**The new California VFC program website**
**A one-stop shop for California VFC providers and their staffs**
**A repository of a wide variety of immunization skills training and VFC program resources, including:**
- VFC forms (e.g., vaccine order and recertification forms, vaccine temperature and usage logs, patient eligibility screening forms)
- Job aids, guidelines, posters, and up-to-the-minute news about vaccine availability

## eziz.org—home page

## Webinar learning objectives

**By the end of this webinar you will be able to:**
**List the objectives of the evaluation plan**
**Explain why CDPH IZ branch selected the EZ-IZ elearning lessons to evaluate**
**Describe the methodology CDPH developed to evaluate the training**
- Indicators
- Targets
- Methods, including data sources
**Describe lessons learned**

## Why evaluate EZ-IZ’ elearning?

**Is IZ Branch’s first elearning project**
**Has potential for high impact re: ability to reach unlimited number of learners with limited resources**
**Primary tool needed to conduct evaluation (LMS) was already being created**
**IZ Branch wanted to compare the ****“reach” of the elearning to existing ****classroom training**

## Goal of activity to be evaluated

**To educate MAs in California VFC provider practices about best practices for vaccine preparation and administration**

## Evaluation objectives

**By September 5, 2008 the first two elearning lessons of the EZ-IZ curriculum (*****Preparing Vaccines***** and *****Administering Vaccines*****) will launch, and the learning management system will begin tracking usage and learner performance.**
**By March 31, 2009, at least one person from 5% (approximately 180) of California’s active VFC provider sites (approximately 3,600) will have completed one of the elearning lessons (either***** Preparing Vaccines***** or *****Administering Vaccines*****) and have passed its posttest.**

## Evaluation objectives (cont’d)

**By September 5, 2009, at least one person from 20% (approximately 720) of active California VFC provider sites will have completed the *****Preparing Vaccines***** and *****Administering Vaccines***** elearning lessons and passed the posttest for each.**

## Evaluation objectives (cont’d)

**By November 30, 2008 all user feedback about the EZ-IZ website will have been gathered and analyzed, and recommendations made.**
- Recommendations concerning website usability will be forwarded to and resolved by the web developer and/or the web editor.
- Recommendations concerning the EZ-IZ Curriculum and the lessons in it will be forwarded to and resolved by the instructional designer.
- Recommendations about programming and usability issues within the elearning lessons will be forwarded to and resolved by the elearning vendor.
- Recommendations concerning the learning management system (LMS) will be forwarded to and resolved by the LMS vendor.

## Stakeholders

**Management**
- John Talarico,  Interim CDPH IZ Branch Chief
- Tiffany Sutter,  Section Chief: Information & Education
- Natalie Nakahara, Project Manager and Creative Director
**California VFC Program**
- Claudia Aguiluz, VFC Program Coordinator
- VFC field staff and field services reps
- VFC provider staff (particularly MAs who work in private provider practices)

## Stakeholders (cont’d)

**Immunization (federal, state, and local)**
- CDC (NCIRD)
- Steve Nickell,  Research Coordinator, IZ Branch
- Sandra Jo Hammer, Nurse Consultant, IZ Branch
- CDPH IZ Branch Evaluation team (Leslie Heyden, Natalie Nakahara, and Steve Nickell)
- IZ Coordinators in LHDs
**Others**
- California Medical Assistants Association (CMAA) and California medical assistant training programs

## Table 3: Program Component/Activity Description 

**Table 3: Program Component/Activity Description **

- Funding
- Staff
- Time

- Review available materials & choose content
- Design training
- Employ vendor to program elearning and create learning management system
- Make training available via website
- Promote training

**Inputs**

**Activities**Initial               Subsequent

**Outputs**

**Outcomes**Short term     Long term

- Review of content by content experts
- Beta test on target audience
- Collect & assemble feedback
- Decide on changes
- Revise training
- Create a “one-stop shop” learning and resource portal for CA VFC providers, immunizers and other staff with clinical or administrative immunization-related responsibilities

- (Live) online elearning lessons, with learning management system tracking

- VFC providers will adapt to elearning as a training delivery  method
- Increased  skill and knowledge (pre- posttest comparison) about vaccination among medical assistants & other office staff
- Steady increase in the number of MAs who take the elearning lessons

- Fewer real-time “person hours” required to train MAs
- Fewer vaccine administration errors
- Higher coverage rates
- Fewer missed opportunities
- Fewer VPDs
- Increased provider/ user satisfaction

## Primary data source—EZ-IZ LMS

**An LMS (****Learning Management System)**** is software that automates the administration of training. **
**The LMS registers users, tracks courses in a curriculum, and records data from learners**
**The LMS provides reports to management.**

## Evaluation question #1 

**Q: Did the *****Preparing Vaccines***** and *****Administering Vaccines***** elearning lessons launch by September 5, 2008 (target)? **
**Indicator and method**
- The *Preparing Vaccines* and *Administering Vaccines* elearning lessons launch: request IZ branch staff to access lessons and report any problems accessing the lessons
**Data source: feedback (verbal and email) from IZ Branch staff (and confirmation from LMS vendor)**

## Evaluation question #2

**Q: Did the learning management system (LMS) begin tracking usage and learner performance data by September 5, 2008 (target)? **
**Indicators and methods**
- LMS is operational: verify with vendor that LMS is operational
- LMS is gathering usage data: obtain registration and login data from LMS report
- LMS is gathering learner performance data**: **obtain pretest, lesson start, lesson completion, and posttest data from LMS report

## Data source—registrationdata

**Data source—****registration****data**

## Learning History Page

**Learning History Page**

## Data source—LMS

**Pre-test**

**Lesson**

**Post-test**

## Evaluation question #3

**Q: How many provider sites have at least one person who has completed one lesson and passed the posttest?**
**Target and indicator**
- 5% of provider sites have at least one person who has completed one lesson and passed its posttest by March 31, 2009
**Method**
- Generate LMS report of provider sites that shows lessons completed and posttest scores by provider site and user
**Data source: LMS**

## Data source—pre- and posttests

## Evaluation question #4

**Q: How many provider sites have at least one person who has completed one lesson and passed the posttest?**
**Target and indicator**
- 20% of provider sites have at least one person who has completed one lesson and passed its posttest by September 5, 2009 (one year after launch)
**Method**
- Generate LMS report of provider sites that shows lessons completed and posttest scores by provider site and user
**Data source: LMS**

## Evaluation question #5

**Q: What feedback has been received from EZ-IZ users that could indicate potential obstacles to accessing and completing the elearning lessons? **
**Targets and indicators**
- 100% of  emails and 100% of all other feedback that could indicate technical problems with the website
- 100% of  emails and 100% of all other feedback that could indicate technical or any other problems with the lessons
**Method**
- Emails about technical problems and overall feedback about the EZIZ website will be automatically sent to a specific (ezig.org) MS Outlook folder; phone calls will be logged and categorized by November 30, 2008
**Data source: emails submitted by users via links on eziz.org**

## Data source—feedback from users

## Development of evaluation plan—Challenges

**Very few challenges. IZ Branch Research Coordinator collaborated on and provided very valuable input into plan.**
**Due to limited resources, not able to build as robust an LMS as originally planned.**
**Proposed revisions to methodology during plan development required consulting with LMS vendor who then had to modify LMS database or do additional programming.**

## Lessons learned

**Did not add question about MA’s experience; could not separate primary target audience from secondary target audience.**
**Learned that more data was needed than provided by LMS. Added question to QAR for field reps to administer.**
**The project team learned a lot during this entire pilot effort (including the evaluation plan) and is applying lessons learned during the current development of the next two lessons.**

## Added question to QAR

| 1. Have you or anyone else in your practice completed either of the interactive training lessons (Preparing Vaccines or Administering Vaccines) at EZIZ.ORG, the new VFC website? | YES☐  NO☐
 |
| --- | --- |
| 2. If NO, what is the main reason? | ☐Haven’t had time  ☐Didn’t know about them  ☐ Couldn’t figure out how to do it   ☐Not allowed to use computer ☐ Office has no internet access              ☐ No interest, don’t need to      ☐There were technical problems with the site   ☐ Other? specify:______________						

 |

## Questions?

## Leslie HeydenHealth Educator & Instructional DesignerLeslie.Heyden@cdph.ca.gov(510) 620-3754www.eziz.org

## This document:

**This document can be found on the CDC website at:**
_**[http://www.cdc.gov/vaccines/programs/progeval/downloads/cali-eval-02-09.ppt](http://www.cdc.gov/vaccines/programs/progeval/downloads/cali-eval-02-09.ppt)**__** **_