# Presentation: 089030

## Session 3: Assessment & Evaluation

- Lisa Jacques-Carroll, MSW
- NCIRD, CDC

**Notes:**
Hello.  Today I will discuss methods for assessing and evaluating perinatal hepatitis B prevention programs.

## Learning Objectives

- Describe key components of evaluation of a perinatal hepatitis B prevention program
- describe different methods for assessing and evaluating a perinatal hepatitis B prevention program
- identify several tools that are available

**Notes:**
The learning objective for this session is to describe the key components of evaluation of a perinatal hepatitis B prevention program.  Specifically, you should be able to describe different methods for assessing and evaluating a perinatal hepatitis B prevention program AND identify several tools that are available to you to assist with program assessment and evaluation.

## Assessment & Evaluation

- Program evaluation vs. program monitoring or assessment
- The focus of this session is on assessing performance measures, which is a component of program evaluation
- For more information on program evaluation see:
- 	_www.cdc.gov/vaccines/programs/progeval/_

**Notes:**
Before we begin, I would like to take a moment to talk about the differences between program assessment and program evaluation.  Program evaluation involves the systematic application of scientific methods to measure the implementation (or process evaluation) and short or long term outcomes (or outcome evaluation) of a program or intervention.  Program evaluation can take significant resources and time.

We don’t all have the time or money for formal program evaluation, but that doesn’t mean you can’t apply some of the techniques or procedures.  We’ll be talking today about program monitoring or assessment – which is the less formal application of these techniques.  Programs should be monitored or assessed in order identify areas for improvement and measure the results of different actions or policies.

During this session, I will be focusing specifically on assessing performance measures for perinatal hepatitis B prevention programs.  If you would like to learn more about program evaluation, please visit the CDC immunization program evaluation link at the bottom of this slide.

## Activities Required by CDC

- Requirements for assessment of the perinatal hepatitis B include:
- frequency of assessment
- performance measures
- target levels
- For more information:
- 	_http://www.cdc.gov/vaccines/vac-gen/policies/ ipom/downloads/chp-05-perinatal-hepb.pdf_

**Notes:**
CDC’s National Center for Immunization and Respiratory Diseases funds 64 immunization grantees—all 50 states, six cities, and eight territories—to conduct various immunization program activities, including perinatal hepatitis B prevention.    CDC provides the immunization grantees with various requirements that should be met for each program component.  The required CDC activities for assessment of perinatal hepatitis B prevention program activities include the frequency with which the assessment should take place, the performance measure for each activity, and the target level.  

For example, one of the required activities for the 2008 through 2012 CDC grant cycle for perinatal hepatitis B prevention is to assess the presence of HBsAg test results in infant and maternal medical records in all delivery hospitals.  The frequency of the activity is once every five years.  The performance measure for this activity is the percentage of hospital medical records with documented maternal HBsAg results for the current pregnancy and the target is for 100% of the medical records to include maternal HBsAg status.

The perinatal hepatitis B prevention chapter of CDC’s Immunization Program Operations Manual (or IPOM) may be accessed at the weblink at the bottom of this slide.  This chapter includes all required and recommended activities for the perinatal hepatitis B prevention program for the 2008 through 2012 CDC immunization grant cycle.

## Overview

- Laboratory reporting audits
- Hospital record reviews
- Hospital policy surveys
- Expected births to HBsAg+ women
- Assessing case management
- Monitoring birth dose

**Notes:**
During this conference, I will review laboratory reporting audits, hospital medical record reviews, hospital policy surveys, CDC’s expected births to HBsAg-positive women, assessment of case management outcomes, and monitoring the birth dose of hepatitis B vaccine.

## Laboratory Reporting Audits

- Use to assess:
- completeness of reporting of HBsAg+ results
- timeliness of reporting
- Include delivery hospital labs
- Collaborate with communicable disease and state laboratory staff to help plan and execute reviews

**Notes:**
All immunization grantees should conduct laboratory reporting audits.  These audits are useful to assess the completeness of reporting of HBsAg-positive results from laboratories to the health department as well as the timeliness of reporting these results.  Delivery hospital laboratories should also be included in these audits to ensure all HBsAg-positive test results are received.  

Programs should collaborate with health department communicable disease and state laboratory staff to help plan and execute these laboratory reviews.  In some cases, communicable disease staff may already be conducting routine laboratory audits and may be able to add HBsAg as one of the results that is assessed.  Your state laboratory staff may be able to help out with planning and provide contact information for laboratories in your jurisdiction.

## Prioritize Labs to Evaluate

- Priority labs:
- labs serving high-morbidity areas or populations
- labs reporting large volume of hepatitis serology
- labs serving prenatal clinics
- delivery hospital labs
- How often?
- ideally, once/year for priority labs
- every 2–3 years for others
- minimally every 5 years

**Notes:**
Let’s talk now about how to prioritize the labs to be evaluated.  The labs that would be considered the highest priority should be those serving high hepatitis B morbidity areas or populations (such as an Asian immigrant community), those that report a large volume of positive hepatitis results, those labs that serve prenatal care clinics, and delivery hospital labs.

Next, let’s talk about how often labs should be evaluated.  Although the CDC immunization grant requirement is once every five years, it would be ideal to evaluate your priority labs once a year and to visit the remaining labs every two to three years.  

It is important to evaluate laboratory reporting this frequently because in most programs, the majority of HBsAg-positive pregnant women are identified through follow-up on HBsAg-positive laboratory reports.  Without good laboratory reporting, HBsAg-positive pregnant women and their babies may slip through the cracks.

## Laboratory Audit Tool

- A CDC tool available at:
** **

_**http://www.cdc.gov/ncidod/diseases/hepatitis/resource/LabReportingAssessment.doc**_

**Notes:**
CDC’s Division of Viral Hepatitis has developed a tool to audit laboratories.  This tool may be found at the link that was provided in the email you received with the log-in information for this netconference and can also be accessed at the weblink on this slide.

The tool contains some basic information on a laboratory’s hepatitis B virus testing, and asks questions that measure the completeness and timeliness of HBsAg reporting to the health department.

I will now walk through examples of measuring a laboratory’s completeness and timeliness of reporting...

## Completeness of Lab Reporting

- Pick a time period (at least 3 months)
- Calculate:

- Total # HBsAg+ test results reported
- Total # specimens that were HBsAg+

- =

- Proportion of HBsAg+ tests reported, or
- completeness of reporting

**Notes:**
To measure a laboratory’s completeness of HBsAg reporting, you must first choose a time period ,which should be at least three months.  Then, determine the total number of HBsAg-positive test results that were reported to the health department for the specified time period.  Next, determine the total number of specimens that were HBsAg-positive at the laboratory for the specified time period.  Finally, divide the total number of HBsAg-positive test results reported to the health department by the total number of specimens that tested HBsAg-positive at the laboratory to determine the proportion of HBsAg-positive tests that were reported to the health department.  This will give you the lab’s completeness of reporting.

Ask Annemarie about the “i” and “ii”—which one do they use??

## Timeliness of Lab Reporting

| Date of HBsAg serology | Date HBsAg+ report rec’d by health dept | # Days between serology and report |
| --- | --- | --- |
| 8-18-2007 | 8-23-2007 | 5 |
| 8-23-2007 | 8-29-2007 | 6 |
| 9-7-2007 | 9-12-2007 | 5 |
| 11-18-2007 | 11-27-2007 | 9 |
| SUM |  | 25 |

**Notes:**
Now let’s look at an example of measuring a laboratory’s timeliness of reporting.  You will need to choose a time period to measure the timeliness, which should again be a period of at least three months.  First, create a table similar to the one shown on this slide.  At the laboratory, access all HBsAg-positive serologies for your specified time period and record the date that each HBsAg-positive test was performed.  Next, record the date that the HBsAg-positive report was received at the health department.  Then, calculate the number of days between when the serology was ran and when it was received at the health department.  

## Timeliness of Lab Reporting - cont’d

- =

- Mean reporting time for the laboratory,
- or timeliness of reporting

*Time period for reporting is typically set by state statute*

- Sum of column “# days
- between serology and report”
- Total # of serologies evaluated

- 25
- 4

- =

- 6.25 days

**e.g. from the **
**previous slide**

**Notes:**
Next, sum the column in your table of the number of days between the serology and the report.  Sum the total number of HBsAg-positive serologies that are being evaluated.    Finally, divide the “sum of number of days between serology and report” by the “total number of serologies” to determine the lab’s mean reporting time to the health department, or their timeliness of reporting.  

In our example, the sum of days between serology reporting and report was 25, and we evaluated 4 serologies, so the mean reporting time for this laboratory was 6 days.  The time period for laboratories to report HBsAg-positive serologies to the health department varies and is typically set by state statute.

## Tips for Working with Laboratories

- Contact other areas of health department to see if activity is already being done
- See if an existing lab report can be used for the audit—ensure it is source data
- When making contact with labs, include:
- lab director
- technician
- IT staff

**Notes:**
This slide lists some tips for working with laboratories that I gathered through conversations with several city and state health department communicable disease and laboratory staff.  First, check with other areas of the health department, such as your communicable disease division, to see if laboratory audits are already being conducted.  You may be able to have HBsAg added to ongoing laboratory audits.  There are also several hepatitis EIP or (emerging infection program) sites which may already be conducting hepatitis audits in laboratories.  The hepatitis EIP sites are currently in Colorado, Connecticut, Minnesota, New York State, New York City, Oregon and San Francisco.

When auditing a laboratory, see if an existing laboratory report can be used to collect the audit data.  However, make sure that the data is source data, and that it’s not the routine report that is sent to the health department.

When you make contact with labs, include the lab director, a technician, and someone from the IT staff.  Sometimes the technician or IT staff will know of ways to make the lab audit quicker and easier since they work directly with the specimen data and databases.

## Hospital Record Reviews

- Use to assess:
- maternal HBsAg screening
- birth dose administration
- post-exposure prophylaxis (PEP) for infants born to HBsAg+ and unknown-status women
- Audit paired maternal and infant records
- Partner with health dept programs to collect other perinatal data
- rubella antibody		•  HIV
- group B strep		•  early hearing
- syphilis

**Notes:**
Another important type of program assessment is hospital medical record reviews.  Reviewing hospital records is a very labor intensive activity, but can also give you precise information on what is happening in a particular hospital.  These reviews can be used to assess:
Maternal HBsAg screening
Administration of a dose of hepatitis B vaccine to all infants
And administration of post-exposure prophylaxis for infants born to HBsAg-positive and unknown status women

Paired maternal and infant records should be audited to ensure documentation is consistent between the charts

You should consider working with other health department staff to develop medical record review tools and try to enlist these staff to assist with data collection at hospitals.  If staff time is not available, you could try to get resources from other areas of the health department to support the record reviews.  Some areas of the health department to consider partnering with on this activity are STD, HIV, group B strep, and early hearing detection.  It will save time for both health department staff and hospital staff if record reviews are consolidated.

## Determining Sample Size

- Tool available in *Guide to Life (Table 2.1)*

_**http://www.cdc.gov/ncidod/diseases/**_
_**hepatitis/resource/perinatalhepB.htm**_

**Notes:**
The first step to take before visiting a delivery hospital to conduct a medical record review is to determine the sample size for the records you will review during your visit.  A tool for determining samples for hospitals or programs of various sizes is available in Chapter 2 of the CDC Resource: Managing a Perinatal Hepatitis B Prevention Program: A Guide to Life, which can be found on the perinatal hepatitis B coordinator resource center webpage—the web address is listed at the bottom of this slide.  The table may also be accessed at the link that was provided in the email you received with the log-in information for this netconference.

This table is intended to be used to assess maternal HBsAg screening and universal hepatitis B vaccination rates.  It should not be used to assess post-exposure prophylaxis administration—a different methodology is needed for determining a sample for that type of assessment.

## Using the Guide to Life table

- Determine hospital’s annual number of deliveries
- Decide where it falls in the “Birth Cohort Size” category
- Estimate HBsAg screening and hepatitis B birth dose coverage for the hospital
- Use the _lower_ coverage to determine the sample size

**Notes:**
Let’s talk about how to use the Guide to Life sample size table.  The first thing you should do is determine the hospital’s annual number of deliveries.  Once you have this information, you can decide where the hospital falls in the “Birth Cohort Size” category—using the category that the hospital’s cohort size is closest to.  Next, you will need to estimate the maternal HBsAg screening rate and hepatitis B birth dose coverage for the hospital.  Most programs have conducted medical record reviews previously and have an idea of the percentage of pregnant women that receive HBsAg screening in their jurisdiction.  In most cases, the coverage is between 85 and 100%.  For hepatitis B birth dose coverage, programs can use immunization registry data or National Immunization Survey birth dose coverage data.  

Once you have estimated maternal HBsAg screening and hepatitis B birth dose vaccination coverage, you should use the lower coverage (which is usually the birth dose coverage level) to determine the sample size for the hospital.

## Sample Size Example

- 	Hospital X had 648 deliveries last year
- 	The expected HBsAg screening rate is 95%
- 	The state National Immunization Survey (NIS) birth dose rate is 65%
- 	Using the 65% coverage level, the sample size for Hospital X would be 111 records

**Notes:**
Let’s walk through an example.  Hospital X had 648 deliveries last year.  You estimate their maternal HBsAg screening rate to be 95%.  The National Immunization Survey hepatitis B birth dose coverage estimate for your state is 65%.  Using the lower coverage level, of 65%, and referencing the table from the Guide to Life, the sample size for Hospital X would be 111 paired mother-infant medical records.

## Hospital Data to Collect

- Delivery date/time
- Mother’s HBsAg test date/result/type in:
- maternal record
- infant record
- Infant hepatitis B vaccination (yes/no)
- Date/time of vaccination
- Infant HBIG administration (yes/no)

**Notes:**
Now let’s discuss what sort of data you should collect when you conduct hospital medical record reviews.  This is a list of the minimum data elements that you will want to gather, you may need to add additional items.  You will want to collect:

Delivery date and time
The mother’s HBsAg test date, result, and type in both the mother and infant records
Whether or not the infant received the hepatitis B vaccine before discharge
If so, the date and time of vaccination
Whether or not the infant received HBIG before discharge

## Hospital Data to Collect - cont’d

- Date/time of HBIG
- Other variables as appropriate

**Notes:**
If the infant received HBIG, the date and time of administration
You may want to collect additional variables, especially if you are collaborating with other areas of the health department such as HIV, STD or group B strep.

## Hospital Policy Surveys

- Use to assess whether hospitals have written policies/standing orders in place:
- universal birth dose of hep B
- HBsAg screening of pregnant women
- PEP to infants born to HBsAg+ and unknown-status women
- documentation of maternal HBsAg status, birth dose, and HBIG (if appropriate) in medical records

**Notes:**
Another important assessment is hospital policy surveys.  These can be used to assess whether delivery hospitals have written policies and standing orders in place to:
Administer a birth dose of hepatitis B vaccine to ALL infants prior to hospital discharge
Screen all pregnant women for HBsAg
Provide the appropriate post-exposure prophylaxis to infants born to HBsAg-positive and unknown status women
Document maternal HBsAg status, hepatitis B vaccine administration and HBIG administration (if applicable) in the appropriate medical records.

## Conducting Policy Surveys

- Mail survey
- Telephone survey
- In-person survey
-    during hospital
-    medical record
-    reviews

- Keep in mind, policies do not always match practice,
- so medical record reviews are essential

**Notes:**
There are a variety of ways in which a hospital policy survey can be done.  Probably the most common method for conducting a hospital policy survey is through the mail.  This is the least time consuming and least expensive method but the drawback may be a lower response rate.

Some programs prefer to conduct their policy surveys via telephone because it can yield higher response rates.

The other option is to conduct policy surveys in person during hospital medical record reviews.  This method will probably result in a 100% response rate.  Some programs mail the policy survey to the hospital in advance of the medical record review to give them an opportunity to complete it beforehand.  You may also want to ask the hospital for a copy of their policies and standing orders to take back with you.  

For mail and telephone surveys, you could ask the hospital to mail or fax a copy of their policies to you.

It is important to keep in mind that policies do not always match practice—which is why medical record reviews are essential.

## Expected Births to HBsAg+ Women

- CDC expected birth estimates are a benchmark for programs
- CDC provides estimates for states and some cities
- Estimates are calculated using:
- birth data
- prevalence of chronic
-    HBV infection by
-    race/ethnicity

**Notes:**
Now I would like to talk a little bit about the expected births to HBsAg-positive women estimates that CDC calculates annually.  These estimates are calculated for each state as well as selected cities and territories.  

CDC uses birth data that is reported from state vital statistics to the National Center for Health Statistics and applies hepatitis B virus prevalence data based on ethnicity to calculate the estimates.  

These estimates are used as a benchmark to measure national and program-specific progress toward the identification of births to HBsAg-positive women.

## Expected Births - cont’d

- Nationally, about 24,000 births expected annually, only 50% identified in 2005
- CDC encourages programs to improve estimates of births to HBsAg+ women
- More information in presentation at:
- 	_http://www.cdc.gov/ncidod/diseases/hepatitis/resource/presentations/2007-04PeriGrantees/PDFs/May1-p0130-1LJ_2007PeriCoordMtg-ExpectedBirths.pdf_

**Notes:**
Nationally, about 24,000 births to HBsAg-positive women are expected to occur annually.  However, only about 50% of these were identified by public health programs in 2005.  

The CDC encourages programs to find ways to improve state or local estimates of births to HBsAg-positive women.  For example, some states collect mother’s country of birth on their birth certificates.  These data can be used to refine estimates of births to HBsAg-positive women.  For more information on expected births, please visit the weblink at the bottom of this slide to view a presentation I gave during the May 2005 Perinatal Hepatitis B and HIV Coordinator’s Meeting.

## Assessing Case Management

- Number of infants born to HBsAg+ women
- Proportion of infants receiving
- hep B vaccine & HBIG within 12 hours of birth
- on-time completion of hep B vaccine series
- post-vaccination testing
- Reasons for cases lost to follow-up
- Proportion of screened and vaccinated household and sexual contacts

**Notes:**
Another important activity that I would like to discuss is assessing case management.  Programs should monitor and assess the number of infants identified by their program that are born to HBsAg-positive women.  In most places, the number of cases increases each year, so any decreases should be closely examined.  Sometimes decreases in identification are related to case worker turnover or a laboratory that has stopped reporting HBsAg-positive results to the health department.

Programs should also assess the proportion of infants born to HBsAg-positive women who receive hepatitis B vaccine and HBIG within 12 hours of birth, on-time completion of the hepatitis B vaccine series, and post-vaccination serologic testing.  The goal for all three outcomes is 90%.

Cases that are lost to follow up should be examined to determine the reasons for losing the case.  Categorizing and describing these reasons for lost to follow up may allow programs to adopt new strategies to keep cases until completion.

Programs should also routinely assess the proportion of screened and vaccinated household and sexual contacts to see what gaps exist.

## Assessing Case Mgmt - cont’d

- Monitor case management outcomes:
- compare program-wide rates (entire state or city) to national rates (CDC peritable)
- regionally
- locally (county level)
- by case worker
- quarterly or monthly
- over time (trends)

**Notes:**
It is important to monitor the case management outcomes we just outlined in a variety of different ways.  First, programs should compare their case management rates to the national rates—each year, shortly after the completion of CDC’s annual perinatal report, a spreadsheet summarizing aggregate program data from each CDC grantee is shared with the perinatal hepatitis B coordinators.  This spreadsheet contains the national averages for all case management outcomes.

If the program has regional case workers, it is important to analyze case management outcomes by region to see which regions need to improve their rates.  This will also help identify whether education is needed for case workers.

For programs that contract perinatal hepatitis B case identification and/or case management to local health departments, case data should be monitored at the local or county level to see where case identification or case management can be improved and to identify counties that are doing a good job so that strategies can be shared across counties.

In programs that have centralized case workers doing the work, case outcomes should be monitored for each worker.

Case identification and case management data should be monitored frequently—either monthly or quarterly.  If you wait until the end of the year to assess outcomes, it may be too late to make improvements that may impact cases.

It is also important to monitor your program’s outcomes over time and watch the trends.  If your program has good case management or case identification rates which then decrease over a few quarters, you will want to identify the reasons for these decreases.

## Monitoring Birth Dose

- National Immunization Survey (NIS) data
- measures hepatitis B birth dose rates
- Hospital medical record reviews
- Immunization Information Systems (IIS)
- hep B vaccine birth certificate data automatically uploaded into IIS
- calculate birth dose coverage for cities, regions, and individual hospitals

**Notes:**
Another important programmatic outcome that should be routinely monitored is hepatitis B birth dose vaccination.  

The National Immunization Survey, or NIS, which is an annual CDC survey that measures immunization coverage in every state and select cities for children age 19-35 months, measures hepatitis B birth dose rates.  The birth dose is currently measured as a dose of hepatitis B vaccine within 2 days of life.

Hospital medical record reviews are another means of measuring hepatitis B birth dose administration.  This method is very helpful for determining hospital-specific rates.

Some programs are able to use immunization information system or IIS data to measure birth dose hepatitis B rates.  In some states and cities, hepatitis B vaccination is collected on the electronic birth certificate and these data are automatically uploaded into the IIS.  This information can then be used to calculated birth dose rates for cities, regions, and individual hospitals.  

## Continuous Monitoring

- Some aspects of the program require continuous monitoring
- case management outcomes
- laboratory reporting
- Plan ahead for assessments your program will conduct over the next year/s

**Notes:**
Please remember that certain aspects of perinatal hepatitis B prevention programs require continuous monitoring—such as case management outcomes and laboratory reporting.  You will want to detect any problems in these areas as soon as possible so that corrective action can be taken.

It is also wise to plan ahead for the assessments your program will conduct over the next several years.  You may not have the staff to visit all of your birthing hospitals in a single year for medical record reviews, but you could plan a schedule of which hospitals will be visited each year for the next three years.

## Using Your Results

- Determine future direction and focus of program efforts
- Provide feedback to:
- hospitals
- laboratories
- providers
- health departments
- case workers

**Notes:**
Finally, remember to use your results!  The outcomes of program monitoring and assessment should be used to determine the future direction and focus of your program’s efforts.  After you conduct program monitoring and assessment activities, be sure to provide constructive feedback to: delivery hospitals, laboratories, providers, health departments and case workers.  

Good luck and thank you for your attention.