HIVQUAL

Workbook

Guide for Quality Improvement in HIV Care

Health Resources and Services Administration HIV/AIDS Bureau 

New York State Department of Health AIDS Institute

Developed for the U.S. Department of Health and Human Services

Health Resources and Services Administration

HIV/AIDS Bureau 

By the New York Department of Health

AIDS Institute

The National HIVQUAL Project

September 2006

This publication was supported by grant number 

231030010FR from the HIV/AIDS Bureau, Health 

Resources and Services Administration. Its contents 

are solely the responsibility of the authors and do not 

necessarily represent the official views of the Health 

Resources and Services Administration.

“Untitled” - painting by Frank Holliday, HIV-positive artist
NYSDOH AI HIVQUAL Workbook September 2006

2

New York State Department of Health

AIDS Institute

Guthrie Birkhead, M.D., M.P.H.

Director

Humberto Cruz, M.S.

Executive Deputy Director

Barbara Devore, M.P.A.

Executive Deputy Director

Bruce Agins, M.D., M.P.H.

Medical Director

Office of the Medical Director

HIV Quality of Care Program

Bruce Agins, M.D., M.P.H.

Medical Director

Clemens Steinböck, M.B.A.

Director of Quality Initiatives

Franklin Laufer, Ph.D.

Director for Performance Measurement

Todd Mohrmann, M.A.

Manager, HIV Guidelines Program
NYSDOH AI HIVQUAL Workbook September 2006

3

The HIVQUAL Project

Bruce Agins, MD, MPH

HIVQUAL Project Director

Clemens Steinböck, M.B.A.

HIVQUAL Specialist

Lily Jiang

HIVQUAL Project Data Manager

Chris Wells

HIVQUAL Data Analyst

Nanette Brey-Magnani, Ed.D.

QI Consultant

Amelia Broussard, Ph.D., R.N., M.P.H.

QI Consultant

Johanna Buck, M.A., R.N.

QI Consultant

Kathleen Clanon, M.D.

QI Consultant

Onelia Crespo, M.D.

QI Consultant

Lori DeLorenzo, M.S.N, R.N.

QI Consultant

Jeffrey A. Menzer, R.N., A.C.R.N.

QI Consultant

Margaret Palumbo, M.P.H.

QI Consultant

Barbara Rosa, R.N.-C., M.S.

QI Consultant

Dan Sendzik

QI Consultant

Nancy Showers, D.S.W.

QI Consultant

Saneese Stephen, R.P.A.-C., M.P.A.

QI Consultant
NYSDOH AI HIVQUAL Workbook September 2006

4

Index.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Workbook Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

National HIVQUAL Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Overview of the HIVQUAL Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

Program Cycle - HIVQUAL Model

Step I: 

Develop and Plan a Quality Management Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

 Identify Leaders and Key Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

 Form a Quality Management Committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

 Develop a HIV Quality Management Management Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 

Strategize Implementation of the Quality Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Step II:

Facilitate Implementation of HIV Quality Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Establish Performance Measurement Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Review Performance Measurement Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Establish Improvement Project Teams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Support Quality Improvement Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Step III: 

Evaluate HIV Quality Management Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Evaluate HIV Quality Management Program Effectiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Evaluate Project Team Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Learn from and Respond to Past Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

Strategize to Sustain the HIV Quality Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Table of Contents
NYSDOH AI HIVQUAL Workbook September 2006

5

Project Cycle - HIVQUAL Model

Step 1: 

Review, Collect and Analyze Project Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

Review and Collect Project Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

Analyze and Share Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90

Step 2: 

Develop a Project Team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Establish Project Team Dynamics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

 Write an Improvement Project Memo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Step 3:

 Investigate the Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

Understand the Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

 Identify Underlying Causes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Step 4: 

Plan and Test Changes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

Conduct Plan-Do-Study-Act (PDSA) Cycles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

Plan for Pilot Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .120

Conduct Pilot Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125

Step 5: 

Evaluate Results with Key Stakeholders. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

Summarize Improvement Project Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Evaluate Results with HIV Quality Committee. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

Step 6: 

Systematize Changes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Sustain Improvements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

Assess Team Effectiveness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Additional Resources

 Sampling and Data Collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

 Construct a population Sample. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

 Design a Data Collection Tool. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

 Assign and Train Abstractors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

 Validate Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

NYSDOH AI HIVQUAL Workbook September 2006

7

Index

A

Assessment tool, quality program. . . . . . . . . . . . . . . . . . . . 72

B

Baseline data, analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

Brainstorming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

C

Cause-and-effect diagram. . . . . . . . . . . . . . . . . . . . . 110, 113

Cause analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

D

Data abstractors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86, 151

Data collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84, 142

Display of performance data. . . . . . . . . . . . . . . . . . . . . . . . 92

F

Flowchart. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

G

Gantt chart. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

H

HIVQUAL Project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

HIV treatment guidelines. . . . . . . . . . . . . . . . . . . . . . . . . . 85

I

Ideas for pilot tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Indicators for HIV care. . . . . . . . . . . . . . . . . . . . . 45, 58, 88

Indicator selection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Investigating the process. . . . . . . . . . . . . . . . . . . . . . . . . . 103

L

Leaders and key stakeholders. . . . . . . . . . . . . . . . . . . . . . . 30

O

Organizational assessment tool. . . . . . . . . . . . . . . . . . . . . . 72

P

Pareto chart. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

PDSA cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

Performance data, collection. . . . . . . . . . . . . . . . . . . . . . . 86

Performance measurement system. . . . . . . . . . . . . . . . . . . . 57

Pilot test, evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Pilot test, implementation. . . . . . . . . . . . . . . . . . . . . . . . . 125

Pilot test, selecting. . . . . . . . . . . . . . . . . . . . . . . . . . . 120, 121

Presenting data and/or QI projects. . . . . . . . . . . . . . . 90, 92

Project team memo and workplan. . . . . . . . . . . . . . . . 98, 102

Project team, assessment tool. . . . . . . . . . . . . . . . . . . . . . . 73

Project team, establishment. . . . . . . . . . . . . . . . . . . . . . 61. 93

Project team, evaluation. . . . . . . . . . . . . . . . . . . . . . . 75, 132

Q

Quality committee. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Quality committee membership. . . . . . . . . . . . . . . . . . . . 34

Quality goals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44, 40

Quality plan management, development. . . . . 38, 43, 50, 52

Quality program, evaluation. . . . . . . . . . . . . . . . . . . . 42, 69

Quality program, implementation. . . . . . . . . . . . . . . . . . . 55
NYSDOH AI HIVQUAL Workbook September 2006

8

R

Random number table. . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

Responsibilities of quality committee. . . . . . . . . . . . . . . . . 32

Resources, books, articles, and websites. . . . . . . . . . . . . . 26

S

Sample size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Sample, construction. . . . . . . . . . . . . . . . . . . . . . . . . 86, 143

Staff involvement. . . . . . . . . . . . . . . . . . . . . . . . . . . 124, 138

Storyboard. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Support for improvement project. . . . . . . . . . . . . . . . . . . . 78

Support for quality program. . . . . . . . . . . . . . . . . . . . . 78, 41

Sustain improvements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

Sustain quality program. . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Systematize changes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

T

Team effectiveness. . . . . . . . . . . . . . . . . . . . . . . . . . . . 75, 140

Team ground rules. . . . . . . . . . . . . . . . . . . . . . . . . . 100, 101

Team roles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95, 97

V

Validation of performance data results. . . . . . . . . . . . . . . 84

W

Workplan, quality program. . . . . . . . . . . . . . . . . . . . . . 49, 52

Workplan, project team. . . . . . . . . . . . . . . . . . . . . . . . . 93, 61
NYSDOH AI HIVQUAL Workbook September 2006

9

New York State Department of Health AIDS Institute gratefully acknowledges the help of the following HIVQUAL sites and 

staff who contributed their stories and examples of HIV care to the HIVQUAL Workbook.

Acknowledgments of HIV Programs 

STAR Health Center at SUNY Downstate 

Box 1240, 450 Clarkson Avenue

Brooklyn, NY 11203

Caseload: 1,100 patients

Staff: 30

Type of facility: Hospital-based clinic, Designated 

AIDS Center

Contact: Kristina Brown, R.P.A.-C., Quality 

Improvement Coordinator

Phone: 718-270-2842

Fax: 718-270-2298

Email: Kristina.brown@downstate.edu 

Albany Medical Center

47 New Scotland Avenue MC158

Albany, NY 12208

Caseload: 1,400 patients

Staff: 75 – 85 

Type of facility: Academic center

Contact: Sheila Boyle, Process 

Improvement Coordinator

Phone: 518-262-4025

Fax: 518-262-8460

Email: boyles@mail.amc.edu

Brooklyn Hospital Center

121 De Kalb Avenue, Box 197

Brooklyn, NY 11201

Caseload: 800

Staff: 39

Type of Facility: Hospital

Contact: Dan Sendzik, Executive Director 

Phone: 718-940-5934

Fax: 718-940-5515

Email: dps9001@nyp.org

Circle of Care 

260 S. Broad Street, Suite 1000 

Philadelphia, PA 19102

Caseload: 1000+

Staff: 12

Type of Facility: Provider network

Contact: Brian M. Green, Manager-Quality 

Improvement & Planning 

Phone: 215-985-2627 

Fax: 215-732-1252 

Email: brian@familyplanning.org 

Hennepin County Medical Center

701 Park Avenue, 41A North

Minneapolis, MN 55415

Caseload: 1379

Staff: 12.95

Type of facility: Public hospital

Contact: JaneAnn Hall, IT Specialist

Sandy Bergner, R.N.

Phone: 612-873-2121

Fax: 612-904-4684

Email: JaneAnn.Hall@co.hennepin.mn.us 

Interfaith Community Health Center

220 Unity Street

Bellingham, WA 98225

Caseload: 135

Staff: 55

Type of facility: Community health center

Contact: Sam Curd, Care Coordinator for 

Ryan White Primary Care Program

Phone: 360-676-6177 

Fax: 360-671-3574

Email: Sam_curd@opportunitycouncil.org

Kings County Hospital Center

Center for HIV Outpatient Primary Care 

Evaluation (H.O.P.E.)

451 Clarkson Ave, Bldg H

Brooklyn, NY 11203

Caseload: 900

Staff: 20

Type of facility: Hospital-based outpatient clinic 

Contact: Saneese Stephen, R.P.A.-C., M.P.A. 

Associate Director

Phone: 718-245-2808

Fax: 718-735-4165

Email: stephens@nychhc.org

Middletown Community Health Center

21 Orchard Street

Middletown, NY 10940

Caseload: ~110

Staff: 7

Type of facility: Community health center

Contact: Dina Kapzynski, Quality 

Improvement Coordinator

Phone: 845-343-7614, ext 239

Fax: 845-343-5390

Email: dkapzynski@middletownchc.org 

Judy Brooks, Quality Consultant

Phone: 760-471-0418

Fax: 760-471-0418

Email: jbqualnrse@aol.com 
NYSDOH AI HIVQUAL Workbook September 2006

10

Additional thanks to Gavin Imperato, Shalmali Karnad, Sara Brownschidle and Anne Conlin of the AIDS Institute for their 

efforts in collecting the HIV snapshots and their work on this book.

Montefiore Medical Center

AIDS Center

Center for Positive Living

111 East 210th Street

Bronx, NY 10467

Caseload: 2,200

Staff: 97

Type of facility: Hospital-based outpatient clinic

Contact: Joan Vileno, R.N. Quality 

Improvement Coordinator

Phone: 718-920-2859

Fax: 718-405-0610

Email: jvileno@montefiore.org 

North County Health Services

150 Valpreda Road

San Marcos, CA 92069

Caseload: 350

Staff: 35

Type of facility: Community health center

Contact: Alicia Nunez-Liriano, 

Director of HIV Programs

Phone: 760-736-6725

Fax: 760-736-3210

Email: aliriano@nchs-health.org

St. John’s Riverside Hospital

Two Park Avenue

Yonkers, NY 10703-3497

Caseload: 250 – 350

Staff: 25

Type of facility: Hospital

Contact: Kay Scott, A.C.S.W., C.A.S.A.C.

Phone: 914-964-7595/7412

Fax: 914-964-7321

Email: kscott@riversidehealth.org

St. Joseph’s Family And Women’s Care

145 North 6th Street

Reading, PA 19601

Caseload: 375

Staff: 4

Type of Facility: Hospital

Contact: James Pratt, Program Director

Phone: 610-378-2551

Fax: 610-378-2562

Email: jamesprattjr@catholichealth.net

Unconditional Love, Inc.

1495 N. Harbor City Blvd.

Melbourne, FL 32935

Caseload: 598

Staff: 25

Facility: 7 Solo/Group medical practice

Contacts: Phyllis Johnson, R.N., C.Q.I.

Joyce Goode, A.R.N.P.

Phone: 321-253-0846

Fax: 321-253-1004 

Email: chc@spacey.net

Unity Health Care, Inc.

70 N Street, NE

Washington, DC 20002

Caseload: 2,774

Staff: 35

Type of facility: Community and homeless health 

centers

Contact: Julie Ahlrich, R.N., B.S.N. 

Health Center Manager

Phone: 202-727-0660

Fax: 202-727-0375

Email: jahlrich@unityhealthcare.org 

University of Miami School of Medicine

Department of Obstetrics & Gynecology (D-53)

1400 NW 10th Avenue

Miami, FL 33136

Caseload: 4,218

Staff: 55

Type of facility: Academic medical center

Contact: Barbara Messick, M.P.H. 

Phone: 305-243-2160

Fax: 305-243-2152

E-mail: bmessick@med.miami.edu 

University of Pittsburgh Medical Center

200 Lothrop Street, 603 Falk

Pittsburgh, PA 15213

Caseload: 1,000

Staff: 25

Type of facility: Hospital-based university clinic

Contact: Margaret A. Palumbo, M.P.H.

Administrative Director, HIV Program

Phone: 412-647-2038

Fax: 412-647-5519

Email: palumbom@msx.upmc.edu

UT Southwestern Medical Center

Dallas Family Access Network 

CMC-Pavillion

1935 Motor Street

Dallas, Texas 75235

Caseload: 4,076

Staff: 26

Type of Facility: University Medical Center

Contact: Elizabeth (Betty) Cabrera, 

Executive Director

Phone: 214-456-5897 

Fax: 214-456-2860

Email: BETTY.CABRERA@childrens.com
NYSDOH AI HIVQUAL Workbook September 2006

11

Betsy Siegel, R.N., B.S.N.

Infectious Diseases Clinics of Emory, Atlanta, GA

Betsy_Siegel@emoryhealthcare.org

Brian M. Green 

Circle of Care, Philadelphia, PA

brian@familyplanning.org

Cheryl Hamill, R.N., M.S., A.C.R.N.

University of MS Medical Center

Jackson, MS

chamill@medicine.umsmed.edu

Cynthia Gail Manor 

Austin Travis County Health & Human Services 

Dept. Austin, TX 

Cynthia.Manor@ci.austin.tx.us

Daphyne Williams, M.P.H. 

Sixteenth Street Community Health Center, 

Milwaukee, WI daphyne.williams@SSCHC.ORG

Debbie Konkle-Parker, Ph.D., F.N.P.

University of Mississippi Medical Center 

Jackson, MS 

dkparker@medicine.umsmed.edu

Deborah Dewey 

New York State Department of Health AIDS Institute 

Albany, NY 

dmd16@health.state.ny.us

Donald Hardin

Washington University School of Medicine, St Louis, MO 

dhardin@im.wustl.edu

Donna Needham, R.N.

Thundermist Health Center, Woonsocket, RI

donnan@thundermisthealth.org

Elizabeth Cabrera 

Dallas Family Access Network, Dallas, TX

BCABRE@childmed.dallas.tx.us

Ellen Tanner 

New York State Department of Health AIDS Institute, 

Albany, NY

etv02@health.state.ny.us

Erica Avery

Phoenix Indian Medical Center, Phoenix, AZ 

erica.avery@mail.ihs.gov

We also want to thank the following HIV providers for volunteering their time to review the HIVQUAL Workbook. Their

comments and feedback shaped the development of this book and we are tremendously grateful for their commitment to quality 

improvement in HIV care. 

Acknowledgments of Reviewers
NYSDOH AI HIVQUAL Workbook September 2006

12

Fran Rotunno, R.N. 

VidaCare, Inc., New York, NY 

frotunno@vidacare.org

Gloria Jorgensen, R.N.

CHC/RVC, Davenport, IA

gjorgensen@davchc.com

Jan Hufnagle, R.Ph. 

Caring Together Pharmacist, York, PA 

hufnagle@yorkhealthcorp.com

Jeffrey Guidry, Ph.D.

UT Southwestern Medical Center, Dallas Family Network

Dallas, TX 

jguidry@hlkn.tamu.edu

Judith A. Aberg, M.D.

Washington University School of Medicine

St. Louis, MO 

JABERG@im.wustl.edu

Kimberly L. McCombs 

Washington University School of Medicine - Project ARK 

St. Louis, MO

McCombs_K@kids.wustl.edu

Kristen Ries, M.D. 

University of Utah, Division of Infectious Diseases

Salt Lake City, UT 

kristen.ries@hsc.utah.edu

Laura Friedman, C.S.W. 

Mount Sinai Medical Center, New York, NY 

Laura.Friedman@msnyuhealth.org

Paula Merrill

Marks, Mississippi 

Twomarksters@aol.com

Phyllis Johnson, R.N. 

Unconditional Love, Inc, Melbourne, FL 

chc@spacey.net

Richard Johnson 

Comprehensive Care Clinic - Lancaster General Hospital 

Lancaster, PA 

rbjohnso@lancastergeneral.org

Robert J. Di Vito, Esq. 

DeKalb County Board of Health, Decatur, GA 

rjdivito@gdph.state.ga.us

Sally Hughes Lee, M.S.N., Ed.D. 

Quality of Life Health Services, Gadsden, AL 

sallyhugheslee@yahoo.com

Sandy Frentz 

Sioux Falls Health Department, Sioux Falls, SD 

sfrentz@SIOUXFALLS.org

Steve Palermo, R.N., M.B.A., C.P.H.Q. 

USF Infectious Disease Clinic, Tampa, FL 

Steve_Palermo@doh.state.fl.us

Susan Weigl 

Callen-Lorde Community Health Center, New York, NY 

sweigl@yahoo.com

Tacy Kirkendall 

University of Nebraska Medical Center, Omaha, NE

tkirkendall@unmc.edu
NYSDOH AI HIVQUAL Workbook September 2006

13

The road to quality improvement in HIV care may be 

likened to a journey. But that journey is generally not a trip 

on a well-groomed path. The ever-changing nature of HIV 

care presents myriad challenges along the road that leads to 

the summit of improved care for patients. Once this path has 

been forged and trodden, it becomes smooth and familiar. 

Ideally, this proven path will be traveled often and branch 

into new roads in response to the changing system of care. 

Whether a facility is located in the inner city or a rural area, 

a large hospital-based program or a small neighborhood 

center, the challenges of ensuring that HIV patients receive 

appropriate care are shared. To name a few: keeping current 

with the sheer breadth of knowledge, skills and experience 

underpinning effective HIV treatment; serving the increasing 

needs of people living with HIV/AIDS, their partners, family 

members and friends; and dealing with limited resources, 

ever-present time pressures and budget constraints. Successfully achieving and sustaining quality improvements in HIV 

care over time requires not attending to individual processes 

and projects in isolation but rather understanding how all the 

pieces fit together and balance one another, how changing 

one element can change the rest, and how sequencing and 

pace affect the whole structure of delivering care. 

The New York State Department of Health AIDS Institute 

has worked for more than ten years with HIV providers 

to help them build capacity and capabilities for quality 

improvement. The HIVQUAL Workbook captures the combined expertise and accomplishments of participating HIV 

programs around New York State and the nation, and the 

knowledge of HIVQUAL consultants who guide providers 

through the challenges and the key tasks related to improving the quality of HIV care.

The HIVQUAL Workbook is based and structured on the 

HIVQUAL Model and its Program and Project Cycles. 

Workbook content explains the purpose of each cycle and 

then describes the key tasks for its successful completion. 

The Program Cycle focuses on the necessary steps to plan, 

build and sustain an HIV-specific quality infrastructure 

while the Project Cycle shows how to implement a quality 

initiative to improve one aspect of HIV care.

As you review the information, begin to consider its applicability within your facility. Ask yourself how these practical 

materials can help your facility address daily challenges and 

improve comprehensive HIV care.

If your organization participates in the National HIVQUAL 

Project, some of the materials in the Workbook may already 

be familiar to you. We hope you will use this Workbook as a 

resource for the ongoing implementation of quality improvement at your site.

Introduction
NYSDOH AI HIVQUAL Workbook September 2006

14

If your organization has not been part of the HIVQUAL 

Project, we hope this Workbook will give you and your 

colleagues a clear road map for making quality improvement a reality in your HIV program and provide you an 

opportunity to share our quality experience. The Workbook 

materials are adaptable to your facility whether or not you 

participate in HIVQUAL. It is our sincere hope that after 

using the HIVQUAL Workbook you will join us in this 

exciting initiative devoted to improving care for people 

living with HIV. 

Good luck in your quality improvement journey!
NYSDOH AI HIVQUAL Workbook September 2006

15

Workbook Use

Purpose

The HIVQUAL Workbook provides you with a practical guide for improving the quality of HIV care, offering 

step-by-step guidance on how to build a quality program, 

orchestrate quality improvements and then sustain 

those improvements.

Target Audience

The Workbook is applicable to all HIV programs, 

regardless of caseload, geographic location or service 

delivery model used. Anyone interested in making quality 

improvements within a facility can use the HIVQUAL 

Workbook; it does not assume existing quality knowledge 

nor does it require participation in the HIVQUAL 

Project. This section will introduce you to the organization of the Workbook and describe how to get started 

using these materials.

Structure

There are ten different tab sections in the HIVQUAL 

Workbook. As shown in Figure 1, the first nine tabs 

correspond to the steps in the HIVQUAL Model. Supplemental resources are found behind the last tab, Tab 10. 

Sections in the HIVQUAL Workbook covering the 

HIVQUAL Model include the following components 

and corresponding icons.

HIVQUAL Model

The ‘dual circles’ icon shows the two interdependent cycles 

of the HIVQUAL Model and provides a ‘road map location’ 

in this document for each of the nine steps. The Program 

Cycle (‘outer circle’) shows the sequence of activities for 

developing a facility’s HIV quality improvement program. 

(Program Step I is highlighted here). The Project Cycle 

(‘inner circle’) shows the steps for conducting specific 

quality improvement projects.

Figure 1: HIVQUAL Workbook Structure

PROGRAM CYCLE

Tab 1

Tab 2

Tab 3

Tab 4

Tab 5

PROJECT CYCLE

Tab 7

Tab 8

Tab 9

Tab 10

ADDITIONAL RESOURCES

Step I: Develop and Plan a Quality Management Program

Step II: Facilitate Implementation of HIV Quality Program

Step III: Evaluate HIV Quality Management Program

Step 1: Review, Collect and Analyze Project Data

Step 2: Develop a Project Team 

Step 4: Plan and Test Changes

Step 5: Evaluate Results with Key Stakeholders

Step 6: Systematize Changes

Sampling and Data Collection

Tab 6 Step 3: Investigate the Process
NYSDOH AI HIVQUAL Workbook September 2006

16

Snapshot of HIV Care

These real-life examples describe how HIV care providers 

have completed the entire step or a key activity within 

the step. The ‘camera’ icon will call attention to a real-life 

snapshot of HIV care.

Additional Resource

The ‘open-book’ icon directs you to other publications 

or resources that you may want to consult for additional 

information. 

Toolbox

Throughout the Workbook, Toolbox content explains 

how to apply quality tools at appropriate times during 

the quality improvement process. A ‘Toolbox’ icon calls 

attention to Toolbox explanations. 

Real-World Tip

A ‘light bulb’ icon designates suggestions that can help 

you to successfully implement an activity. These concrete 

and practical tips summarize best practices of current 

HIVQUAL Project participants.

Getting Started

This HIVQUAL Workbook may be used both as an 

introduction to quality improvement in HIV care facilities 

and as a reference tool. If your HIV care facility is just 

embracing the quality improvement movement for the very 

first time, you may want to review the Workbook in its entirety and then go back and use the step-by-step guidelines 

to direct a quality improvement cycle and to plan, implement and evaluate a quality program. HIV care facilities 

experienced in quality improvement may simply choose steps 

on an as-needed basis—those that are most relevant to the 

current improvement needs of your facility. 

If you have questions about the HIVQUAL Workbook, wish 

to receive additional copies, have a success story to submit 

for future editions or want to learn about participating in 

the National HIVQUAL Project, visit our website at www. 

hivqual.org or you may call 212-417-4614.

We wish you the very best in your efforts to improve the care 

of people living with HIV and hope that this Workbook 

serves you well in this journey.
NYSDOH AI HIVQUAL Workbook September 2006

17

The National HIVQUAL Project

What is HIVQUAL?

The HIVQUAL Project represents the joint efforts 

of the New York State Department of Health AIDS Institute 

and the Health Resources and Services Administration 

(HRSA) HIV/AIDS Bureau. It is a national project funded 

by HAB Division of Community Based Programs and 

supports Title III and Title IV grantees in their efforts to 

sustain quality improvement in HIV ambulatory care. 

In 1995, the AIDS Institute implemented a pilot project 

to determine the feasibility of using the New York State 

quality improvement capacity-building model model in 

Title III programs nationally. This successful pilot project 

was undertaken in Pennsylvania among six grantees, each 

of which demonstrated significant improvements after a 

15-month program that included bimonthly on-site visits by 

a consultant with quality improvement expertise. Following 

this success, the Title III program supported the expansion 

of the HIVQUAL Project to other states in 1997. During the 

past 5 years, the HIVQUAL Project has steadily evolved to 

support the growth of quality management programs among 

Title III and Title IV grantees, further expand the profile of 

indicators in the project’s software, provide over 100 grantees with on-site consultation, demonstrate improvements in 

the quality of HIV care and incorporate new methods and 

tools in the quality improvement field.

On-site consultation services and other resource materials 

help participating HIVQUAL sites learn how to systematically plan, implement and evaluate a quality management 

program and conduct quality improvement projects. Benchmarking of key clinical issues is promoted. Publications such 

as the HIVQUAL newsletter with examples of best practices 

as well as regional workshops provide numerous opportunities for peer learning and information sharing.

HIVQUAL Goals

The overarching intent of the National HIVQUAL Project is 

to build the capacity and capability among Ryan White Title 

III and Title IV grantees to sustain quality improvement in 

HIV care by providing on-site consulting services. As such, 

key HIVQUAL Project goals and objectives are to:

• Improve the quality of care for persons with HIV receiving care in Title III and Title IV-supported programs.

• Promote quality improvement activities.

• Promote self-reporting of HIV performance. 

measurement data based on clinical guidelines

• Provide site-specific consultation to build quality

improvement capacity which is responsive to the 

specific organizational needs of grantees.
NYSDOH AI HIVQUAL Workbook September 2006

18

HIVQUAL Principles

The National HIVQUAL Project promotes the following 

practical quality improvement principles to help HIV providers improve the quality of HIV care within their facilities:

• Infrastructure enhances systematic implementation of 

quality improvement activities.

• Performance measurement lays the foundation 

for quality improvement.

• Indicators are based on accepted guidelines 

or developed through formal group-decision 

making methods.

• Quality improvement should be integrated into ongoing 

planning and work of HIV programs.

HIVQUAL Indicators

Improving elements of HIV care are best based on 

a system of routine performance measurement. The 

HIVQUAL methodology helps HIV care facilities to 

measure and analyze performance data in an effort to:

• Shape and guide quality improvement activities.

• Identify critical aspects of HIV care for improvement.

• Monitor the quality of care and services provided.

• Assess baseline and follow-up performance data to determine if improvements have been made.

The Project uses a variety of quality indicators. HIV-QUAL 

Project indicators are listed in Figure 2. A more detailed 

description of these indicators can be found on page 88.

HIVQUAL INDICATORS:

Figure 2: HIV Indicators

HIV specialist care

HIV monitoring (CD4 cell count and viral load testing)

Antiretroviral (ARV) therapy medication

ARV therapy management

ARV treatment education

Adherence to ARV therapy

Lipid screening

PCP prophylaxis

MAC prophylaxis

Gynecological care

Mycobacterium tuberculosis (PPD) screening 

Syphilis serology

Pneumococcal vaccination 

Substance use assessment

Mental health screening

Dental care

Ophthalmologic care

Basic patient education
NYSDOH AI HIVQUAL Workbook September 2006

19

Benefits of HIVQUAL Participation

The benefits of participating in the National HIVQUAL 

Project include:

• On-site quality improvement consultation service with 

HIVQUAL consultants.

• Organizational assessment of your existing quality management program.

• Establishment of an HIV quality infrastructure for ongoing monitoring and continuous improvements in care.

• Performance measurement software, HIVQUAL3, and 

technical support.

• Benchmarking of performance data with other participating Title III and Title IV grantees.

• Development of skills and experience in applying quality 

improvement methods to HIV care.

• Sharing of best practices and success stories with other 

clinics via HIVQUAL publications, workshops and other 

peer-learning opportunities.

• Stronger links to HRSA.

The HIVQUAL Project has led to performance improvement at participating facilities. An analysis of HIVQUAL 

performance data found that facilities that participated in 

the HIVQUAL project improved their performance in all 

but one of HIVQUAL indicators. Figure 3 shows the five 

year performance trends of facilities participating in the 

HIVQUAL Project over the five year period 2001-2005. 

The improvements in performance, with the exception of 

pelvic exams and MAC prophylaxis, are statistically significant at p<0.01; with the improvement for MAC prophylaxis 

is significant at p<0.05. Improvements were not seen in viral 

load monitoring; this decrease is most likely attributable to a 

change in the standard of care and the corresponding indicator for more frequent viral load monitoring. (These results 

were presented at the International AIDS Conferences in 

Bangkok in 2004 and Toronto in 2006.)

HIVQUAL3 Software

The HIVQUAL Project supports quality improvement and 

performance measurement among grantees through the use 

of its existing software, called HIVQUAL3. This software 

program is an MS-Access runtime application that provides 

an efficient means of measuring and reporting clinical performance. Data from participating grantees are compiled in 

annual blinded benchmarking reports, which are distributed 

to grantees at the end of each data collection cycle. These 

comparative data provide a powerful stimulus to grantees, 

while recognizing the successes of those with higher scores. 

This HIVQUAL3 software includes indicators based 

on clinical practice guidelines developed by the New York 

State Department of Health AIDS Institute, its expert advisory committees, and the Title III and Title IV HIVQUAL 

Advisory Committees. Indicator definitions are consistent 

with public health guidelines.
NYSDOH AI HIVQUAL Workbook September 2006

20

Figure 3: HIVQUAL Performance Data

5 - Year Trends

Features of the HIVQUAL3 software include:

• A simple vehicle for the rapid generation of reports to 

monitor clinical performance for those patients entered 

in the database.

• Current performance data for use in internal quality programs and external benchmarking with other 

HIVQUAL Project participants.

• Data analysis for different subgroups, including age, 

gender, viral load, race/ethnicity and exposure categories, CD4 count, and clinic site.

Use of HIVQUAL3 software is not required for participation in the project if existing systems produce reports that 

are equivalent to those generated by HIVQUAL3. Please 

note that HIVQUAL3 is not designed to be a clinical 

tracking program for patients in a clinic.

If your program is interested in fully participating in the 

HIVQUAL Project, please contact your HRSA Project Officer. Or, for more information about the HIVQUAL Project, 

visit our website at www.hivqual.org or you may 

call 212-417-4614.

$%$PVOU.POJUPSJOH

7JSBM-PBE.POJUPSJOH

"37ɨFSBQZ"EIFSFODF

)""35ɨFSBQZ$%

1$11SPQIZMBYJT

."$1SPQIZMBYJT

11%4DSFFOJOH

1FMWJD&YBN

4VCTUBODF6TF4DSFFOJOH

5PCBDDP4DSFFOJOH

)FQBUJUJT$

4ZQIJMJT4FSPMPHZ

%FOUBM&YBN

'&&

.&

,&

*&

(&

(&&' (&&( (&&) (&&* (&&+
NYSDOH AI HIVQUAL Workbook September 2006

21

The HIVQUAL Model, pictured below Figure 4, was developed by the New York State Department of Health AIDS 

Institute, in collaboration with the HIVQUAL consultants, 

to assist health care facilities developing a quality infrastructure that supports ongoing processes to improve the quality 

of HIV care. The Model emphasizes the concept that in 

Overview of the HIVQUAL Model

order to achieve sustainable improvements in quality, activities need to focus both on the structural programmatic level 

and on the project level. Accordingly, the Model is divided 

into two interdependent cycles: 1.) a sequence of activities 

for developing a grantee’s HIV quality management program 

or Program Cycle (‘outer cycle’) and 2.) steps for conductFigure 4: HIVQUAL Model

Step 4:

Plan and test 

changes

Step 5:

Evaluate results 

with key stakeholders

Step 6: 

Systematize 

changes

Step III:

Evaluate

 HIV quality 

management 

program

Step I:

Develop and

plan a quality 

management 

program

Step II:

Facilitate

implementation 

of HIV quality 

program

Step 2: 

Develop a project 

team

Step 3: 

Investigate the 

process

Step 1:

Review, collect 

and analyze 

project data

Program Cycle

Project Cycle
NYSDOH AI HIVQUAL Workbook September 2006

22

ing specific quality improvement projects or Project Cycle 

(‘inner cycle’). By incorporating parallel cycles of activities at 

both the program and project levels, the Model allows grantees to build a sound HIV-specific quality infrastructure that 

will support the quality improvement processes. The Model 

is a practical guide for HIV programs, regardless of their 

service delivery model, patient caseload, or site location.

Program Cycle

The HIV Quality Program Cycle describes the activities 

necessary to build and sustain an HIV-specific quality 

infrastructure: developing and planning an internal quality management program, facilitating the implementation 

of the program and evaluating the quality program. These 

program activities are completed by the leadership of the 

HIV program in conjunction with HIV quality staff. 

Generally, leadership and staff form a quality committee 

to organize and facilitate the HIV program’s efforts.

In working with a variety of health care facilities, 

HIVQUAL consultants observed that HIV programs 

have significantly better results in achieving and sustaining improvement efforts over time when an HIV- specific infrastructure for quality is in place. Such a quality 

infrastructure is important whether an organization is 

newly undertaking quality improvement or experienced 

in quality initiatives.

Program Step I: Develop and Plan a Quality 

Management Program.

The Program Cycle’s first step is devoted to establishing and/or maintaining a quality management program 

throughout the organization. The HIV program leadership, together with the HIV staff, develop or refine an 

HIV-specific quality management plan that describes the 

intent of the program, prioritizes annual HIV quality 

goals and projects, establishes accountability, and outlines resources for quality improvement activities. 

Program Step II: Facilitate Implementation of HIV 

Quality Program.

Once the infrastructure for the HIV quality program 

is set, the work necessary to achieve the goals is initiated. Most quality improvement activities are achieved 

through a series of improvement projects each addressing 

single quality issues in the areas where change is needed 

the most. During program facilitation, the HIV quality 

committee helps implement the quality plan by orchestrating these projects. Specifically, quality committee 

members establish cross-functional project teams, educate 

the teams in quality improvement concepts and techniques, and oversee the progress. They also coach teams 

through project difficulties and help maintain a free flow 

of information with, and among, team members.
NYSDOH AI HIVQUAL Workbook September 2006

23

Program Step III: Evaluate HIV Quality 

Management Program.

In this step of the Program Cycle, the HIV quality 

committee periodically (at least, annually) evaluates the 

improvements achieved by individual project teams and 

develops strategies to sustain the improvements over time, 

including effective communication methods, recognition of performance, and promotion of success stories. 

Leadership also assesses the effectiveness of the entire 

HIV quality management program, including goals and 

infrastructure (as outlined in the quality management 

plan). Opportunities for improvement in future HIV 

quality workplans are identified.

Together, the three steps in Program Cycle are an ongoing process. Once the HIV program leadership and the 

HIV quality committee evaluate their efforts from the 

previous year, they return to the first step to develop and 

plan quality efforts for the upcoming year. 

Project Cycle

HIV quality leaders and the HIV quality committee 

select areas in which a facility needs to improve at 

the program level. But the improvement activities themselves are made by quality improvement teams at the 

project level.

An HIV quality improvement project is the main vehicle 

by which a facility makes incremental process improvements to its various care elements and changes its 

processes and systems of care. Specific indicators (such as 

ARV management, PCP prophylaxis and GYN exams) 

are used to measure care elements to ensure sustained 

success over time.

There are six primary steps in a Project Cycle. Although 

the Model and the subsequent discussion cover only one 

improvement project effort, several projects can and often 

do, occur simultaneously.

Project Step 1: Review, Collect and Analyze Project Data.

In the first step of the Project Cycle, team members 

collect and review performance measurement data and 

identify specific indicators for process improvement. 

Data collection methods are identified and collection 

tools are designed to measure the current level of performance. The results are shared with the HIV quality 

committee. This step helps the project team to set realistic 

process improvement goals and make informed improvement decisions.

As an example, consider a quality improvement team which 

has been asked to improve the annual GYN exam process. The 

team begins by collecting data from a sample of female medical 

records and determines that 65 percent of patients have received 

a pelvic exam in the previous year. After discussing the results 

with the HIV quality committee, they decide to investigate 

why the rate is lower than expected and how to improve the 

GYN exam rate.
NYSDOH AI HIVQUAL Workbook September 2006

24

Project Step 2: Develop a Project Team.

In this step, the project team workplan outlines what and 

how work will be accomplished during the Project Cycle, 

including project goals and resource limitations. 

Returning to the example, the GYN improvement team sets a 

goal to improve the exam rate to 80 percent and outlines a plan 

to specifically investigate the pelvic exam appointment process.

Project Step 3: Investigate the Process.

During this step, team members investigate the process 

being reviewed and chart the sequential steps of the process 

flow for better understanding. The resulting flowchart helps 

reveal potential problem areas. From here, members identify 

and prioritize possible causes.

In the GYN example, the project team members create a flowchart to outline the GYN exam appointment process. Based on 

this information, the team further investigates potential causes 

and identifies the inconsistency of making annual pelvic exam 

appointments as a major underlying cause. In other words, 

providers are not alerted when the next GYN exam is due. 

Project Step 4: Plan and Test Changes.

With the information gathered in Project Step 3, team 

members select potential solutions for pilot testing. A pilot 

test is a small-scale implementation of a change. It is used to 

determine if the change works and if it should be implemented HIV program-wide. During this step, the team 

plans for and implements pilot tests (PDSA cycles), and 

then assesses the impact of those changes to ensure that 

they result in improvements.

For the GYN improvement team, one of the proposed changes 

is to identify records for GYN exam appointments directly 

through the HIV program’s electronic medical record system 

rather than manual scheduling. For the pilot test, team members implement the new identification process for 2 weeks and 

then measure the percentage of records that were properly identified along with the increased number of pelvic exams.

Project Step 5: Evaluate Results with Key Stakeholders.

During Project Step 5, pilot test results are reviewed and 

evaluated. The team reviews the pilot test results with the 

HIV quality committee and other stakeholders (those who 

have a vested interest in the process). Together, they discuss 

whether the change should be implemented system-wide.

For example, the GYN improvement team members discover 

that 100 percent of the records were properly identified by the 

HIV program’s electronic medical record system and that the 

pelvic exam rate increased to 81 percent. They discuss their 

results with the HIV quality committee and key stakeholders.
NYSDOH AI HIVQUAL Workbook September 2006

25

Project Step 6: Systematize changes.

Finally, in Project Step 6, team members make successful 

project-related improvements part of the daily work process 

in an effort to sustain the improvements over time. They also 

assess the project’s effectiveness against the original plan and 

make plans to re-measure performance at regular intervals 

and monitor improvements.

In the GYN example, the team updates front desk procedures to 

reflect the new identification process and orients all staff on the 

change. After a few weeks, members re-measure the percentage 

of women receiving an annual pelvic exam and discuss how to 

sustain appointment gains long-term. 

The six steps of the Project Cycle help to ensure that process 

improvements are based on data rather than anecdote, are 

piloted before implemented facility-wide, and are re-measured for long-term effectiveness. Similar to the Program 

Cycle, the Project Cycle is ongoing in order to support continuous quality initiatives. Once changes are systematized 

for one project, you are ready for the next improvement opportunity. Both cycles of the HIVQUAL Model are essential 

to a facility’s quality strategy. Without a well-grounded 

quality improvement program, project efforts are poorly coordinated and quality improvements are difficult to sustain. 

Without cycles of projects, the quality program is essentially 

an effort on paper only.
NYSDOH AI HIVQUAL Workbook September 2006

26

Resources

HIVQUAL Publications

• HIVQUAL Group Learning Guide: Interactive Quality Improvement Exercises for HIV Health Care Providers. A publication 

of the New York State Department of Health AIDS Institute and the Health Resources and Services Administration HIV/AIDS 

Bureau, Updated 2006; www.hivqual.org

• Measuring Clinical Performance: A Guide for HIV Health Care Providers. A publication of the New York State Department of 

Health AIDS Institute Updated 2006; www.hivqual.org

• Patient Satisfaction Survey for HIV Ambulatory Care. A publication of the New York State Department of Health AIDS 

Institute, 2002; www.hivqual.org

HRSA Publications on Quality

• Improving Care for People Living with HIV/AIDS Disease. Institute for Healthcare Improvement, HRSA/HAB. HIV/AIDS 

Bureau Collaborative. Order via the HRSA Information Center at www.ask.hrsa.gov or call 888-ASK-HRSA.

• The Modular Quality Improvement Curriculum for Improving HIV Care. Institute for Healthcare Improvement, 

HRSA/HAB, HIV/AIDS Bureau; www.ihi.org/IHI/Topics/HIVAIDS/HIVDiseaseGeneral/Tools

• Quality Management: Technical Assistance Manual. HIV/AIDS Bureau (HAB) of the Health Resources and Services 

Administration (HRSA); www.hab.hrsa.gov/tools/QM

Books on Quality

• Agresti, Alan. An Introduction to Categorical Data Analysis, Wiley Series in Probability and Statistics. Applied Probability and 

Statistics, May 1996.

• Brassard M, Ritter D, Rilter D, Oddo F. The Memory Jogger II. Goal /QPC, Lawrence MA; 1994.

• Carey, R.G., and Lloyd, R.C. Measuring Quality Improvement in Healthcare. New York: Quality Resources, 1995.

• Delbecq, A., Vand de Ven, A. and Gustafson, D. Group Techniques for Program Planning: A Guide to Nominal Group and 

Delphi Processes. Middleton, WI: Green Briar Press, 1975/1986.

• Gerteis M, Edgman-Levitan S, Daley J, Delbanco T L. Through the Patient’s Eyes – Understanding and Promoting PatientCentered Care. San Francisco, CA: Jossey-Bass; 1993.

• Graham, N. Quality in Health Care: Theory, Application, and Evolution. Gaithersburg, MD: Aspen Publications, 1995.

• Institute of Medicine Committee on Quality of Health Care in the US, Institute of Medicine, Crossing the Quality Chasm: A 

New Health System for the 21st Century, National Academy Press, 2001.
NYSDOH AI HIVQUAL Workbook September 2006

27

• Langley Gerald J., Nolan, Kevin M., Nolan, Thomas W., Norman, Clifford L., and Provost, Lloyd P. 

The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. San Francisco, CA: 

Jossey-Bass Publishers, 1996.

• Rogers E. Diffusion of Innovation. New York: Free Press; 1995.

• Scholtes, Peter R., Joiner, Brian L. and Streibel, Barbara J. The Team Handbook. Second Edition. Madison, WI: 

Joiner Associates Inc., 1996.

• Swanson, Roger C. The Quality Improvement Handbook: Team Guide to Tools and Techniques. Delray Beach, FL: 

St. Lucie Press, 1995.

Websites on Quality

• National HIVQUAL Project - www.hivqual.org

• National Quality Center - www.nationalqualitycenter.org

• New York State Department of Health AIDS Institute - www.hivguidelines.org

• Agency for Healthcare Research and Quality (AHRQ) - www.ahrq.gov/qual 

• HIV/AIDS Treatment Information Service - www.aidsinfo.nih.gov

• HRSA Bureau of Primary Health Care Quality Center - www.bphc.hrsa.gov/quality 

• HRSA Center on Quality - www.hrsa.gov/quality

• Infectious Diseases Society of America - www.hivma.org/HIV/tocCEN.htm

• Institute for Healthcare Improvement (IHI) - www.ihi.org

• Johns Hopkins AIDS Service - www.hopkins-aids.edu

• National Quality Measures Clearinghouse - www.qualitymeasures.ahrq.gov

Books on Quality...Continued
NYSDOH AI HIVQUAL Workbook September 2006

28

The Big Picture:

Quality management programs are most successful when 

led and supported by the leadership of the HIV facility. 

Leaders can provide an environment conductive to establishing changes. In Program Step I, those individuals who 

will champion the improvement efforts at strategic levels 

are identified. These individuals then assume responsibility for the HIV quality management plan—a blueprint to 

guide quality initiatives for the upcoming year.

What To Do:

• Identify leaders and key stakeholders.

• Form a quality management committee.

• Develop HIV quality management plan.

• Strategize to implement the quality plan.

Program Cycle

Project Cycle

Program Step I: Develop and Plan a Quality 

Management Program

Snapshot of HIV Care

Training Your Staff – A Great Start 

It’s not necessarily important where you start in developing your 

quality program. It’s that you start. 

Providing training on quality management to staff is often an initial and key step that can build interest, motivation and ownership 

for the quality program. Interfaith Community Health Center, 

Bellingham, WA trained its staff and board members so that everyone had a common knowledge base from the start and were able to 

participate in the planning process. Including the entire staff and 

governance helped jump start the program. “The general sense in 

the clinic was that the time for this kind of consistency had come,” 

says Sam Curd, Care Coordinator. “The idea of making quality 

improvements has been embraced.”
NYSDOH AI HIVQUAL Workbook September 2006

29

Snapshot of HIV Care...Continued

Over time quality experts in the field noted that in order to sustain 

quality improvement gains, a sound quality management infrastructure is needed. Sitting down with staff and defining the quality 

infrastructure is critical to map out who needs to be involved, what 

goals you have for your quality program and when staff meet to talk 

about quality. Whether you are a stand-alone community health 

center or a multi-site agency or network, this process takes time, 

involves others and requires decisions by leaders. 

The Dallas Family Access Network (Dallas FAN) is composed of 

a lead agency and subcontractors that provide medical and social 

services. When developing their quality program, they needed to 

form a consensus on what they were going to measure and how the 

quality program would be structured. Decisions had to be acceptable to both the lead agency as well as to the subcontractors—many 

of which had their own existing quality management programs and 

different data collection systems. 

“Networks are challenging,” states Elizabeth (Betty) Cabrera, 

Executive Director. The process was lengthy. “We didn’t want to 

develop a whole new set of indicators for people to measure, so we 

had to get buy-in from network members.” Identifying indicators 

that were common across the network and were able to be measured 

was also challenging. “We know that our comprehensive services 

work, and we want to be able to measure how it makes a difference 

for our clients.”

Involving Staff And Consumers

Julie Alrich, at Unity Health Care, a multi-site agency in Washington, DC says that it is important to resist the temptation to have 

only one person develop the plan. “This would have simplified the 

process, but we would have missed the opportunity to have discussions and constructive dialog.” States Alrich, “The group approach 

not only allowed everyone to feel ownership of the final product, 

but it also served as an educational forum. By creating the quality 

management plan together, we were able to learn from each other.”

Recruiting quality committee members is often difficult at facilities 

where staff already feel over-worked. However, at the PATH Center 

at Brooklyn Hospital, interested staff had to be turned away because 

of overwhelming interest in serving on its committee. Dan Sendzik, Executive Director attributes the popularity of this committee 

to the leadership’s support. “Staff are eager to participate,” he says. 

The committee’s work is seen as important. “The issues discussed 

are real and relevant. People want to be part of the decisions that 

will affect their day-to-day work.” Also, Sendzik believes, people 

also see professional and personal benefits to participating on the 

committee. “People feel they are learning something from being on 

the committee. The committee has a seminar quality about it that 

attracts professionals who want to learn more.”

The Circle of Care, Philadelphia, PA sought to involve consumers in 

the development of quality indicators for their family-centered primary care and case management services. Providers and consumers 

worked together. A consumer panel using a “talk show” format was 

part of the process. The group developed a consensus document on 

four primary care and four case management indicators. 

Outlining Your Quality Program On Paper

Too often when staff moves on or other priorities appear, no staff 

member seems to clearly remember what the next steps are. A 

simple written quality management plan can help you to remember 

how HIV programs can accomplish their quality goals, where they 

left off at their last quality committee and clarifies the roles and 

responsibilities of staff. During a recent national conference call, an 

HIV provider noted that ‘stealing shamelessly, sharing senselessly’ 

was their approach to learn from other already existing plans. She 

noted: “Seeing and reviewing other plans, made me realize a common framework and how simple they can be. They were 

really helpful.” The message was load and clear, use established 

templates and get started. 

What Should Your Quality Infrastructure Look Like?

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

30

Identify Leaders And Key Stakeholders.

Simply stated: leadership is an essential component of an 

HIV quality program. For HIV programs just beginning on 

a quality journey, leadership helps in getting those quality 

efforts off the ground and making sure they do not peter 

out after a few well-intentioned months. For those HIV 

programs that have an existing quality program, leadership 

is crucial to maintain and support ongoing changes to the 

HIV care delivery system.

Leaders are those individuals who have the ability to formally and informally influence and inspire others providing 

a vision and direction for the quality program. Leaders create 

the culture in which quality is both prized and promoted.

In addition to leaders, key stakeholders are also instrumental 

in an HIV quality program. Stakeholders are those who have 

an interest in seeing quality efforts succeed. This category 

of people might include patients, advocacy groups, board 

members, or representatives from funding organizations.

Dialogue is initiated to listen to the voices of leaders and key 

stakeholders. The objectives are to:

• Better understand the environment in which the quality 

management program works.

• Get input and buy-in in key areas for the quality program.

• Develop a vision for the quality program and 

strategically outline the goals and objectives.

• Identify potential members for a quality management 

committee or leaders of quality activities.

In most cases, this dialogue is informal. But the information gathered should be documented so that the ideas and 

opinions can be integrated later into the development of an 

HIV-specific quality program. 

The Toolbox on page 31 provides a simple chart for 

identifying individuals. Use the chart to answer the 

following question: Who are the key individuals at your 

HIV facility from whom you can obtain input and build a 

sustainable quality management program?

Keep the following guidelines in mind when engaging individuals who will be involved in the HIV quality program:

• Talk with program staff and patients; include as many 

people as is feasible.

• Interview the leaders and stakeholders about their ideas 

for building a successful quality management program.

• Hold focus groups or interviews with patients to identify 

their needs and expectations.

• Consult routinely with the Consumer Advisory Board 

(CAB) and ask for recommendations.

• Meet with the facility’s board to discuss the main directions for the HIV quality program. 

Real-World Tip

Engage Leaders And Key Stakeholders.
NYSDOH AI HIVQUAL Workbook September 2006

31

Toolbox: 

Identification of Quality Improvement Leadership

Formal Leaders

Medical Director

HIV Program Administrator

Practitioners

Nurses

Case Managers

Quality Improvement Staff

Others

Who NAME

Key Stakeholders

Patients

Board Member(s)

Patient Advocate(s)

Project Officer/Grant Representative

Others

OTHERS

Individuals who influence others because of 

their personality, ability to get things done, or 

in-depth knowledge

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

32

Form A Quality Management Committee.

In order to build momentum for HIV quality improvement 

activities, a group of individuals can be brought together—

an HIV quality management committee. These individuals 

build an HIV facility’s capacity and capability for quality 

improvement. Some of the same people identified as leaders or 

key stakeholders may also serve on the quality committee. 

The major task of a quality management committee is to 

help ensure everything is in place at an HIV facility for the 

improvement efforts to succeed and be sustained over time. 

The quality committee plans and oversees all quality program 

activities at the facility, particularly the quality improvement 

projects completed by individual project teams. 

Members of the quality committee have five main 

areas of responsibility:

• Strategic planning. The quality committee is charged 

with strategizing how to best establish and maintain a 

sustainable quality management program. The committee 

develops the facility’s HIV quality management plan and 

prioritizes goals and projects so that the most critical areas 

are addressed first. The committee assumes responsibility 

for outlining the quality management program infrastructure, identifying performance measures, and planning for 

program evaluation.

• Facilitating innovation and change. A quality committee 

removes any negative restraints or barriers to achieving 

and sustaining improvements. Depending on the given 

situation, this may require changing policies that could 

potentially impede improvements. Some level of training may be necessary to prepare staff for change. Or, it 

might be as simple as promoting open, two-way communication to ensure everyone at a facility feels that they 

have a voice in the quality management program.

• Providing guidance and reassurance. On a routine basis 

the quality committee oversees the progress of quality 

activities to ensure that they are staying on track and 

to provide guidance. Quality improvement projects will 

involve some measure of changing the status quo. 

Departures from routine conventions can become challenging. The quality committee needs to listen, observe, 

be responsive to staff needs during the improvement 

process, and provide support and encouragement at 

appropriate junctures.

• Establishing a common culture. In order for an HIV quality management program to succeed, staff will want to be 

"on board." To gain the buy-in of staff and stakeholders, 

the quality committee should demonstrate a true commitment to the quality program. Success in making HIV 

care improvements results in quality being a shared value 

among everyone at a facility.

• Allocating resources. HIV facilities already face many 

time constraints and resource pressures on a daily basis. 

The HIV quality committee needs to ensure that staff 

has both the time and the information necessary to 

actively participate in the projects set forth in the quality 

management plan. 
NYSDOH AI HIVQUAL Workbook September 2006

33

Toolbox: 

Organizational Chart for Quality Management

The following example showcases the organizational structure for quality management:

‘The hospital-wide Quality Management Committee oversees internal quality improvement activities that are performed at 

the HIV/AIDS Center. The overall responsibility for the HIV/AIDS Center lies with the Program Director who authorizes 

the HIV/AIDS Center Quality Committee to plan, assess, measure, and implement performance improvements throughout the entire HIV/AIDS Center. The HIV/AIDS Center Quality Committee reports back to the hospital-wide Quality 

Management Committee.’

Hospital-wide Quality

MANAGEMENT Committee

HIV/AIDS Center

Program Director

HIV/AIDS Center

Quality Committee

HIV/AIDS Center

Program Director

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

34

Additional Resource

The HIVQUAL Group Learning Guide "Leadership for Quality" exercise can help you teach small groups about a quality leader’s roles and 

responsibilities. It could also be used as an icebreaker when a facility’s quality program leaders meet for the first time. You can download this 

publication at www.hivqual.org.

Selecting quality committee members

Quality committee participation depends on several 

factors including HIV service delivery model, co-located 

services, and program size. Typically a cross-functional 

representation of all professional backgrounds proves 

most effective in planning and decision-making. 

Visualize taking a vertical and horizontal slice of 

your organization.

The Toolbox on page 35 lists the titles of individuals who 

are logical candidates to serve on a quality committee at 

two different HIV care facilities. 

If your facility already has a quality management program 

and an HIV quality committee in place, plan to annually review and (as necessary) revise the list of committee 

members. In putting together an HIV quality committee for 

the first time, the HIV medical director in consultation with 

staff and the facility’s leadership decides who should serve on 

the committee.

An HIV quality committee should be an actively engaged 

committee which can make a difference. To build a 

credible committee: 

• Build a cross-functional group; draw from different 

service areas within the clinic.

• Identify individuals who are potential influencers and 

can get things done.

• Start with a small group of individuals who are most 

critical to the program’s success.

• Keep everyone informed about the process.

• If applicable, include cross-title representation (Ryan 

White Title III and Title IV) in this committee.

• Include patients and/or stakeholders on the committee.

• Identify a person who will facilitate the meeting.

Real-World Tip

Build A Credible Committee.

Notes
NYSDOH AI HIVQUAL Workbook September 2006

35

Toolbox: 

Quality Planning Meeting Agenda

Purpose: Family Health Center’s first draft of the annual quality management plan

Date: December 3, 2005

Time: 1:00 – 4:00pm

Place: Conference Room A

Participants: Dr. Jane Dissan, Dr. Vincent Seaton, Taimi Miller, René Santos

TIME

1:00

TOPIC

1:05

1:45

2:15

3:45

WHO

All

Dr. Dissan

All

All

All

René Santos

Quality Committee Members for a

Teaching Hospital (HIV Caseload: 700)

Quality Committee Members for a Neighborhood

Health Center (HIV Caseload: 100)

Chief of Infectious Diseases

AIDS Center Administrator

Director of Ambulatory Care

Director of Quality Improvement

Director of Nursing

AIDS Center Nurse Practitioner

Clinic Coordinator for Case Management

Senior Staff Nurse

Patient Representative

Title 4 Provider

Medical Director

Senior Staff Nurse 

HIV Nurse

Case Manager

Patient Representative

Toolbox: 

Quality Improvement Leaders

This Toolbox provides two examples of quality committee membership to highlight the differences between various 

HIV delivery systems.

1:15

Check-in

Review meeting purpose and agenda topics

Review and discussion of 2006 quality improvement goals (Handout A)

Review of project improvement team results for 2005 (Handout B)

Determine the focus of the quality management plan for 2006

Summarize next steps/action items

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

36

Getting committee work done

To be effective, the HIV quality committee needs to

have basic operational ground rules in place.

• Chairperson identification. One individual is responsible 

for directing the activities of the quality committee, 

mediating and resolving conflicts among committee 

members, and representing the quality committee to 

other parts of the HIV facility. This chairperson needs

to fully understand the HIV facility’s quality goals 

and principles. 

• Meeting frequency and duration. A regular quality 

committee meeting schedule should be set up at least 

bi-monthly. The scheduled time should be as convenient 

as possible for all committee members. Meetings should 

start and end on time to facilitate attendance.

• Documentation. Every quality committee meeting 

should have an agenda to guide points of discussion and 

prevent the session from running over time. If it’s not 

possible to develop an agenda in advance, the first few 

minutes of the meeting should be spent writing one on 

a flipchart. Likewise, minutes of the meeting should be 

recorded to summarize the discussion points and remind 

committee members about action items to be performed. 

In the event that new staff or consumers join the quality 

committee, such historical documentation can be valuable in getting them up to speed.

• Communication. It is important to keep staff and 

consumers informed about committee activities. Many 

quality committees do this effectively by sharing meeting minutes with the facility board, staff and consumers. 

HIV quality committee members cannot be expected to 

be experts in all the knowledge and skills required to plan 

and oversee quality activities. The following practices can 

contribute to committee’s success:

• Educate all committee members about quality

improvement methodologies and tools.

• Develop a committee charter to outline the committee 

purpose and member responsibilities.

• Initially provide a wide range of performance data to 

inform members about the status-quo.

• Spend time to develop an agenda and necessary handouts for meetings; share them prior to the meeting.

• Keep minutes; brief minutes are better than 

no minutes.

• At least once a year, dedicate a quality committee 

meeting to reviewing and/or revising the 

committee’s charter.

• Create a ‘notebook’ that includes the quality management plan, workplans, agendas, and minutes.

Real-World Tip

Prepare The HIV Quality Committee.

This helps staff to understand the improvement issues 

and challenges facing the quality committee. You may 

consider posting your findings in the hallways and waiting rooms for consumers to see how the facility 

is working to improve HIV care.
NYSDOH AI HIVQUAL Workbook September 2006

37

Toolbox: 

Quality Improvement Committee Minutes

Quality Committee Meeting

Date

Members Present

December 20th, 2005

Victoria S (chair)

Catherine G; Hannah W; Daphnee C; Sheldon M; Barbara W; Michael D; Susan G; Stephen 

A, MD; Tom D; Janet W; Isaac K; Leona W; Peter S; Clemens S

Call to order at 8:00am

8:00am

Reviewed and Approved

Meeting was adjourned at

Minutes from last meeting

Announcements None

Quality Committee Minutes

Topic/Agenda Discussion/Findings Follow-up

• The rate of compliance for the growth/ 

nutrition indicator will be followed up by the 

forms committee. Actions regarding PPD reading 

need to be discussed by the program’s leadership.

Follow-up by Steve A

• It was suggested that a meeting take place 

between Catherine G, Barbara W and Clemens S 

to revise the review tool and 

to discuss the presentation of the results.

Follow-up by Catherine G

• Pediatrics

• Social work

• MMTP

• The results of a Pediatric Outpatient review were presented 

indicating compliance rates of 73 percent for growth/nutrition. It was discussed that the compliance rate of the Dental 

Care indicator has increased from 22 percent in 8/2005 to 75 

percent in 11/2005. It was also discussed that the 30 percent 

rate of compliance for the TB status indicator is due to inaccessibility for follow-up appt. for the PPD reading.

• The results of a Social Work Inpatient Review were presented indicating that 89 percent of our inpatients were receiving 

medical care previous to the admission. It was discussed that 

patients are being followed up by a private attending. It was 

discussed that some clarification needs to be provided regarding the results of the review.

• The results of an Annual Medical Review were presented. It 

was discussed that the indicators used in the review tool are 

based from OASIS’ Standards. It was also discussed that as 

of 11/19 MMTP began working jointly with CTRPN. The 

results of a Counseling Admission Review were presented 

indicating low compliance rates for agreement for testing 

(25 percent), completed within first week of treatment (20 

percent). Orientation completed and signed w/in 30 days (67 

percent). It was discussed that there was additional information added to the interviewing process and the conversation. 

• It was suggested that we go beyond what 

OASIS asks for as indicators and include 

HIV testing and Hepatitis indicators to the 

review tool. It was discussed that there should be 

automatic referrals for HIV counseling and testing (for new admission to program and to annual 

review). Follow-up by Susan G 

submitted by: Daphee C; December 27th, 2005

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

38

Develop An HIV Quality Management Plan.

Lasting improvements in HIV care do not happen overnight. True to the HIVQUAL Model, the planning process 

is also continuous. In particular, an HIV quality management plan serves as the blueprint for quality initiatives. 

It describes the overriding purpose of a facility’s quality 

program, the infrastructure that supports quality activities, 

and its goals for the upcoming year. It also serves as a reference tool for both current and future staff. The following 

two steps are needed:

• Deciding on a planning approach 

• Creating the quality management plan

Deciding On A Planning Approach

The planning process to develop a quality management plan 

provides an opportunity to create a sense of ownership among 

staff members and consumers for the facility’s improvement 

initiatives. Before diving into the details of your quality management plan, decide on a general approach for developing and 

finalizing the plan that includes a wide representation of staff 

and stakeholders. Select from one, or a combination, of the 

following planning options.

• Planning meeting. Facilitate a yearly meeting in which 

decisions are made regarding the key components of the 

quality management plan. Prior to the annual meeting, 

gather and distribute background information to 

participants pertaining to the meeting and prepare a 

draft of recommendations to give focus to the 

decision-making process.

• Series of planning meetings. Break the annual 

meeting down into smaller steps and plan a series 

of short meetings.

• Planning group. Rather than a large-group meeting, 

form a group comprised of two to three individuals who 

are responsible for delineating a process to gather staff 

and stakeholder input to subsequently finalize the quality 

management plan.

Whichever option is chosen, the planning meetings should 

be held at a scheduled time and place with an agenda 

prepared. A sample meeting agenda is shown in the 

Toolbox on page 35.

Additional Resource

For guidance in teaching small groups how to develop a quality improvement management plan, see the HIVQUAL Group Learning Guide 

"Quality Improvement Management Plan" exercise. The exercise could also be used to familiarize a leadership group before starting to develop 

a plan. You can download this publication at www.hivqual.org.
NYSDOH AI HIVQUAL Workbook September 2006

39

Quality Management Infrastructure

How are we organized?

The quality management infrastructure describes how the 

program is structured and staffed in order to get work done:

• Leadership: Who is ultimately responsible for the HIV 

program’s quality initiatives?

• Quality committee structure: Who chairs the 

HIV quality committee? Which staff serves on the 

quality committee?

• Quality committee meeting frequency: When will 

the quality committee meet to assess progress and plan 

future activities?

• Quality committee reporting: What is the relationship 

of the quality committee to the facility at large? How 

will the quality committee communicate its progress to 

staff and consumers?

If your facility uses organizational charts to illustrate 

how groups and individuals are aligned across the facility, 

consider using the same model to create a quality program 

organizational chart. The Toolbox on page 33 shows 

an example.

Creating The Quality Management Plan

A quality management plan defines a quality program’s 

strategic direction and provides a blueprint for upcoming 

improvement activities for the HIV program. While there 

is no universal "how-to" template for creating a quality 

management plan, this section outlines the basic elements 

that should be covered:

• Quality statement

• Quality improvement infrastructure

• Performance measurement

• Annual quality goals 

• Participation of stakeholders

• Evaluation

The Toolbox on page 43 showcases an HIV facility’s quality 

improvement management plan which illustrates examples 

for each element. More examples are listed later in this step.

Quality Statement

What do we want to be? 

A quality statement describes the purpose of the HIV quality program. It is the end to which all other program activities are directed. Some programs may refer to this as their 

quality mission statement, others as their guiding purpose 

for quality activities. To write a quality statement for the 

quality management plan, assume an ideal world and ask 

yourselves, "What do we want to be for our patients 

and our community?" 

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

40

Performance Measurement

How will we assess progress?

Performance measurement is a method for identifying 

and quantifying the critical aspects of care within your 

facility. This is essential to assembling baseline performance 

data and measuring the effectiveness of improvement efforts 

over time.

Most HIV quality programs assess progress using quality 

of care indicators. A quality of care indicator is a measured 

aspect of patient care used to evaluate the extent to which a 

facility provides a particular element of care. Generally, indicators are based on specific standards of care derived from 

guidelines issued by a professional society and/or government agency. For example, the frequency of viral load tests is 

an indicator for how well a facility conducts HIV monitoring. Based on current guidelines, a viral load test should 

occur every three to four months. 

Identify aspects of care for performance measurement, 

keeping in mind four main criteria:

• Relevance. Does the indicator relate to a condition that 

occurs frequently or has a great impact on the patients 

at the facility?

• Measurability. Can the indicator realistically and efficiently be measured given the facility’s resources?

• Improvability. Can the performance rate associated with 

the indicator realistically be improved given the limitations of your clinical services and patient population?

• Accuracy. Is the indicator based on accepted 

guidelines or developed through formal group-decision 

making methods?

When choosing quality of care indicators, feel free 

to adopt indicators that your facility has already designated 

as a priority from the following websites: 

• National HIVQUAL Project (www.HIVQUAL.org)

• National Quality Center 

(www.NationalQualityCenter.org)

• New York State Department of Health AIDS Institute

(www.hivguidelines.org)

• Improving Care for People Living with HIV/AIDS

Disease. Institute for Healthcare Improvement, HRSA/

HAB. HIV/AIDS Bureau Collaborative. Order via the 

HRSA Information Center

(www.ask.hrsa.gov) or call 888-ASK-HRSA

• National Quality Measures Clearinghouse

(www.qualitymeasures.ahrq.gov)

Real-World Tip

Beg, Borrow, And Steal Indicators.

If you answer "no" to any of these questions, the indicator—

while still relevant to patient care—is probably either too 

difficult to measure or less than critical to patient care. On 

the other hand, if you answer "yes" to all of these questions, 

you have most likely found a viable indicator that will give 

you the most benefit for your measurement resources.

The Toolbox on page 45 shows several elements of HIV care 

and corresponding aspects which may be used as indicators. 
NYSDOH AI HIVQUAL Workbook September 2006

41

Annual Quality Goals

What are the priorities for the quality program?

Quality goals are endpoints or conditions toward which the 

facility will direct its efforts and resources during project 

work. Quality goals help staff focus on improving aspects of 

care. While an HIV program is able to measure several key 

performance indicators, the available resources for quality 

improvement work might limit the HIV program to conduct one to three quality improvement projects per year. 

The quality committee needs to work with staff and stakeholders to develop annual goals so that they are understood 

and embraced by everyone at the facility. Prioritization 

helps the facility to direct resources toward the issues that 

need it the most. The following three criteria can be helpful 

to a quality committee in prioritizing annual HIV-specific 

improvement goals:

• Frequency: How many patients received and how many 

did not receive the standard of care?

• Impact: What is the effect on patient health 

if they do not receive this care?

• Feasibility: Can something be done about this 

problem with the resources available?

For further description of how to establish annual HIV 

quality goals, see the Toolbox on page 46. 

Participation Of Stakeholders 

How will we get staff members and consumers on board?

If HIV quality improvement activities are to become a 

reality at a facility, provisions need to be outlined in the 

quality management plan for actively engaging staff and 

consumers, communicating information about quality 

improvement activities and providing opportunities for 

learning about quality.

• Engage staff and consumers. Gaining staff and consumer 

support for quality improvements requires capturing 

and integrating their voices. The needs and expectations 

should be understood and their feedback are reflected 

in the quality improvement management plan. To accomplish this, the quality committee should seek staff 

and consumer input to the extent feasible. Staff meetings 

and other informal one-on-one discussions are both 

appropriate methods. A short questionnaire might be 

developed and circulated. 

• Communicate information about quality improvement 

activities. It is important that staff and consumers know 

about the facility’s quality initiatives on an ongoing 

basis. A quality management plan should document 

how the facility will share information about its quality 

activities and project results. Options include providing 

Additional Resource

Measuring Clinical Performance: A Guide for HIV Health Care Providers is a step-by-step guide to measuring HIV performance. Excerpts 

of this publication are included at the back of this Workbook on page 142. If needed, you can refer to the information for details about how to 

select and measure quality of HIV care. You can download this publication at www.hivqual.org. 

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

42

updates at staff meetings, sharing quality committee 

meeting minutes, publishing newsletters, external committees and to consumers. 

• Provide opportunities for learning about quality. Because 

staff members ultimately bring the quality management 

plan to life, it is likely that staff will need to be educated 

in some basic quality concepts and skills. The quality management plan should describe how the facility 

intends to provide staff training and learning opportunities. Options include self-study of quality manuals and 

quality posters, or attendance at formal training sessions 

about quality. As appropriate, these learning interventions could be shared with consumers.

Evaluation

How will we evaluate our overall performance as a program?

Performance measurement provides the hard data about 

improvements to care delivery over time, but it is also important to assess how efficiently the program is operating as 

a whole. There are two areas to consider in evaluation:

• Quality improvement projects conducted during the 

year. The projects should be a worthwhile investment in 

the facility’s quality of care and result in improvements 

that are sustainable over time.

• Effectiveness of the quality management plan. The 

quality plan should provide the vision and organization 

required to evaluate the effectiveness of the entire 

quality program.

At this point, the management plan only need to outline 

when a program evaluation will be conducted. Program Step 

III on page 69 provides additional information about how 

to actually conduct a quality program evaluation and who 

should be involved in the process.

The Toolbox on page 50 provides another example of one 

HIV facility’s quality management plan.

Additional Resource

Delbecq, A., Vand de Ven, A., and Gustafson, D. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. 

Middleton, WI: Green Briar Press, 1975/1986.
NYSDOH AI HIVQUAL Workbook September 2006

43

Toolbox: 

Quality Management Plan

Consider the following example of a quality management plan for ‘Campus Care Center,’ an ambulatory clinic serving over 

250 HIV+ adults located within a large academic hospital:

Quality Statement

The mission of the HIV Quality Management Program is to plan, assess, measure, and implement performance improvements in the systems and processes which affect the quality of care and services of the AIDS Center at the "Campus Care 

Center." The Quality Program strives to continuously improve the quality of care and services in a multidisciplinary team 

approach and is consistent with the organization-wide approach to quality improvement.

Quality Improvement Infrastructure 

The overall responsibility and leadership for the HIV quality program lies with the Medical Director, who authorizes the 

quality committee to plan, assess, measure, and implement performance improvements throughout the entire clinic.

The membership of the quality committee reflects the diversity of disciplines within the "Campus Care Center." The 

members of the committee include the Medical Director (who serves as chairperson), one medical provider, one nurse, 

one case manger, one peer counselor, and one support staff member. The chairperson will report back to the overall committee responsible for hospital-wide quality activities. Membership is approved by the Medical Director.

The quality committee will have at least 10 scheduled meetings per year, tentatively planned for the second Wednesday 

of each month from 8:30-9:30 a.m. Additional meetings may be called as needed. Minutes of meetings will be kept and 

will be distributed to each member of the committee and to all necessary hospital-wide committees. A written summary 

of the meeting will routinely be made available to staff and consumers.

Performance Measurement

We will measure the following quality of care indicators on a quarterly basis:

• HIV monitoring (viral load, CD4 count)

• Antiretroviral therapy management

• Treatment adherence

• Treatment readiness education 

• Access to expert HIV care 

• Opportunistic infections prophylaxis (PCP, MAC)

• Tuberculosis screening (PPD)

• Gynecologic care (pelvic exam with PAP smear, GC, chlamydia screening)

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

44

Toolbox: 

Quality Management Plan...Continued

• Substance use screening

• Dental and ophthalmology care

• Mental health screening and referral

• HIV patient education

Annual Quality Goals 

Based on last year’s performance rates, the quality committee prioritizes the following quality projects: 

Two quality improvement project teams will be initiated in order to improve 

• GYN rate (annual pelvic exam) to 90 percent or above and

• Annual Mental Health Screening of all HIV patients rate to 60 percent or above.

Participation of Stakeholders

One of the quality goals of "Campus Care Center" is to involve staff actively in the HIV Quality Program and its quality 

improvement activities. All staff members will be asked to participate in at least one quality improvement team. The 

participation in the quality program should be part of job expectations.

Findings of quality improvement activities as well as summary reports of quality committee meetings will be shared 

with staff to ensure open communication flow within the HIV program. Based on the belief that staff should be actively 

involved in the HIV quality program and its activities, all current and new staff members will receive the hospital’s quality manual of quality improvement methodologies and review key chapters during bi-weekly staff meetings. In addition, 

staff will be provided annually with a 2-hour training session about quality improvement principles and will receive the 

hospital’s quarterly newsletter on quality tools and techniques. 

Consumers will be invited to the 2-hour training session about quality improvement principles and will receive the 

hospital’s quality newsletter.

Evaluation 

At the end of the year, the quality management plan will be assessed against its goals. This will facilitate planning of 

future quality improvement plans. An annual organizational assessment will be performed using the 

most current HIVQUAL Orginizational Assessment Tool. Additionally, all quality improvement projects will be revisited 

and evaluated to learn more about future quality improvement projects regarding goal accomplishment, resource efficiency, and team collaboration.
NYSDOH AI HIVQUAL Workbook September 2006

45

Toolbox: 

Quality of Care Indicators

• CD4 cell count test performed every 4 months

• Viral load measurement test performed every 4 months

ELEMENT OF HIV CARE Quality of Care Indicator(s)

• Appropriate management of patients on ARV therapy based on whether 

the patient is deemed clinically stable or unstable

• Antiretroviral (ARV) therapy medication

• Treatment adherence to ARV

• Annual PPD screening

• Annual lipid screening

• Annual hepatitis (HCV) screening

• Annual mental health screening

• PCP prophylaxis (CD4<200/mm3)

• MAC prophylaxis (CD4<50/mm3)

• Annual pelvic exam

• Annual chlamydia test

• Annual PAP smear

• Annual gonorrhea culture

• Annual syphilis serology

• Annual assessment of substance use 

• Annual assessment of tobacco use

• Annual dental exam 

• Annual ophthalmologic care (CD4<50/mm3)

• Basic patient education

• ARV treatment education

• Comprehensive assessment 

• Service plan

• Follow-up on service plan goals and referrals every 4 months

• Coordination of services every 4 months

HIV Monitoring

Antiretroviral (ARV) Management

STD Management

Screening

Opportunistic Infection (OI) 

Prophylaxis

Gynecological Care

Substance Use

Coordination of Care

Patient Education

Case Management

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

46

Toolbox: 

Establishment of Annual HIV Goals

The following four steps may assist an HIV program to identify and establish annual goals for the HIV 

quality management plan:

1. Assess where you are. Analysis of historical performance data helps to identify areas of strength and weakness where improvement may be needed the most. Understanding the current status-quo most often leads to meaningful goals that both 

staff and stakeholders will relate to and support. Consider sources such as internal performance data, consumer satisfaction survey results, staff input, consumer advisory committee, or external benchmarks.

2. Understand your parameters. Identify the basic parameters that describe your HIV program and the community it 

serves. Putting together such a succinct description of the program, including the aspects of HIV care you currently 

deliver, the demographics of patients served, and the external expectations of funding/regulatory agencies helps to identify 

where to focus quality improvement efforts. 

3. Prioritize your annual goals. The assessment of past performance and the picture of the current environment give the 

quality committee the necessary information to brainstorm a list of annual HIV goals. A list of potential annual goals 

can be easily generated at a single quality committee meeting once an initial assessment is performed. The quality committee can then prioritize the goals. 

4. Quantify where you want to be. Annual HIV quality goals need to be measurable. Based on the information gathered 

in the previous three steps, the annual quality goals need to be restated in quantitative terms such as: "85 percent 

adherence to antiretroviral therapy for all HIV+ patients receiving HAART therapy" or "To reduce patient ‘no shows’ by 

15 percent."
NYSDOH AI HIVQUAL Workbook September 2006

47

Toolbox: 

Checklist for the Review of an HIV-Specific Quality Management Plan

• Provides brief purpose describing the end goal of the HIV quality program and 

a shared vision to which all other activities are directed; assume an ideal world 

and ask yourselves, "What do we want to be for our patients and our 

community?"

DESCRIPTION

The quality infrastructure includes the following elements:

• Leadership: Identifies who is responsible for the quality management initiatives

• Quality commitee(s) structure: Documents who serves on the quality commitee, 

who chairs the committee, and who coordinates the QM activities

• Roles and Responsibilities: Defines all key persons, organizations, and major 

stakeholders and clarifies their expectations for the quality management program

• Resources: Identifies the resourses for the QM program

• Quality goals are endpoints or conditions toward which quality program will 

direct its efforts and resources

• Selects only a few measurable and realistic goals annually (not more than 5); 

uses a broad range of goals

• Indicates that those annual goals are established priorities for the QM program

• Establishes thresholds at the beginning of the year for each goal

COMMENTS

Quality Statement

Quality Infrastructure

DOMAIN IN QM PLAN

Annual Quality Goals

Participation of Stakeholders • Lists internal and external stakeholders and specify their engagements in the 

QM program

• Provides opportunities for learning about quality for staff

• Includes community representatives, as appropriate

• Specifies how feedback is gathered from key stakeholders

How to Use this Plan:

A Quality Management (QM) Plan defines a quality program’s strategic direction and provides a blueprint for upcoming 

improvement activities for the HIV program. While there is no universal “how-to” template for creating a quality management plan, this checklist outlines the basic domains that should be covered in each plan: Quality statement, Quality 

improvement infrastructure, Quality Plan Implementation, Performance measurement, Annual quality goals, Participation of stakeholders, Evaluation, Capacity Building, Process to update the Plan, and Communication. This checklist can 

be used to assess your quality management plan for completeness and to identify areas which need to be incorporated into 

the plan. Keep in mind that this checklist should be used as a reference and assessment tool and that the most important 

step is to get started. 

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

48

Toolbox: 

Checklist for the Review of An HIV-Specific Quality Management Plan...Continued

• QI capacity building of providers and spread QI performance measurement 

systems and QI activities

• Identifies methods for QI training opportunities

• Provision of technical assistance on QI and support for QI activities

• Indicates how data are being fed back to providers and key stakeholders

DOMAIN IN QM PLAN DESCRIPTION

• Evaluates the effectiveness of the QM/QI infrastructure to decide whether to 

improve how quality improvement work gets done.

• Evaluates QI activities to determine whether the annual quality goals for quality 

improvement activites are met

• Reviews performance measures to document whether the measures are appropriate to assess the clinical and non-clinical HIV care

COMMENTS

• Specifies timelines for implementation to accomplish those goals - workplan

• Specifies accountability for implementation steps

• Provides milestones and associated measurable implementation objectives

• Identifies routine schedule to at least annually update QM plan

• Specifies accountability - indicates who will initiate process to 

update/revise plan

• Indicates a sign-off process to finalize plan; potentially include internal/external 

stakeholders; include signatures of key stakeholders

• Outlines process to share information with all stakeholders at appropriate intervals

• Identifies format of communication

• Identifies communication intervals

• Clear and easy to follow layout and organization of content

• Clear dating of document, including date of 'experation'; page numbers 

• Identifies and quantifies the critical aspects of care and services provided in the 

organization; ensures integration with other titles or accrediting bodies, GPRA, 

Program Assessment Rating Tool (PART) measures and unmet need

• Identifies indicators to determine the progress of the QM program

• Indicates who plans to develop collect, and analyze data

• Indicates who is accountable for collecting, analyzing, and reviewing 

performance data results and for articulation of findings

• Includes strategies on how to report and disseminate results and findings; 

communicate information about quality improvement activities

• Processes in place to use data to develop new QI activities to address 

identified gaps

Evaluation

QM Plan Implementation

Performance Measurement 

Capacity Building

Process to update QM Plan

Communication

Formatting
NYSDOH AI HIVQUAL Workbook September 2006

49

Strategize To Implement The Quality Plan.

An annual workplan benefits the quality implementation 

efforts by:

• Clearly documenting the necessary steps to implement 

the quality management plan.

• Assisting the quality committee to allocate the appropriate resources essential for quality activities, including 

project teams, staff training, data collection, and 

evaluation efforts. 

• Effectively communicating quality activities to 

staff and stakeholders.

• Creating a template to monitor the implementation 

process of the quality management plan.

Simply stated, an annual workplan answers the questions of 

what, when, where, and how a quality management plan is 

implemented. Although there are different approaches writing this workplan, a template should include, at minimum, 

the following categories:

• Major quality goals. A straightforward goal statement 

divides the workplan into categories under which several 

activities are noted to accomplish each goal. 

• Quality activities. Each activity is briefly explained. The 

documentation should be informative and concise at the 

same time, to keep the form practical and user-friendly. 

• Responsibility. A staff person or team is identified 

to oversee and report back on the implementation 

of each activity. 

• Date of Completion. The duration and/or date by which 

each activity is completed should be noted.

During the evaluation stage, the quality committee can use 

this workplan to assess the implementation efforts by the 

facility. Additionally, it generates a template for future planning efforts and workplans. 

The Toolbox on page 52 illustrates an annual workplan by 

one HIV facility which outlines several goals, related activities, the staff person responsible for implementation, and the 

date of completion.

On page 102 of this Workbook, the process of developing 

a Gantt chart is described in detail. This tool is helpful 

for many programs to visually display project goals 

and milestones.

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

50

Toolbox: 

Quality Management Plan from a Large Hospital - Based Facility 

ALBERT EINSTEIN MEDICAL CENTER

The Immunodeficiency Center

Quality Assessment and Improvement Plan

Fiscal Year 2002

1. Mission Statement

The purpose of the Quality Improvement (QI) Program is to ensure that all patients of the Immunodeficiency Center (IDC) are receiving healthcare as 

recommended by the Department of Health and Human Services (DHHS), The International AIDS Society of the United States of America (IAS-USA), 

and the United States Center for Disease Control (CDC). We will utilize QI principals and methodologies as a basis for improvement of care and 

services. By identifying opportunities for improvement, collecting and analyzing data, developing and implementing plans and subsequently evaluating 

those plans we can continuously improve the processes and systems that influence patient outcomes. This QI Program is consistent with the organization-wide approach to quality improvement at Albert Einstein Medical Center.

2. Priorities for IDC QI Program 

• Establish an organizational structure within the IDC that supports QI. This includes but will not be limited to: membership, frequency 

of meetings, roles and responsibilities, resources, and reporting systems.

• Adopt standards of care set forth by the DHHS, IAS-USA, and the CDC.

• Track clinical outcomes to ensure standards of care are being met for all patients of the IDC.

• Educate IDC Staff about QI methodologies and techniques through training sessions.

• Facilitate the active involvement of IDC staff in the quality program and its quality improvement activities.

• Ensure that QI activities are routinely conducted to continuously improve the quality of care and services.

• Facilitate communication among Administration, Department of Medicine, and IDC staff on findings, conclusions, actions, and resolution of 

performance improvement issues.

• Participate in the Albert Einstein Medical Center Quality Improvement Program through the Performance Committees of the Departments of 

Medicine and the Immunodeficiency Center.

• Document and report performance improvement activities.

3. Scope and Organization

The Immunodeficiency Center (IDC) at The Albert Einstein Medical Center provides full-service care for patients at all stages of HIV disease. The 

patient population served at present is approximately 250 in number and comes from various socioeconomic, ethnic, and racial backgrounds. Included 

among the services provided are: HIV primary care, HIV prevention and disease education, mental health counseling, nutritional assessments and 

education, and care access counseling. Service for referrals to Gynecology, Obstetrics, Psychiatry, Radiology, Oncology, Gastroenterology, Cardiology, 

Nephrology, Ophthalmology, Pulmonary, Neurology, Surgery, Orthopedics, Rehabilitation, and Rheumatology is available on site.

It is the goal of the IDC Quality Improvement Team to provide compassionate and comprehensive care in accordance with the guidelines mandated by 

the United States Department of Health and Human Services, the International AIDS Society of the United States of America and the United States 

Center for Disease Control. The Immunodeficiency Center’s quality program incorporates the values of the Albert Einstein Healthcare Network and is 

reflective of its stated mission. The IDC QI committee core membership consists of the Medical Director, Nurse Practitioners, and the CQI Coordinator. Other members of the IDC Staff such as Social Workers, Dietitian, Nurse Manager of the Community Practice Center, Medical Assistants and the 

Office Manager will be involved, as appropriately indicated. 

4. Statement of Authority and Accountability 

Hospital Accountability

IDC quality improvement activities are reported to the performance committee of the Department of Medicine and the AEMC Quality Management 

Department. These departments report to the Board of Trustees of the Albert Einstein Medical Center, which in return support The Immunode-ficiency 

Center’s QI program. The Board of Trustees is ultimately responsible for all quality related activities.

Immunodeficiency Center Accountability

The Medical Director, with the assistance of other staff, has primary responsibility for the quality of patient care provided as well as the daily operation 

of the IDC. The Medical Director reports to the Chairperson of the Department of Medicine. The Chairperson of the Department of Medicine appoints 

a Department Quality Representative, who is responsible for coordinating the development and implementation of the Quality Assurance Improvement 

Program (QAIP). The Quality Management Department assists the development and implementation of the Department of Medicine Quality Assurance 

Improvement Program (QAIP).

The IDC has a CQI coordinator who is responsible for the coordination of all QI activities between the hospital, the Department of Medicine and 

the IDC. The CQI Coordinator is accountable to the Medical Director of the IDC, the program administrator, and the Hospital Quality 

Management Department.
NYSDOH AI HIVQUAL Workbook September 2006

51

Toolbox: 

Quality Management Plan...Continued

Consumer Input

An important goal of the Immunodeficiency Center is meeting the needs of our patients and ensuring their satisfaction with our services. To achieve this 

goal, the IDC will incorporate feedback received from our Consumer Advisory Board (CAB) and suggestions culled from the distribution of periodic 

patient satisfaction surveys. Information will also be gathered from two monthly educational meetings that are held in the IDC conference room. 

QI Committee Structure and Function

The Immunodeficiency Center Quality Improvement Committee will be responsible for:

• Development and implementation of an annual Quality Improvement Plan.

• Overseeing implementation of team projects.

• Peer review of patient charts.

• Monitoring and measuring performance of service standards with regard to clinical treatment, case management, and related services.

• Educate team members in tenants of QI process and implementation.

The IDC Quality Improvement Committee will meet bi-monthly at a time that will allow attendance by all members. 

The CQI coordinator will compose an agenda consisting of:

• A review of minutes from previous QI meeting to ensure open issues have been or are in the process of being resolved.

• Review current findings of ongoing QI projects, such as newly internally or externally acquired data, problems, and corrective actions.

• Implementation and evaluation of QI team projects.

• Address all concerns of the Medical Director, QI Core Committee and QI team members.

• Suggestions and recommendations for new improvement projects.

• Minutes from QI Core Committee meetings will be recorded by the CQI Coordinator. Documentation of meetings and activities of individual 

QI projects will be the responsibility of that project's team leader.

CQI Team Structure/Function

CQI teams shall be selected at the discretion of the IDC quality committee. Team members will utilize their process experience and knowledge to: 

• Improve one quality aspect through routine meetings of a multidisciplinary CQI Team. 

• Use QI methodologies to investigate and improve the HIV delivery process.

• Document and report, internally and externally, QI team activities.

5. Key Indicators

The IDC will utilize the HIVQUAL QI program strategy to measure key indicators of HIV health care for patients. The program provides software that 

allows for computerized accountability of quality improvement. Clinical indicators that will be measured include, but are not limited to:

• CD4 cells and viral loads

• Pneumonia and flu vaccines

• PAP smears and GYN care

• Prophylaxis for PCP, MAC

• PPD screening

• Antiretroviral therapy

• Referrals to specialists, specifically Ophthalmology, Dermatology and Gynecology

The Immunodeficiency Center will also track medication adherence, emergency room use, annual RPR’s, eligibility and administration of Hepatitis 

vaccines, and patient no-show rate.

Data Collection

Quarterly chart review will be conducted to gather data on key indicators. The appropriate information will be captured on data entry sheets, which will 

facilitate transfer into the HIVQUAL database. The data will be entered into the database by the CQI coordinator or assigned medical assistant. The 

responsibility for generating all reports for review or analysis will fall to the CQI coordinator. Reports will be presented to the 

hospital QA Department and the Department of Medicine quality committee.

6. Improvement Evaluation

All projects will be evaluated on a quarterly basis (until improvement of key indicators reaches an 85 percent compliance level). If the level of 

performance indicators measured does not improve from baseline, continued monitoring and intervention is indicated. IDC interventions include: 

training and education of team members, revision of present clinic policies, or development of new policies. When a measured indicator reaches a satisfactory level of improvement, the project will be discontinued. Periodic monitoring of discontinued project indicators will be reviewed 

to ensure continued compliance with agreed upon thresholds.

Updated by: Patrick Coady

Date of approval: March 31, 2003 Date(s) modified: July 17, 2003

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

52

Toolbox: 

Annual Quality Management Workplan

Revise HIV quality management plan

Goal: Establish an effective HIV quality management program

2005 - 2006

activity who jun jul aug sept oct nov dec jan feb mar apr may

Develop annual quality workplan

Prepare planning information 

(data collection, program assessment/evaluation, organizational 

priorities, HRSA grant) – for bimonthly meeting.

Review HIV quality management plan. 

Make changes if needed 

Discuss and set annual goals

Monitor implementation of plan 

Revise as needed

Evaluate quality management program

Quality improvement project teams – at 

bimonthly meetings

Program goals

Annual organizational assessment

John

John

Mindy

Committee

Mindy

John

Committee

Committee

Committee

X X

X X X

X X due

X due

X X due

X X X

X

X X X X X X

X X X

X

Determine and define quality indicators

(clinical, non-clinical) measured thru 

HIVQUAL

Goal: Establish ongoing data collection and reporting to support performance measurement

2005 - 2006

activity who jun jul aug sept oct nov dec jan feb mar apr may

Collect and analyze data on indicators

Committee

Train data abstractors

Validate results

Review with Quality Committee

Conduct HIV patient satisfaction survey 

Adapt survey and brief test cycle

Revise survey, if necessary

Review with Quality Committee

Share and report results

Prepare findings

Share with staff 

Share with Board

John

John

John

Committee

Committee

Committee

John

Alex

Alex

Mindy

Mindy

Mindy

X X

X X X X

X X

X

X X X X

X X

X X

X

X X

X

X

X X

X

Committee
NYSDOH AI HIVQUAL Workbook September 2006

53

Recognizing different service delivery models for HIV care, the following table highlights some nuances for several delivery 

systems how to best develop and plan a quality management plan. 

SMALL HIV PROGRAMS

• Given the reduced number of quality committee members and the likelihood that the same members will conduct quality 

activities, envision a quality structure that accommodates participation. Piggy back on the existing meeting structures, and find 

informal ways to meet (but do not forget to document).

• Find a balance between available resources and goals for the quality program. Although the same elements of a quality 

management plan (quality statement, quality improvement infrastructure, performance measurement, annual quality goals, 

evaluation) apply for smaller HIV programs, make sure that each element is covered in your plan. Be realistic when establishing 

the goals and objectives for your program.

• Make the quality committee membership as inclusive as possible. Include stakeholders such as providers from referral services and 

consumers on your committee.

• Be sure to document your quality activities and committee meetings to ensure consistency beyond staff turnover.

HIV NETWORKS AND MULTI-SITE AGENCIES

• Identify and include representatives from the entire network on your quality committee so that they can voice their input in the 

development of the quality program. This will create a sense of ownership when outlining the quality expectations.

• Find strategies to best communicate results and updates about the quality program. Share minutes, reports and findings with all 

sites in the network. Rotate site presentations of their quality activities at committee meetings. 

• Create a sense of collaboration beyond the individual HIV facility. Promote benchmarking and sharing best practices among sites. 

• In addition to clinical measures, identify indicators that assess the network performance regarding communication, sharing of 

information, data collection, etc.

• Establish consistency across the network regarding data collection, data reporting, and methodologies to conduct quality 

activities while allowing individual providers to address their unique challenges. 

• Rotate quality committee meeting locations among network members. 

Toolbox: 

Different HIV Program Models of Care

Develop QM Program
NYSDOH AI HIVQUAL Workbook September 2006

54

ADVANCED HIV PROGRAMS

• Identify ways to include consumers in the quality improvement structure. Educate and promote consumers to become members 

on the overall quality committee. Invite consumers to participate in project teams.

• Gather routine feedback from consumers to shape your quality program. Create a routine communication link to your Consumer 

Advisory Committee (CAB). Hold frequent focus groups with them and sincerely follow-up on their recommendations. 

• Hire a full-time staff person for quality improvement in your HIV program. 

• Establish stronger links with other advanced HIV programs around the country to share experiences. 

• Include all staff in the quality program. Conduct a survey with all staff to identify barriers to quality. Ask staff which quality 

projects should be prioritized. Create ‘competition’ among staff (e.g., two quality teams work on the same quality aspect).

• Create ways to rapidly and systematically respond to quality findings.

• Select non-clinical and service-oriented measures in addition to clinical indicators.

Toolbox: 

Different HIV Program Models of Care...Continued
NYSDOH AI HIVQUAL Workbook September 2006

55

Program Step II: Facilitate Implementation of 

HIV Quality Program The Big Picture

During Step I of the Program Cycle, the HIV quality 

program committee develops a quality management plan. 

During Program Step II, the HIV quality committee helps 

to establish systems to make the plan a reality and gives 

those staff members who are involved in quality activities 

the resources and support necessary to succeed.

What To Do

• Establish performance measurement systems.

• Review performance data. 

• Establish improvement project teams.

• Support quality improvement activities.

Program Cycle

Project Cycle

Setting Things In Motion: Implementing The Plan

Moving from paper to action is the first order of business of the 

HIV quality committee. Sometimes, after completing the quality 

management plan, quality committee members feel that their work 

is done and that they need to get back to their “regular” work. 

Sometimes, they feel overwhelmed by the tasks outlined in the 

plan, or discouraged by their initial level of performance.

It’s important to keep the momentum generated through the planning process. Keep in mind, ‘Quality is a Journey,’ and requires 

many small steps. It’s useful to build some excitement about the 

quality management program and to identify some activities 

that can be successfully accomplished. Create an “early success” 

which everyone can celebrate. The HIV program at Kings County 

Hospital, NY routinely shares their data reports not only to staff at 

the bulletin board but also in the waiting room for everyone to see. 

These steps generated the necessary buzz and allowed the quality 

committee to routinely report on their progress.

Snapshot of HIV Care

Facilitate Quality 

Program
NYSDOH AI HIVQUAL Workbook September 2006

56

Snapshot of HIV Care...Continued

Make the committee work as routine as possible

Quality managers have also found it useful to establish standardized formats for committee materials and meetings. Making the 

order of the meeting and communications consistent help the 

meetings move efficiently because everyone understands how the 

meeting will proceed and to analyze the materials. 

To be flexible in their approach to improve HIV care and avoid 

long meetings, one HIV program decided to quickly meet and create simple ‘huddles.’ As one member pointed out: “We decided that 

we all had too many meetings every day. To sit down and meet, 

took too much time. So we agreed to huddle by simply standing 

around. Once we sat down, it was a meeting.” 

To maintain the flow from meeting to meeting, HIV programs 

decides before they adjourn from the quality committee meeting 

to quickly review the tasks and ensure that all members knew 

exactly their roles. “This simple step helped us all to keep 

everyone on track.”

Support committee members

Sometimes, in their implementation, quality committees discover 

something they hadn’t considered when developing the program 

plan. For example, one agency found that before it could start to 

measure its performance, it realized that they had to do some 

training for the quality committee members at how to understand 

and look at data. 

“At first I thought that people were hesitant to review performance 

data because they were concerned about what they would show. 

But after talking with some committee members privately, I 

realized that they felt that they didn’t know enough about data to 

be able to talk about it in a meeting,” a quality committee 

chair reports. 

“So I asked one of our facility’s ‘data geeks’ to provide a basic 

presentation to the entire committee. She kept it simple, and 

provided some good tools. We also made it mandatory for everyone 

to attend, so not to single out people. It was also good to have a 

non-committee member do this because we encouraged people to 

ask her questions on their own.” 
NYSDOH AI HIVQUAL Workbook September 2006

57

Establish Performance Measurement Systems.

Define Performance Indicators

The quality committee needs to ensure that each routine 

performance indicator is sufficiently defined. The following 

should be considered:

• Complete definitions of indicators that are based on current guidelines (e.g., ARV treatment education: 

‘Patients who initiate ARV treatment or undergo a 

change in class of drug during any 4-month review 

period(s) will be evaluated whether treatment education 

has been provided.’)

• Frequency of data collection (e.g., monthly, bi-monthly)

• Efficiency of data collection

Select The Data Collection Method

Determine if any data related to your indicators have already 

been collected. If your data sources capture the patient-specific information you are measuring and can be queried to 

produce information in the specified time frames, you may 

substantially reduce your workload. Investigate all possible 

sources. If you find an existing data source, determine the 

ease with which you can generate specific patient-level data. 

More likely than not, you will need to collect at least some 

data on your own through chart reviews. 

The quality improvement management plan details the indicators selected for routine performance measurement and 

the annual goals for which project teams will be formed. 

To ensure that the quality committee has routine and 

accurate data on the facility’s performance, the following 

steps are necessary:

• Validate data systems

• Define performance indicators

• Select the data collection method

• Establish accountability for data collection

Validate Data Systems

The task of the quality committee is to find the most efficient way to collect information from your facility’s records 

and to draw a sample of those records for measurement. The 

most complete source of information on diagnosis, treatment, and clinical outcomes of care is the medical record. 

Different facilities store information in different ways. Some 

maintain paper medical records, others use software to 

create electronic medical records (EMRs). In addition, data 

can be gathered from billing systems, appointment books, or 

from areas such as the pharmacy or lab. 

Notes

Facilitate Quality 

Program
NYSDOH AI HIVQUAL Workbook September 2006

58

Toolbox: 

Performance Measurement Plan

ELEMENT OF HIV CARE Quality of Care Indicator(s)

• CD4 cell count test performed

• Viral load measurement test performed 

• Appropriate management of patients 

on ARV therapy 

• ARV therapy medication

• Treatment adherence to ARV

• PPD screening

• Lipid screening

• Hepatitis (HCV) screening

• Mental Health screening

• PCP prophylaxis (CD4<200/mm3

• MAC prophylaxis (CD4<50/mm3)

• Pneumococcal vaccination (10 years)

• Pelvic exam

• Chlamydia screening

• PAP smear

• Gonorrhea culture

• Syphilis serology

• Substance use screening

• Tobacco use screening

• Annual dental exam 

• Annual ophthalmologic care 

(CD4<50/mm3)

• Basic patient education

• ARV treatment education

HIV Monitoring

ARV Management

STD Management

Screening

Opportunistic Infection (OI) 

Prophylaxis

Gynecological Care

Substance Use

Coordination of Care

Patient Education

DATA SOURCE COLLECTION METHOD

Lab data, patient 

medical record, electronic medical record 

system

Patient medical record, 

electronic medical 

record, pharmacy data

PPD log, patient medical record, electronic 

medical system, lab 

data

Lab data, pharmacy 

data 

Patient medical record, 

electronic medical 

record

Lab data

Patient medical record

Referral tracking system, lab data, chart 

review

Chart review 

Every 4 months via 

lab data system

Every 4 months via chart 

review and pharmacy data

Annually through 

chart review 

Annually through 

lab data, pharmacy

Annually through chart 

review and lab data

Annually through 

lab data

Annually through 

chart review

Annually through 

chart review

Annually through 

chart review

Case Management Chart review Annually through 

chart review

• Comprehensive assessment

• Service plan

• Follow-up on service plan goals and 

referrals

• Coordination of services
NYSDOH AI HIVQUAL Workbook September 2006

59

Establish Accountability For 

Data Collection

The quality committee decides who will routinely and 

accurately measure the selected indicators. Usually, a staff 

person or a team will be assigned with this task. At minimum, the designated person or team should receive a brief 

training to review the measurement process and to fully 

understand each indicator on which data are collected. 

The Toolbox on page 58 provides an overview of a Performance Measurement Plan which outlines the various 

indicators, their data sources and collection methods.

Additional information about sampling and data collection 

can be found on page 142 of this Workbook. This section 

particularly addresses issues of :

• Constructing a population sample

• Designing a data collection tool

• Assigning and training abstractors

• Collecting data

• Validating results

Additional Resource

You can download a software program called HIVQUAL3 for free at www.hivqual.org. This software which includes clinical indicators based on 

clinical guidelines and case management indicators providers a means of measuring and reporting HIV care and services.

Keep the following rules in mind when the data collection 

process is planned and the performance measurement plan 

is under way:

• Keep your data collection as simple as possible.

• Be sure that data collection is limited to quality 

improvement activities.

• Reduce the sample size to a minimum (select 24 - 107 

records per review).

• Measure the high performing indicators less frequency 

(e.g., change from monthly to semi-annually).

Real-World Tip

Collecting Data.

Facilitate Quality 

Program
NYSDOH AI HIVQUAL Workbook September 2006

60

Review performance measurement data.

When developing the quality management plan, take advantage of any existing performance data for selected indicators. 

Using data in the development of the quality management 

plan is helpful for: 

• Problem identification: Reviewing data helps to verify 

staff’s assumptions and anecdotes about the nature of 

existing problems.

• Planning and goal setting: Performance measurement 

data can be used in developing the quality management 

plan for identifying the annual quality goals and activities.

• Focus: Data help to focus the efforts of the quality 

improvement activities. Often, data will give the quality 

committee what to address first.

• Buy-in: A preliminary look at the existing performance 

level can make problems visible and establish the need 

for change.

Data that are already available provide a quick and easy 

way to get started. Sources of existing performance data 

might include:

• Reports in HIVQUAL3 software.

• Reports from other clinical software programs.

• External audit results from regulatory agencies.

• Information collected through organization-wide 

quality activities.

• Written logs in HIV programs.

Once you’ve reviewed these data, determine which areas are 

in need of improvement and include these in your quality 

management plan and workplan.

Notes
NYSDOH AI HIVQUAL Workbook September 2006

61

Establish Improvement Project Teams.

Improvement projects are the vehicle by which staff members 

address the quality goals set forth in the facility’s quality management plan. Typically, a cross-functional group 

of staff members is assigned to each project. This helps 

to ensure that multiple viewpoints are represented in the 

improvement effort.

To the degree possible, the team should include those staff 

members who influence the project goal as well as those 

impacted by the goal. A broad representation strengthens 

the team’s ability to make informed decisions and signals to 

the organization that the program values input from those 

staff members who are most impacted by the project work. 

Someone who feels personally invested in a project is much 

more likely to actively seek its completion. If feasible, include 

staff members in the selection process. 

Creating The Basic Structure For Teams

The facility's quality committee needs to communicate the 

following information for each project:

• Define the quality goal for the project. This basic 

information frames the scope of the project and helps 

improvement teams launch their efforts. For example, a 

quality committee might describe a project goal as, "improve patient adherence to ARV therapy to 85 percent 

using a 3-day self-report." The project team starts with 

this goal. 

• Team members assigned to the project. In many clinics, the HIV quality committee decides who will be 

included on each quality improvement project team. At 

a minimum, the quality committee should select staff 

to serve as team leader and/or team facilitator for each 

quality improvement project.

• Project deliverables. Specifying deliverables (such as the 

improvement project memo, weekly written updates or a 

final report) clearly communicates the quality committee’s expectations for a project and ensures the committee and the improvement team are ‘on the same page.’

• Deadline for completion and any interim deadlines.

Clear deadlines for reporting back to the quality committee (such as weekly project updates and a final report 

in 5 weeks) allow an improvement team to efficiently 

Facilitate Quality 

Program
NYSDOH AI HIVQUAL Workbook September 2006

62

plan project activities. 

Notifying Team Members And Staff

The quality committee shares the basic project documentation 

with each staff member so that everyone is aware of the improvement project work. Every opportunity (e.g., staff meetings, internal emails and newsletters) can be used to publicize 

current and upcoming quality improvement projects.

Additional Resource

Scholtes, Peter R., Joiner, Brian L. and Streibel, Barbara J.. The Team Handbook. Second Edition. Madison, WI: Joiner Associates Inc., 1996.

Consider the following practices in putting together an 

improvement project team:

• Include at least one member of the HIV quality 

committee to a project team.

• Choose individuals who have previous experience to 

serve as team leader or team facilitator.

• Recruit a consumer to participate on a project team to 

voice concrete concerns and suggest improvements from 

the consumer perspective. Consumers can also help to 

promote system improvements to other consumers.

• Provide a training session or update to the team prior to 

initiating the quality improvement team.

• Provide a list of available projects and allow candidates 

to self-select. Or ask staff members to list the types 

of projects they are interested in and factor in 

their preferences. 

Real-World Tip

Build A Solid Base.

Notes
NYSDOH AI HIVQUAL Workbook September 2006

63

Support Quality Improvement Activities. 

The success of the quality management program and its 

activities requires the support the quality committee and 

the integration of quality into the facility’s HIV care 

delivery. Key techniques to create a quality culture in 

HIV programs include 

• Providing training and education

• Creating communication pathways

• Recognizing staff members efforts

• Demonstrating program successes

Providing Training And Education

Staff members are trained to provide HIV care, but not 

necessarily trained to improve the quality of care. Quality 

improvement has its own body of knowledge and skills that 

are necessary in completing project work and implementing quality management strategies. Core topics include the 

principles and techniques of quality improvement and the 

methodologies of making change. 

Staff members may be most familiar with the "workshop" 

format, where a facilitator presents information to a small 

group and opportunities are provided to apply the information in structured exercises. 

Additional Resource

For guidance in teaching small groups about the primary responsibilities of program facilitation, see the HIVQUAL Group Learning Guide "Facilitation of Quality Program" exercise. The exercise could also be used to kick off a new HIV quality committee’s initial facilitation efforts. You can 

download this publication at www.hivqual.org.

Informed staff members are better participants in quality 

activities. The following strategies can be used to keep 

staff up-to-date: 

• Provide a copy of this Workbook to all staff.

• Invite outside speakers to present their quality 

projects to staff.

• Create a training bulletin board in staff areas that highlights one quality tool concept per month. 

• Create and distribute a quarterly newsletter about quality or share articles on quality.

• Discuss key quality concepts during every staff meeting.

• Include information about the quality management 

program in new employee orientation and training.

• Start small. Focus on areas in which staff 

members have already asked for assistance.

Real-World Tip

Informing Staff About Quality Improvement.

Facilitate Quality 

Program
NYSDOH AI HIVQUAL Workbook September 2006

64

Toolbox: 

Quality Improvement Team Reporting Form

TEAM MEMBERS

TEAM REPORTING FORM

TEAM

DATE

Improving Patient Retention for Initial Medical Appointment 

John Guercio, Jenny Smith, Susan Rodriguez (team leader), Catrin Papros, SD Huang 

May 15, 2005

• Mailing out new orientation package to 

patients

PROJECT UPDATE

INTERVENTIONS (SINCE LAST REPORT) findings and results next step

• Patients appreciated mailing; contact 

information in many cases incorrect

• Need to get better contact information 

of patients to send out orientation 

packages

DATA POINTS

PROJECT OVERVIEW

PROJECT GOAL The team was initiated to increase the show-rate for medical initial appointments from 

currently 23 percent to 60 percent. 

The team collects the show-rate for medical initial appointments from the appointment 

system every month. Indicator is defined as the percent of patients with initial medical 

appointments being seen by medical provider on the day of appointment (no walk-ins).

• Update patient contact information at 

every visit

• Front staff updated contact information 

at every visit; special effort was made 

to get the entire contact information 

before initial visit

• Ask for additional contacts to get in 

touch with patient ("Who knows 

where you are?")

• New field in computer system was 

added to enter additional contact 

information; staff enters information 

in computer

• Staff needs to be reminded to enter 

additional contacts in computer

• Make reminder calls • Phone script for staff is used to make 

initial appointment; adjustments were 

made to shorten the script

• A new script needs to be written for 

initial medical appointments for contacting additional contacts

• Generate report of current contact 

information to review with patient at 

time of registration; write corrections 

onto the form for data entry into 

the computer.
NYSDOH AI HIVQUAL Workbook September 2006

65

Creating Communication Pathways

An open flow of communication between the quality committee and staff involved in quality activities helps to ensure 

steady progress toward established goals. The simplest way to 

endorse communication is to create an open door policy. 

Any reporting expectations or requirements will depend 

upon individual experience with the improvement process. 

Checkpoints should be set to review the team’s progress or 

when staff members are expected to report on their activities. 

If a more structured process is required, consider adding status updates to the agenda of quality committee meetings or 

regularly set time aside to discuss the project updates. If 

meetings aren’t possible, ask for a status report in writing and 

discuss it with team members over the phone or via email.

If a project has fallen behind, the committee should help 

and encourage staff members to identify and remove any 

project barriers.

An HIV quality committee ‘jump starts‘ improvement 

projects, guides teams in their efforts and keeps improvement activities aligned to the quality management plan. The 

following practices can help the committee to be effective:

• Appoint one quality committee member to

be directly responsible for a quality improvement team.

• Have the quality committee chairperson occasionally 

participate in improvement project team meetings to 

answer questions and provide input.

• Reward the progress of improvement teams by officially 

acknowledging their work or by throwing a ‘quality 

lunch’ for the next team meeting.

• Ask quality improvement team members to routinely update staff at internal staff meetings about team progress.

Real-World Tip

Support Quality Improvement Teams.

Notes

Facilitate Quality 

Program
NYSDOH AI HIVQUAL Workbook September 2006

66

Toolbox: 

Data Follow-up Form

1) WHAT ARE THE MAJOR FINDINGS OF YOUR DATA ANALYSIS?

• Overall substance use screening score is 78 percent.

• Documentation of past substance use history is 75 percent.

• Referral for active substance users in only 20 percent.

2) What are the immediate steps for follow-up resulting from your analysis?

• Share report with all staff; make individual copies and present at next staff meeting.

• Post data points in hallway.

• Present findings to HIV leadership.

a) Sharing of Data

• Ask leadership for input how to improve referral system for active substance users.

• Ask case managers for their input to improve system.

• Create quick questionnaire for staff about barriers for substance use referral.

b) Generating Ownership

• Identify patients in the review with no referral and arrange referrals.

• Create a list of active substance users and follow-up with them about referrals.

c) Individual Patient Follow-up

• Arrange meeting with substance use program to improve referral.

• Create new form to track substance use referrals.

d) Addressing Immediate Systems Issues

3) What are your overall plans for systemic improvement?

• Propose a project team to be formed to address this issue.

• Increase reporting frequency of this indicator to monthly.

• Invite representative from substance use program to the overall quality committee.

The following Toolbox provides a framework for immediately taking action steps once the data results are presented to the quality committee project team. 
NYSDOH AI HIVQUAL Workbook September 2006

67

Recognizing Staff Members’ Efforts

Staff members are the backbone of the HIV quality 

program. Recognize their efforts regularly to reinforce the 

importance of their quality improvement project work. For 

example, write letters of recognition and post them in clinic 

locations where staff, patients and stakeholders will see 

them. Send an email of recognition to staff. If budget allows, 

offer small gifts such as movie passes or gift cards to team 

members upon completion of successful quality improvement activities. Or host a lunch where patients give testimonials of how a particular improvement project directly 

benefited their health. 

Demonstrating Program Successes

Build excitement for the HIV quality program by publicizing its success stories. Recognize team results formally 

at board meetings, full-staff meetings, conferences, and 

in reports to internal and external councils or committees. Publish successes in internal newsletters and journals. 

Mount success storyboards in the waiting room so patients 

are aware of quality improvement efforts. Look for ways to 

show how quality improvements affect every aspect of the 

staff’s daily activities.

Keep in mind that achieving quality improvements in 

HIV care is multifaceted. Quality successes in HIV care 

result from top-down commitment and empowered staff 

at all levels. Work to build a sense of staff ownership for 

the quality program.

Quality improvements become part of an HIV facility’s daily 

routine. All staff members, regardless of job titles or positions, should think of themselves as contributing to quality 

efforts and improving the care of people living with HIV.

Additional Resource

For guidance in teaching small groups about the importance of organizational support to an HIV quality program, see the HIVQUAL Group 

Learning Guide "Support for Quality Program" exercise. The exercise could also be used to plan specific tasks for how to build the program’s base 

of support. You can download this publication at www.hivqual.org.

Facilitate Quality 

Program
NYSDOH AI HIVQUAL Workbook September 2006

68

Recognizing different service delivery models for HIV care, the following table highlights how several delivery systems 

can facilitate the implementation of an HIV-specific quality program. 

HIV NETWORKS

• Establish a central topic to focus quality activities across the entire network (e.g., ARV management of unstable patients). Share 

results and findings with all network sites and benchmark across sites. Create ‘competition’ between teams.

• Create quality champions who can become internal resources for the network. Bring sites with similar quality activities together 

and showcase the success of your quality champions. 

• Identify a data collection system that consistently and regularly measures and compares the performance across the network. 

Train the data abstractors together. Use standardized data forms.

• Bring representatives of network sites together to jointly develop strategies to implement the quality plan. Train staff together 

about quality methodologies and techniques. Share policies and procedures. 

Advanced HIV Programs

• Provide a quick training on quality for team leaders and/or facilitators of project teams before engaging them in quality 

activities. Include consumers who participate in these activities in the training. 

• Develop a frequent quality newsletter to promote the successes of the quality program. Showcase successes of teams 

and individuals.

• Develop an award system for staff members which includes financial incentives for quality successes. 

• Train all staff annually (e.g., 1-day quality workshop) about quality improvement and the HIV quality program.

Toolbox: 

Different HIV Program Types
NYSDOH AI HIVQUAL Workbook September 2006

69

Program Step III: Evaluate HIV Quality 

Management Program.

Program Cycle

Project Cycle

Completing the cycle 

Reflecting on what you’ve accomplished many quality management 

programs—successful or unsuccessful—often never make it to the 

third and final step—assessing their program’s performance and 

identifying changes to make to improve their quality program. 

“We spend so much time looking at other people and their work 

that we just don’t have the energy when we’re done with our 

improvement projects to look at ourselves,” explained one quality 

Snapshot of HIV Care

The Big Picture

In the same way clinical aspects of care are routinely 

reviewed and changed, the HIV quality program needs to 

be continuously and systematically evaluated and improved. 

The HIV quality committee with participation of a crosssection of staff members evaluates the quality infrastructure 

and activities to assess what worked well, what did not, 

and why. The findings are incorporated into future planning 

efforts to ensure that quality improvements continue 

and are sustained. 

What To Do

• Evaluate HIV quality program effectiveness.

• Evaluate project team performance.

• Learn and respond to past performance.

• Strategize to sustain the quality program.

manager. To assist with their own review, the quality committee 

asked a colleague, not on the quality committee, to help facilitate 

the evaluation process. “We realized we needed some outside help 

to make sure it got done.”

Another facility “turned the tables” and enlisted a couple of staff 

who had not been involved with the quality program to conduct 

its own “independent” evaluation of the quality program. Armed 

Evaluate 

QM Program
NYSDOH AI HIVQUAL Workbook September 2006

70

Snapshot of HIV Care...Continued

with the quality management plan, workplan, committee agendas, 

minutes and project team materials, they presented their findings to 

the committee, staff, consumers and board at both a meeting and 

through a storyboard posted where the project teams show their 

information. “It was scary, but incredibly flattering—they used our 

materials and methods to help us improve our own 

quality,” the quality manager reported.

One quality committee chair when asked about how her committee 

had evaluated its plan responded that “we never again looked at the 

plan once we approved it.” To ensure that the quality management 

plan is reviewed, the implementation plan should routinely include 

its evaluation. Either a special meeting can be scheduled or a regular meeting of the quality committee can be used.

Learn from past performance

The University of Pittsburg Medical Center learned through their 

annual evaluation that they needed to find new ways to communicate their quality work. They sought staff input into their annual 

assessment and the responses indicated that staff were not aware 

enough of the program to provide the desired feedback—this 

lack of feedback indicated that the quality program needed to 

better communicate its efforts and to involve more people in 

the quality project. 

To respond to this need, a “buddy system” has been established—

each quality management committee member has a “buddy” to 

inform about the committee’s work and to attend meetings if they 

are not able to. Sharing the quality committee’s work throughout 

the program year helps to build knowledge among staff about 

the quality efforts so they feel equipped to evaluate the quality 

program when asked.

Build support for your quality program

The evaluation process can be used to also garner support for the 

quality program and to help sustain it. The evaluation process can 

reveal examples of how the quality program has lead to improvements in the facility’s other systems. For example, other committees and departments may model their own communication efforts 

on the quality committee’s sharing of information and creative 

displays of information. Consumer involvement in the facility’s 

improvement activities may increase as consumers are asked to 

participate in the quality management program.

Staff, board and consumers can be asked to provide their input 

through a variety of ways. One facility established a message board 

that encouraged staff and consumers to provide written comments 

on the quality projects. “We had quality committee members write 

a response to the comment and it was posted back to the message 

board for everyone to read,” reports the quality program manager. 

“It’s important that people see that we valued their input. We also 

observed that people began to read the responses and write new 

comments to us. It was a great way to provide a mechanism encouraging dialogue. Some of the comments have provided us with great 

ideas for quality projects, and others haven’t really been that helpful. Regardless, we always respond.”
NYSDOH AI HIVQUAL Workbook September 2006

71

Evaluate HIV Quality Management 

Program Effectiveness.

A quality committee needs to spend time thinking about 

how to evaluate the following key areas of quality management plan and the annual workplan which outlines the 

implementation process:

• Quality program infrastructure: Was the quality 

committee effective in its efforts to improve the quality 

of HIV care? Does the quality infrastructure require 

any changes to improve how quality improvement 

work gets done?

• Annual quality goals: Were annual quality goals for 

quality improvement activities met? How effectively did 

you meet your goals? What were the strengths 

and limitations?

• Performance measures: Were the measures appropriate to 

assess the clinical and non-clinical HIV care in the facility? Are the results in the expected range of performance?

• Staff and consumer involvement: Did the appropriate 

staff and consumers participate in quality improvement 

activities? Were staff informed about ongoing quality 

activities and about quality improvement methodologies? 

Were consumers informed about ongoing quality activities? See Snapshot on page 69 to learn how one facility 

kept everyone involved and informed.

• Annual workplan: Did the implementation 

process go as planned? Did you meet established 

milestones? What were the strengths and limitations? 

Designing A Program Assessment Tool

An assessment tool can be used to guide the evaluation 

process. If your organization has an existing assessment 

tool, see if it could be modified to help evaluate the facility’s 

quality management program. If not, consider adapting the 

HIVQUAL Organizational Assessment Tool on page 72 

to evaluate the HIV quality program and the Annual 

Quality Workplan on page 52 to evaluate the annual 

implementation process.

Additional Resource

For the entire HIVQUAL organizational assessment tool, including the scoring criteria, please visit www.hivqual.org.

Evaluate 

QM Program
NYSDOH AI HIVQUAL Workbook September 2006

72

Toolbox: 

HIVQUAL Organizational Assessment Tool

Functions

A.1. Does the HIV program have an organizational structure to assess 

 and improve the quality of care?

A.2. Were appropriate resources committed to support the HIV quality program?

score

A) Quality Structure 1 2 3 4 5

A.3. Did the HIV leadership support the HIV quality program?

A.4. Does the HIV quality program have a comprehensive quality plan?

B.1. Were annual goals established for the HIV quality program?

B.2. Does the HIV program clearly described roles and responsibilities for 

 the HIV quality program?

B) Quality Planning

B.3. Did the workplan specify timelines for the implementation of the HIV quality program?

1 2 3 4 5

C.1. Were appropriate quality indicators selected in the HIV quality program?

C.2. Did the HIV program routinely measure the quality of care?

C) Quality Performance Measurement 1 2 3 4 5

D.1. Did the HIV program conduct quality projects to improve the quality of care?

D.2. Were quality improvement teams formed to improve specific quality aspects?

D) Quality Improvement Activities

D.3. Are systems in place to sustain quality improvements?

1 2 3 4 5

E.1. Is the staff routinely educated about quality?

E.2. Does the HIV program routinely engage staff in quality program activities?

E) Staff Involvement

E.3. Are consumers involved in quality-related activities?

1 2 3 4 5

F.1. Is a process in place to evaluate the HIV quality program?

F.2. Does the quality program integrate findings into future planning?

F) Evaluation of Quality Program 1 2 3 4 5

G.1. Does the HIV program have an information system in place to track patient care 

 and measure quality?

G) Clinical Information system 1 2 3 4 5

(Scale: 1 = min 5 = Max)
NYSDOH AI HIVQUAL Workbook September 2006

73

Toolbox: 

Project Team Assessment Tool

Evaluation Question

GOAL ACHIEVEMENT

Comparison of the improvement project goal with:

• Baseline data

• Pilot test results

• Follow-up measurement data

information needed

To what extent did the improvement team meet its goal?

• Number of team meetings

• Frequency of team meetings

• Length of meetings

• Cost of project materials

• Number of team members

• Additional staff time/resources needed to sustain efforts

What resources were needed to achieve the goal?

• Comparison of expected vs. needed resources to achieve gain 

• Monitoring data on improvement indicators following 

project completion

Can the goal be attained more efficiently?

Interventions

• Statement regarding degree of goal achievement

• Description of how the interventions helped the patients

• Comments from staff implementing the interventions

How effective were the changes over the short term?

Description of the extent to which the program or other parts of 

the organization were involved in the interventions (e.g., which 

staff; how they were involved; quality committee involvement; 

policy changes)

How effective were the changes spread throughout the program?

Staff attitude toward the intervention; reasons why favorable 

or unfavorable.

How responsive was the program to making these changes?

TEAMWORK

Did the team composition help the achievement of 

the improvement objective?

Names of team members and disciplines/functions cross referenced 

with causal factors and staff attitude toward the interventions

Did team members have sufficient skill level in making quality 

improvements to direct their own efforts?

• Description of project results and the degree of 

goal achievement

• Previous exposure and training in quality improvements

• Degree of comfort with quality improvement tools

Did the team run effective meetings? Team members' attendance and participation at meetings

Evaluate 

QM Program
NYSDOH AI HIVQUAL Workbook September 2006

74

Completing The Program Assessment

"Who evaluates the program?" is a key question. To ensure a 

range of perspectives, the quality committee and a cross-section of staff members and other stakeholders can complete 

the program assessment. 

It is best to provide time for quality committee members to 

complete the assessment individually. Staff should also be 

invited to voice their assessment of the quality program, a 

simple survey for all staff members is an effective strategy to 

solicit their feedback. Upon completion of these assessments, 

convene a quality committee meeting to discuss the composite responses. Quality committee members should reach 

consensus about the findings and take appropriate actions.

During the process of evaluating the quality program, use 

the results of the quality program for future planning to 

shape an effective quality program and update the workplan. 

Additional Resource

For guidance in teaching small groups about the basic activities required to evaluate an HIV quality program, see the HIVQUAL Group Learning 

Guide "Evaluation of Quality Program" exercise. The exercise could also be used to gear staff members up for their own evaluation cycle. You can 

download this publication at www.hivqual.org.

Consider the following ideas to evaluate your HIV 

quality program: 

• Decide on the areas to evaluate first and then identify 

the information you need to collect.

• Reflect the needs of staff, patients and other stakeholders 

when designing the evaluation process and tools.

• Assign one or two quality committee members 

to draft the program evaluation tool.

• Dedicate a quality committee meeting to 

reviewing and discussing the evaluation findings.

• Keep the evaluation tool simple. 

• Make your evaluation cycle coincide with funding 

cycle reporting.

Real-World Tip

Plan For The Evaluation.

Notes
NYSDOH AI HIVQUAL Workbook September 2006

75

Evaluate Project Team Performance.

The purpose of project team evaluation is to assess how effectively project teams have succeeded in improving the delivery of care to HIV patients against the initial team charter. 

There are three main aspects to quality project evaluation: 

attainment of goals, effectiveness of interventions, and teamwork. Key questions to consider are:

• Goal achievement. To what extent did the improvement 

team meet its goal? Can the goal be attained more efficiently? Were the improvements sustained over time? 

• Interventions. How effective were the changes? How 

effective were the changes spread throughout the program? How responsive was the program to making these 

changes? What did we learn?

• Teamwork. Did the team composition help the achievement of the improvement objective? Did team members 

have sufficient skill level in making quality improvements to direct their own efforts? Did the team run 

effective meetings?

Designing A Project Team 

Assessment Tool

Project teams can create or modify an existing assessment 

tool. The Toolbox on page 73 shows an example of how a 

facility could gather information to assess the three areas of 

goal achievement, interventions and teamwork.

Completing The Project Team 

Assessment

Sometime before the end of the project—often at the last 

team meeting—all members of the improvement team 

should complete the project team assessment. 

The Toolbox on page 76 provides a modified version of a 

group interview technique that could be used to collect the 

evaluation information. It takes little time and displays 

information related to several of the questions listed in the 

Improvement Project Team Assessment Tool on page 73. 

The steps and instructions are written in an agenda format 

based on the assumption that it is a team meeting. 

Many facilities find it helpful to gather feedback from consumers about how the interventions affect their HIV care. 

Ensuring input from those affected by the modified care 

system will provide a complete picture of the project work. 

When the project assessment is complete, the improvement 

team should document the results and submit a copy to 

the quality committee. The quality committee can review 

and discuss all the project evaluation data and consider the 

impact for future planning.

The Toolbox on page 141 provides another approach for 

project teams to evaluate project team performance. 

Evaluate 

QM Program
NYSDOH AI HIVQUAL Workbook September 2006

76

Toolbox: 

Agenda for Project Team Assessment 

Date: October 22, 2005

Project team members present: John Waters, Michele Goss, Michael Sacks

Invited guests: Susan Rodrigues (Medical Director)

Purpose: To assess the effectiveness of the project team efforts

agenda

time

5 minutes Introduce the purpose of the meeting. Review the agenda.

activity

Discussion of project team performance.

 1. Brainstorm a list of factors that helped us achieve the goals of the project.

2. Brainstorm a list of factors that hindered us from achieving a higher performance. 

3. For each list, identify the most significant factors by voting.

4. Re-write a new list with the factors that received the most votes.

5. Discuss each factor in greater detail. 

30 minutes

Make recommendations to the quality committee on the following:

 • Future goal setting for this project.

 • Team effectiveness/efficiency for future teams.

 • Sustaining interventions.

30 minutes

5 minutes Summarize and discuss next steps.
NYSDOH AI HIVQUAL Workbook September 2006

77

Learn From And Respond To Past Performance.

Evaluation results can help the quality committee to identify 

critical issues for future planning. 

Reviewing Data

To maximize the lessons learned from evaluation data, the 

committee should review all program and project evaluation 

results and identify common themes or problems. Additional 

data (such as audit results or quality evaluations performed 

by external agencies) can be used to supplement the internal 

program and project evaluations and may help to identify future improvement opportunities.

Assessing Impact

In reviewing the data and linking them to overall 

program objectives, the quality committee assesses the 

impact on the HIV program and compiles a list of 

critical issues to be addressed during the upcoming annual 

planning process. The past performance of the quality 

management program can be used to learn lessons for future 

quality activities and how to best adapt the quality infrastructure. Those steps can include changes to the existing 

quality program by adapting annual goals and performance 

measures or changes to the annual workplan. It is critical to 

immediately take action while the evaluation results are still 

fresh in the minds of the quality committee. 

Notes

Additional Resource

Review the HIVQUAL Group Learning Guide "Team Self Evaluation" exercise as it describes the important opportunity for teams to discuss the 

positive aspects of their group interactions along with the areas that need improvement. In this way, team members help to build a "template" for 

success for future team projects for their own evaluation cycle. You can download this publication at www.hivqual.org.

Evaluate 

QM Program
NYSDOH AI HIVQUAL Workbook September 2006

78

Strategize To Sustain The HIV Quality Program.

The key to sustaining the HIV-specific quality program over 

time is to make quality improvement an integral part of the 

program’s daily acitivities. Taking appropriate actions to 

sustain the efforts of the quality committee helps to avoid 

the pitfall of ‘taking two steps forward, one step back.’ 

The quality committee employs the following strategies:

• Clear lines of accountability

• Spread the ownership of the quality program

• Communicate program success

• Routinely assess quality program

Clear Lines Of Accountability

The quality committee maps a clear structure of the HIV 

quality program, including the staff’s roles and responsibilities. Quality improvement activities are incorporated as 

part of everyone’s job description and job expectations. This 

increases staff awareness and links them directly into the 

quality program.

Spread The Ownership Of The Quality 

Program

Quality activities should be embraced by every staff member 

in order for the HIV quality program to be sustained over 

time. Explain that all staff members are part of the quality 

management program and, in order to succeed, the quality 

program takes root through their involvement. The goals 

and ongoing activities of the quality program are routinely 

communicated with staff.

Communicate Program Success

The key messages about quality should be doggedly repeated 

over and over, and reinforced at every opportunity. Communicating successes of the HIV quality program widens the 

program’s base of support. Techniques include:

• Develop a system for staff reward and recognition

• Champion staff through internal and external 

 promotion of success stories

• Showcase successes visually in facility

• Exchange "best practices" with other HIV programs 

Look for ways to create buy-in for ongoing quality of care 

improvements throughout the HIV facility:

• Identify patients who can provide testimonials about the 

effects interventions had on patient care.

• Routinely survey staff and consumers about 

their needs and review at the quality committee.

• Train all new staff members about quality improvement 

activities at the facility.

• Encourage staff members to become experts in quality 

improvement; publicly recognize their efforts.

• Involve staff in the decision-making and prioritization of 

future improvement goals.

Real-World Tip

Broaden The Support For Quality.
NYSDOH AI HIVQUAL Workbook September 2006

79

Recognizing different service delivery models for HIV care, the following table highlights how several delivery systems 

can evaluate the HIV-specific quality program. 

SMALL HIV PROGRAMS

• Devote one quality committee meeting to evaluate the entire quality program since the same staff members participate on the 

quality committee and on project teams. 

• Use pre-existing forms as much as possible (e.g., HIVQUAL Organizational Assessment Form – page 72) to 

facilitate evaluation process.

• Take advantage of the small size of the HIV program and plan future quality activities right away based on the evaluation results. 

Toolbox: 

Tips for Different HIV Program Types

HIV NETWORKS

• Create assessment forms that are consistent across the network for the sake of comparison and peer-learning. 

• Organize a special meeting with network representatives to assess the network performance including individual site performance; add measures to assess the performance of the entire network.

Advanced HIV Programs

• Present evaluation findings to the Consumer Advisory Committee and ask consumers for their input.

• Create a survey for all staff members to assess the quality program: Generate ideas for future quality management plans. Use 

findings to make adjustments to the quality management plan. 

Evaluate 

QM Program
NYSDOH AI HIVQUAL Workbook September 2006

80

Routinely Assess Quality Program

To effectively know whether the HIV program meets its 

goals, the quality committee routinely assesses the overall 

performance of the quality program. To keep this strategy 

alive, it is important to:

• Re-assess performance at pre-established intervals

• Review evaluation data regularly

• Respond to the findings in an immediate and 

systematic manner 

The Workbook describes in Project Step 6 on page 136 the 

details of how to sustain improvements made by quality 

improvement project teams.

Additional Resource

For guidance in teaching small groups about the importance of sustainability to an HIV quality program’s long-term success and how to improve 

sustainability, see the HIVQUAL Group Learning Guide "Sustainability of Quality Program" exercise. The exercise could also be used to document 

practical ideas for improving your program’s sustainability. You can download this publication at www.hivqual.org.

The following strategies are helpful to sustain gains made by 

the quality committee:

• Make quality improvement part of staff members’ job 

descriptions and expectations.

• Incorporate quality concepts into training of new

 staff members. 

• Provide ongoing quality training opportunities for 

existing staff.

• Provide opportunities for all staff to participate in 

quality improvement projects.

• Incorporate best practices identified during 

quality projects into daily processes.

• Reward successes of quality activities. 

Real-World Tip

Incorporate Quality Into The Fabric Of 

The Facility.

Notes
NYSDOH AI HIVQUAL Workbook September 2006

81

HIVQUAL Model:

Introduction to the Project Cycle 

Step 4:

Plan and test 

changes

Step 5:

Evaluate results 

with key stakeholders

Step 6: 

Systematize 

changes

Step III:

Evaluate

 HIV quality 

management 

program

Step I:

Develop and

plan a quality 

management 

program

Step II:

Facilitate

implementation 

of HIV quality 

program

Step 2: 

Develop a project 

team

Step 3: 

Investigate the 

process

Step 1:

Review, collect 

and analyze 

project data

Program Cycle

Project Cycle
NYSDOH AI HIVQUAL Workbook September 2006

82

Project Step 1: Review, Collect and Analyze 

Project Data. The Big Picture

Before HIV staff members can begin to orchestrate quality improvement work, the current performance level is 

assessed. If existing data systems do not provide sufficient 

information on a specific aspect of care, an indicator is 

defined and baseline data are collected. Once a decision is 

made to improve the performance, a project team, the main 

vehicle of the quality improvement activities is initiated.

What To Do

• Review and collect project data.

• Collect project data.

• Analyze and share results.

Project Cycle

Program Cycle

Snapshot of HIV Care

By the numbers: Collecting data to identify 

improvement opportunities

The quality improvement project cycle starts with determining the 

current level of the quality of care at the facility. The quality management plan may have incorporated performance measurement if 

data were available. If not, the quality committee needs to assemble 

the data it needs in order to focus its improvement efforts. 

Frequently, many facilities—both large and small—do not have 

adequate data systems to provide the needed initial performance 

measurement data. Often their response is “we need a new, facilitywide data system before we can even begin our quality improvement work.” While such data systems can support and facilitate 

quality improvement, it is not a necessity. Existing data collections 

methods can be used or modified to adequately collect performance 

measurement data.
NYSDOH AI HIVQUAL Workbook September 2006

83

Snapshot of HIV Care...Continued

The North County Health Services (NCHS) in San Marcos, CA 

used a diverse team to determine how to best measure indicator 

performance. They first explored the feasibility of adapting their 

existing data collection tools to capture their identified indicators 

and presented their findings to the quality committee. “The dialogue between the providers and the quality committee resulted in 

slight modifications to an existing data collection instrument that 

was well suited for staff implementation. The team easily figured 

out how to modify the tool to include several indicators suggested 

by the care providers,” notes Judy Brooks, Quality Consultant. 

Improving the facility-wide data system can be a long-term quality goal, but the lack of one should not be used as a reason not to 

proceed. Existing data collection tools and systems can provide 

the information needed to start quality projects. HIVQUAL3 

software or CAREWare can be used to collect and analyze 

performance data.

Presentation counts

Quality committees have learned that how and to whom data are 

presented is important. If staff have been reluctant to get involved 

in the quality program, effective presentation of performance data 

can be used to elicit an “Oh my gosh” moment—the realization 

that their facility’s performance is not what they had assumed and 

that quality improvement were needed. Often, it is not until the 

facility’s leadership sees the performance measurement data that 

they realize the need for a quality management program and quality improvement activities. 

“Showing where the facility is in relation to the national benchmarks has proven to be a great way to get the support you need 

from key managers,” advises a quality consultant. “Physicians 

and nurses are often data driven, evidence-based. It often takes 

display of the data to get them to acknowledge the facility’s actual 

quality of care. Unfortunately, it may take shocking or embarrassing them to get their involvement and support for quality 

improvement. Conversely, facilities that demonstrate high quality 

care want to maintain it. Data can be effectively used to help 

sustain quality care.”

Staff may be used to using their own impressions or anecdotes to 

drive their decisions. Performance measurement takes the “guess 

work” out of the process and ensures that quality improvement 

efforts are based on actual data. HIVQUAL consultants report 

a transformation in the way facilities operate: “Once you realize the 

power of having and using actual data to make decisions and 

set priorities, you’ll never go back to not having data for 

decision-making.” 

The quality management plan may need to address the need for staff 

training on data collection and data analysis. Many staff may feel 

that they don’t know how to read graphs or interpret tables. Providing some basic training may help overcome this “math anxiety.” 

Measurement is not an end. It’s the beginning.

Unfortunately, many quality committees stop their work once 

they’ve collected their initial data. “Performance measurement 

alone is not quality improvement,” states one HIVQUAL consultant. “However, if you want to do quality improvement, you 

have to do performance measurement.” The initial performance 

data “should be used by the quality committee to guide its 

improvement efforts.”

Review Project 

Data 
NYSDOH AI HIVQUAL Workbook September 2006

84

Review and Collect Project Data.

Before committing valuable resources for quality improvement activities, it is important to assess the current 

performance level of the selected aspect of care for the 

following reasons:

 • Validation: Reviewing data will provide staff with 

actual performance measurement, rather than relying on 

hunches or assumptions. 

 • Baseline information: Data at baseline provide critical 

background information at the beginning and will allow 

project teams to track progress over time.

 • Prioritization: The project team can use data to prioritize their quality improvement efforts.

If existing data are not available or adequate for the quality 

improvement project, then baseline data should be collected 

as part of the quality improvement project. Depending on 

your facility's information systems, these data can be collected using your existing data systems. If the quality committee decides that existing data systems do not capture what 

is needed for the quality improvement work, then data need 

to be collected manually using newly defined indicators.

Additional Resource

For additional background on performance measurement and how to select an indicator, refer to Measuring Clinical Performance: A Guide for 

HIV Health Care Providers. You can download this publication at www.hivqual.org.

To collect project data, three key steps are identified:

• Defining measures

• Collecting performance data

• Analyzing project data

Defining Measures

A challenge in making quality improvements in HIV care is 

to select specific quality of care indicators that are relevant 

to the improvement project and best represent the issues 

under review. A quality of care indicator is a carefully 

defined measure of a specific aspect of patient care quantifying how a facility provides patient care against standards of 

care. Generally, indicators are based on specific standards of 

care derived from guidelines issued by professional societies 

and/or government agencies. 

To initiate the selection process of potential indicators 

based on internally- and externally-developed guidelines, 

quality-related funding requirements as well as individual 

performance measurement ‘wish lists’ are generated. A broad 

representation of staff and leaders facilitates this process and 

secures support throughout the improvement project. 
NYSDOH AI HIVQUAL Workbook September 2006

85

For the final selection of indicators, all measures are 

prioritized each indicator based on the following four 

measurement criteria: 

• Relevance. Does the indicator relate to a condition that 

occurs frequently or has a great impact on the patients at 

your facility?

• Measurability. Can the indicator realistically and efficiently be measured given the facility’s finite resources? 

Will the indicator show the impact of changes?

• Accuracy. Is the indicator based on accepted 

guidelines or developed through formal group-decision 

making methods?

• Improvability. Can the performance rate associated with 

the indicator realistically be improved given the limitations of the facility and patient population?

If those who are responsible for the selection process answer 

"no" to any of these questions, the indicator—while still 

relevant to patient care—is probably either too difficult to 

measure or less than critical to patient care. On the other 

hand, if the team answers "yes" to all of the questions, they 

have most likely found a viable indicator. At times, more 

than one indicator is selected to best assess and to balance the 

core aspect under review.

Treatment guidelines for HIV and AIDS are available

from several online sources, including:

• New York State Department of Health AIDS Institute 

(www.hivguidelines.org)

• AETC National Resource Center (www.aids-ed.org)

• Johns Hopkins AIDS Service (www.hopkins-aids.edu)

• HIV/AIDS Treatment Information Service 

(www.aidsinfo.nih.gov)

• Infections Diseases Society of America 

(www.idsociety.org)

• HIV InSite (www.hivinsite.org)

Real-World Tip

Available Treatment Guidelines.

Once a list of potential indicators has been identified, the 

group further defines them. This is accomplished by writing 

the indicator in the form of a question (e.g., ‘Was the CD4 

count measured and the result documented in the past four 

months?’) to which there are a certain range of responses 

based on patient documentation (e.g., ‘yes’, ‘no’, ‘NA’).

At this step, it is important to define the measure by clearly 

documenting the ‘yes’ and ‘no’ responses. For example, a 

team can further define the criteria for ‘yes’ and ‘no’ reReview Project Data 
NYSDOH AI HIVQUAL Workbook September 2006

86

sponse options by specifying what kind of documentation is 

acceptable, the specific timeframe that is needed to perform 

the activity, or any other clinical (e.g., patient in care, CD4 

<200) or demographic (e.g., age, gender) parameters needed 

to make the measure as clear and precise as possible.

The Toolbox on page 88 shows examples of indicator definitions for a variety of indicators. 

The Toolbox on page 91 shows the minimum number of 

charts to review based on the patient population size.

Collecting Performance Data

Either quality management committee or project team 

members will need to identify data collection methods 

and design collection tools to measure the current level of 

performance. The most complete source of patient information on diagnosis, treatment, and clinical outcomes of care 

is the medical record. Team members will need to find the 

most efficient way to collect information from the facility’s 

records, to draw a sample of those records for measurement, 

and to select the person(s) who will collect the data.

The Additional Resources section on page 142 provides 

information that the quality committee or the project team 

need to complete this step. Consult the material for detailed 

information on how to:

• Construct a population sample. Data sampling allows 

the facility to make inferences about a total patient 

population based on observations of a smaller subset of 

that group (the sample), saving both time and resources 

during data collection. To select a sample population, 

the larger eligible population (measurement population)

must first be identified. Defining this measurement 

population requires identifying those patients who are 

eligible to be selected for the sample based on pre-established criteria (e.g., patients with two medical visits during the year) and those who are not eligible for selection 

(e.g., patients with CD4 >200).

• Design a tool. Based on selected indicators to assess the 

performance level, a data collection tool is created to 

assist and facilitate the data collection process.

• Train data collectors. Those who are assigned to collect 

data should be given an opportunity to review the measurement process. They should also be instructed on how 

data collection will contribute to the project team and to 

the facility’s overall quality management program.

• Collect data. Clinical data abstraction, the process of 

gleaning data from a larger data set, is achieved through 

record review and/or administrative review. With record 

review, a designated data collector directly collects data 

manually from individual medical records, whereas with 

administrative review, the individual gathers information 

from data previously collected in the facility’s administrative database or log.
NYSDOH AI HIVQUAL Workbook September 2006

87

• Validate results. Performance measurement data are only 

as good as the process from which they are collected. 

Steps should be taken to ensure the process works by 

assessing its reliability and effectiveness.

Analyze Project Data

Analysis of project data provides a starting point to determine whether care currently falls short, meets, or exceeds 

the desired quality level. By analyzing project data the quality management program is able to answer the following 

basic questions:

• What is the current level of performance?

• Will improvement in the current level make a 

difference in the quality of care at our facility?

Additional Resource

Carey, R.G., and Lloyd, R.C. Measuring Quality Improvement in Healthcare. New York: Quality Resources, 1995.

Keep in mind the following practices in reviewing 

performance measurement data:

• Use only as much data as necessary; more is not 

necessarily better.

• Train team members in data collection process.

• Realize that there is no ‘perfect indicator’ and agree 

early on the best indicator.

• Limit data analysis to the achievement of the 

identified indicators.

• Keep in mind that performance measurement is only 

the first step—quality improvement is about changing 

the current system, not just measuring.

• Communicate project data early on; don’t wait to get 

‘perfect results.’

Real-World Tip

Keep Data Measurement Simple.

Notes

Review Project 

Data 
NYSDOH AI HIVQUAL Workbook September 2006

88

	 	

	 	

	 	Toolbox: 

Examples of Indicator Definitions 

These are the HIVQUAL indicators for clinical and case management performance. Further details, page 152.

ARV Management

For patients receiving antiretroviral therapy, treatment is managed appropriately.

Adherence to ARV Therapy

For patients receiving antiretroviral therapy, adherence is discussed and measured every four months.

HIV Specialist Care

A consultation with an HIV specialist is provided every four months.

CD4 Cell Count

A CD4 cell count test is performed every four months for all patients, with the exception of those incarcerated, 

hospitalized and recently relocated during the four-month review period.

Viral Load

A viral load test is performed every four months for all patients, with the exception of those incarcerated, hospitalized 

and recently relocated during the four-month review period.

Lipid Screening

For patients receiving antiretroviral therapy, a lipid screen is performed every year. 

PCP Prophylaxis

For patients with CD4 counts <200 cells/mm3, appropriate PCP prophylactic therapy is prescribed.

MAC Prophylaxis

For patients with CD4 counts <50 cells/mm3, appropriate MAC prophylactic therapy is prescribed.

TB Screening

For patients without a history of previous TB treatment or positive PPD test result, a PPD is placed and results read every 

year.

Hepatitis C Screening

The patient’s Hepatitis C status is documented in the medical record. For HCV+ patients, alcohol counseling and HCV 

education is provided. The patient’s Hepatitis A status is documented in the medical record.
NYSDOH AI HIVQUAL Workbook September 2006

89

Toolbox: 

Examples of Indicator Definitions...Continued

Mental Health Screening

A mental health screening is performed during the review period. Assessment components include: cognitive function, 

screening for depression and anxiety, psychiatric history, psychosocial assessment, sleeping and appetite assessment.

Ophthalmology Exam

For patients with CD4 counts <50 cells/mm3, a referral for an ophthalmology exam is documented.

Annual Pelvic Exam

For female patients (>18 years and 13-18 sexually active), a pelvic exam and Pap smear is performed every year.

Annual Syphilis Serology

A serum syphilis screening (i.e., VDRL or RPR) is performed every year.

Annual Discussion of Substance Use

A discussion of substance use (and treatment) is provided every year.

Annual Discussion of Tobacco Use

A discussion of tobacco use (and cessation) is provided every year.

Annual Dental Exam

A dental exam is performed every year.

Comprehensive case management assessment

A comprehensive case management of client needs occurs within 30 days of initial client contact.

Case management service plan development

A case management service plan is developed with client participation within 45 days of initial client contact and is based 

upon needs identified in the case management assessment.

Case management follow-up on service plan goals and referrals

Case management follow-up regarding service plan goals, referrals and patient attendance at appointments is documented in 

the client record every 120 days. All goals and referrals identified in the service plan are addressed.

Case management coordination of services

Case management coordination of services (communication between the case manager and any health or social service pro 

vider) is documented on a quarterly basis in the client’s record.

Review Project 

Data 
NYSDOH AI HIVQUAL Workbook September 2006

90

Analyze And Share Results.

Baseline data analyses and results are shared with the HIV 

quality committee. Communicating baseline data to health 

care providers, consumers and governing boards at the facility helps everyone to better understand the data collection 

process and the meaning of the data.

Whenever possible, the summary of performance measurement data should include graphics such as tables or charts. 

Graphic displays help to convey outcomes at a glance. Text 

should be used sparingly for background and/or explanatory 

information. The level of detail provided through graphics 

and text will generally depend on the target audience.

Commonly used charts to graphically present performance 

measurement data include:

• Run chart: a graph showing measurements on the vertical axis against time on the horizontal axis

• Pie chart: a circle divided into wedges to show relative 

proportions; the sum of all portions equal 100 percent

• Control chart: a run chart with statistically 

determined upper and lower control lines drawn on 

either side of a process average; used to analyze different 

types of variations

• Histogram: a bar graph that shows the distribution 

(variation) in a set of data, illustrating how often 

different values occur

The Toolbox on page 92 shows examples of these types of 

charts. Most computer spreadsheet programs can be used 

to construct them.

After measuring the current performance level, it is important to use the results and subsequent analyses effectively:

• Begin the analysis with a few questions or 

hypotheses before spending too much time 

‘digging’ through the data.

• Limit the display of data results to summarize your most 

important findings.

• Display data graphically whenever possible.

• Publicize the results; post graphic displays in hallways 

and waiting rooms so that staff and patients can see the 

baseline and progress. 

Real-World Tip

Make Your Case With Your Results.
NYSDOH AI HIVQUAL Workbook September 2006

91

Toolbox: 

HIVQUAL Sample Size Chart

The following table indicates the minimum number of records for performance reviews based on eligible population. 

See page 142 for further information on sampling and data collection.

All

24

30

35

39

43

46

49

52

57

61

64

67

70

75

79

82

85

87

88

94

97

105

107

Total eligible

population

Up to 20

21 - 30

31 - 40

41 - 50

51 - 60

61 - 70

71 - 80

81 - 90

91 - 100

101 - 119

120 - 139

140 - 159

160 - 179

180 - 199

200 - 249

250 - 299

300 - 349

350 - 399

400 - 449

450 - 499

500 - 749

750 - 999

1000 - 4999

5000 or more

Minimum number of

records to review

Review Project 

Data 
NYSDOH AI HIVQUAL Workbook September 2006

92

Toolbox: 

Display of Measurement Data in Graphic Form

Control Chart: Waiting Time

Pie Chart: Did patient receive PCP prophylaxis?

7 per91 per2 per9/3/XX 9/4/XX 9/5/XX 9/6/XX 9/7/XX 9/8/XX 9/9/XX 9/10/X 9/11/X 9/12/X 9/13/X 9/14/X 9/15/X 9/16/X 9/17/X 9/18/X 9/19/X 9/20/X 9/21/X

Upper Control Limit

LOWER Control Limit

60 per50 

40 

30 per20 

10 

GYN 

exam not 

done

GYN 

exam 

done, but 

results 

not documented

Referral 

made, but 

GYN appt. 

not kept by 

patient

GYN 

exam 

offered 

but 

patient 

refused

Medical 

record 

not 

found

Other 0 perJan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

100 

90 

80 

70 

60 

50 

40 

30 

20 

10 

Run Chart: Annual PPD Rate

Histogram: Reasons for No GYN Exam in 

Medical Record

0 percent
NYSDOH AI HIVQUAL Workbook September 2006

93

Project Step 2: Develop A Project Team.

The Big Picture

Team members gear up for the project at hand during 

Project Step 2. They become familiar with their individual 

roles and responsibilities and determine how they will work 

together during the project. As a team they write an 

improvement project memo to define their direction and 

scope of work and they strategize about key tasks and timelines ahead of them. 

What To Do

• Establish project team dynamics.

• Write an improvement project memo.

Project Cycle

Program Cycle

Creating teams that work

“Not another committee meeting to go to.” A frequent response of 

over-worked, stressed health care providers. 

How can quality managers create quality improvement project 

teams that break out of the facility’s committee routine and 

establish quality improvement project teams that are effective 

vehicles for quality improvement? Quality managers report using 

many strategies.

Involve new staff and consumers in 

project teams

Involve new people. “Don’t focus on the ‘usual suspects’ when 

establishing a project team,” suggests on program manager. “Too 

often, we look to the same people to serve on committees. Because 

of they’re over-extended, we don’t get the type of participation we 

need.” Project team members should be selected because they will 

bring a needed skill or insight to the improvement process. “This 

isn’t about business as usual. It’s about changing the usual.” 

Snapshot of HIV Care

Develop 

Project Team
NYSDOH AI HIVQUAL Workbook September 2006

94

Snapshot of HIV Care...Continued

Administrative and support staff who aren’t usually involved in 

the management functions of the facility should be included in the 

project team. Their experiences and perspective may lead to an 

“oh my gosh” moment—the realization of how processes actually 

work in “real life,” 

Consumers can also be involved in project teams. You may need 

to provide some training and support to facilitate consumers’ 

effective involvement in the team, but their perspective can often 

be important to the team’s work. Try mentoring consumers in the 

role—pair an experienced consumer with someone who hasn’t 

yet been involved.

Establish expectations of participation. Project team members need 

to know their roles and responsibilities and understanding that all 

members are expected to participate. “We started off our first meeting by telling team members that everyone was going to be active. 

No benchwarmers here,” reports one project team leader. “While it 

was my responsibility to keep everyone on track, my team members 

also developed the understanding that if we all worked together, we 

were more likely to complete our work on schedule, implement our 

improvements, and get back to work!”

New leadership should be involved in the project teams. Don’t 

rely on the same individuals to provide leadership to the team 

– the Medical Director does not have to be the leader in each team. 

Establish a process for mentoring future project team leaders by 

purposively involving new people in leadership roles. Provide 

training and support so they can assume independent leadership 

roles in the future.

Focus the team’s work

Clearly outline the work of the team. Defining the goal of the team 

and creating a workplan gets everyone on-board quickly and creates 

an understanding of what needs to be done. However, the goal 

needs to be focused and achievable. Kay Scott at St. John’s Riverside Hospital, Yonkers, NY reports that their initial improvement 

goal was to “go global” and created a plan that had great ideas but 

“was too ambitious.” By starting with narrow and focused efforts, 

Scott reports that they were able to achieve much more than that 

had anticipated.

Multi-site facilities can create multiple teams—each working on 

their own project or working on the same cross-site project. Unity 

Health Care in Washington, DC, a multi-site community health 

center, has site-specific quality project teams that work on their 

projects. They participate in cross-site meetings to report their 

work and share their experiences. This structures works because 

it fits into the overall corporate structure where responsibility for 

clinic operations is decentralized to each site. 
NYSDOH AI HIVQUAL Workbook September 2006

95

Establish Project Team Dynamics.

HIV project improvement teams, vehicles of quality improvement activities, meld together the skills, experiences 

and insights of different staff. Each team will have some 

unique combination of people, improvement goals and 

performance measures. 

Successful outcomes of improvement projects result most often when a team has clear objectives to guide their activities, 

the necessary resources to complete project work, support of 

the quality committee, and the willingness of team members 

to learn from each other and maintain open communication 

with the quality committee, staff and consumers.

Defining Roles And Responsibilities

Effective team functioning becomes everyone’s responsibility. Team members should take time at the beginning of 

the project to get acquainted with team members’ roles and 

responsibilities and agree upon how the team will function 

to get work done.

• Team leader: Someone who fully understands HIV care 

delivery issues needs to be on the team as team leader. 

The person selected as team leader also understands the 

entire breadth of the improvement project so he or she 

can effectively plan and lead team meetings.

• Team facilitator: Team leaders may wear two hats and 

also serve as the team facilitator. Generally, the facilitator assists the team leader in planning meetings and 

developing agendas. A facilitator also tends to the 

meeting process, for example, ensuring that everyone 

participates, and helps keep participants on track with 

the agenda and scheduled times. 

• Team member: Team members reflect the range of functions and departments involved in the process being improved in order to build and maintain consensus from key 

individuals on the solutions to the problems. They should 

also have intimate knowledge of the process, personal 

involvement and interest.

The quality committee usually assigns staff members to serve 

as the team leader and team facilitator for a quality improvement team. Improvement team members can be selected by 

either the quality committee or by the team leader.

The Toolbox on page 97 chart further illustrates the roles 

and responsibilities for an improvement project team.

Develop 

Project Team
NYSDOH AI HIVQUAL Workbook September 2006

96

Preparing Teams To Do Work

In the initial phase of an improvement project, the team leader 

and/or facilitator serve as the driving force to build effective 

relationships between team members and ensure everyone 

understands the team’s assignment. 

The team leader and/or team facilitator ensure team members 

know each other and recognize how members can complement each other through their expertise and perspectives. 

Any educational training to successfully start the improvement project is provided by the team leader and/or team 

facilitator. Team members’ experience and the specific nature 

of the project will dictate the appropriate level of necessary 

team-building and training.

Additional Resource

For guidance in teaching small groups about the roles and responsibilities of project team members, see the HIVQUAL Group Learning 

Guide "Team Roles" exercise. The exercise could also be used as an ice breaker for the project team’s first meeting. You can download this 

publication at www.hivqual.org.

Whether an improvement project is small or large, simple or 

complex, the following suggestions can help a team to successfully complete a project: 

• Include a quality committee member on the team.

• Involve additional staff who is willing to participate; 

ask for volunteers.

• Rotate functions of the team.

• Start with a presentation of a successful quality 

improvement project.

• Generate enthusiasm and excitement at the first team 

meeting; explain why the work is important or how staff 

and consumers will benefit.

• Explain the potential for change and limitations early on 

to explain the project’s framework.

• Rotate the role of recorder and timekeeper among 

team members.

• Include an engaged consumer on the team and be proactive preparing the consumer. 

Real-World Tip

Launch A Team With Purpose.
NYSDOH AI HIVQUAL Workbook September 2006

97

Toolbox: 

Team Roles for an Improvement Project

Team leader: Lindsey Cabrelli, R.N., Director of Operations

Team member: Bruce Dixon, Medical Liaison (Team facilitator)

Angelique Martinez, Clerk

Michael Sappal, Case Manager

Lan Huong, Staff R.N.

Gloria, Consumer

Recorder and timekeeper: To be rotated among team members

Updated: October 31, 2005 by Cheryl March, R.N.

team responsibilities

Provide direction and focus to team activities

Team Leader Team Facilitator Team Member

Ensure productive use of team members' time

Represent team to clinic management and quality committee

Facilitate team meetings

Ensure balanced participation by all team members

Provide feedback and support to team leader

Suggest problem-solving tools and techniques

Offer perspective and ideas and participate actively

Adhere to meeting ground rules

Complete assignments on time

Support implementation of recommendations

Keep up-to-date on QI training, research and methods

Manage the team’s time

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Take and distribute minutes of meetings X

Develop 

Project Team
NYSDOH AI HIVQUAL Workbook September 2006

98

Write An Improvement Project Memo.

An improvement project memo serves as a project blueprint. 

Teams develop memos to help ensure that all members work 

toward the same goals according to a single set of operational 

guidelines. Completion of the improvement project memo 

is important to:

• Clarify and focus the team’s direction and scope of work.

• Create a standard document for communicating what 

the project is, what it intends to accomplish, when it is 

likely to be completed, and who is responsible for the 

project implementation.

• Refocus team efforts if a team gets stuck on a specific issue.

• Educate new team members coming on board during the 

project cycle and getting those individuals up to speed.

An improvement project memo typically includes:

• Problem statement

• Improvement goal

• Team leader and team members

• Other (resources, authority, frequency of reporting, 

ground rules)

The improvement project memo may evolve and change over 

time as new information and data analysis results become 

available or additional knowledge is gained. Consider the 

memo a ‘living document.’ 

For example, a team may increase a target improvement 

goal from 70 percent to 85 percent as information and data 

substantiate the new goal can be realistically achieved.

The Toolbox on page 99 provides a sample Improvement 

Project Memo that teams can easily modify to suit their 

specific purposes. 

Defining A Project Problem Statement

A problem statement describes the problem to be addressed. 

It should:

• Be stated in concrete terms—terms that clearly describe 

the problem to be addressed.

• Include quantifiable numbers that indicate the 

current level of performance (e.g., "Currently 

60 percent of women in the HIV clinic receive annual 

GYN exams.")

• Be relevant to HIV care and services provided 

by the facility.

The following fundamentals are helpful in drafting an improvement project memo:

• Ask for input from facility leaders or the HIV committee 

in writing the memo to gain their buy-in.

• If necessary, redefine the goals over time.

• Communicate the memo to others; email to all staff and 

post in visible places for consumers.

• Make it simple.

Real-World Tip

Keep The Memo Purpose Aligned.
NYSDOH AI HIVQUAL Workbook September 2006

99

Toolbox: 

Improvement Project Memo

COMPLETION DATE:

PROJECT START DATE:

INDICATOR:

October 22, 2005

April 15, 2006

PCP prophylaxis

Currently, only 65 percent of patients with CD4 count less than 200 receive appropriate 

PCP prophylaxis, compared to the state-wide average of 92 percent. In the last year the 

performance rate declined by 31 percent.

Problem statement:

improvement goal: The team will work to improve the clinic’s performance on this important prevention 

measure. The team should focus on increasing the number of patients with CD4 count less 

than 200 receiving appropriate PCP prophylaxis to 95 percent and above. 

Team Member: Ann Cavanaugh, C.S.W. (team leader)

Peter Brown

Paul Sabo, M.D.

Santiago Rodriguez

Helen Kearney

Cheryl March, R.N.

Other: (resources, authority, 

frequency of reporting, 

ground rules)

Team will be given time to meet.

There’s money for supplies or other similar expenses, but not for additional staff.

Mac Martin (MIS department) will be available to help with data analysis.

Team members should give a verbal report at the next quality committee meeting, 

November 15.

All team members should be on time and no excuses. 

Updated: October 31, 2005 by Cheryl March R.N.

Develop 

Project Team
NYSDOH AI HIVQUAL Workbook September 2006

100

Defining A Project Goal

Effective teams work with clearly defined goals. There is 

some debate on how high to set the goal: an achievable, 

realistic or a "stretch" goal. A basic guideline would be: set 

the goal and then continue making changes until the level 

is reached at which the effort expended is too great for the 

gain. In other words, the value of meeting the goal should 

exceed the cost of doing so.

A good project goal clearly sets forth a goal that is measurable and achievable. The following template can be used in 

formulating the goal statement: "The team will work to improve the clinic’s performance on the (name of the indicator) 

indicator. The team should focus on (action verb) the 

number of patients who (state the desired result). Currently, 

our performance is (percentage). The team should aim to 

improve this performance to (percentage)."

See the Toolbox on page 99 for a completed example of an 

Improvement Project Memo. 

Setting Team Expectations

Team members should discuss and agree upon ground rules. 

Ground rules or team norms are what team members can 

expect from each other. Examples include:

• Agree to start on time and end on time.

• Frequency of meetings, length and time.

• Regular attendance.

An example of team ground rules developed by one HIV 

clinic is shown in the Toolbox on page 101.

Because the improvement project memo serves as a project 

blueprint everyone on the team needs to review it and agree 

on the content. When the team has reached consensus about 

the improvement project memo, it is submitted to the HIV 

quality committee for final review.

The project team is ready to get started. A workplan is sometimes used to provide a road map for a team to complete its 

work. Consider developing a Gantt chart for the improvement project as illustrated in the Toolbox on page 102. 

Additional Resource

For guidance in teaching small groups about the purpose and primary elements of an Improvement Project Memo, see the HIVQUAL Group 

Learning Guide "Improvement Project Memo" exercise. You can download this publication at www.hivqual.org.
NYSDOH AI HIVQUAL Workbook September 2006

101

Toolbox: 

Team Ground Rules

Members of the Performance Improvement Project Team agree on the following:

1.

2.

3.

4.

5.

6.

7.

8.

Meeting will start and end on time. 

All opinions will be considered with respect. 

Meetings will last 45 - 60 minutes and not longer! 

Meetings will be held in the group room. 

Meetings will be held in an atmostphere that will encourage team members to share their perspectives. 

Team members given assignments between meetings will complete their task on time. 

All team members will attend meetings on a regular basis. 

Team members will inform other team members one week in advance (if possible) of unavoidable absence from scheduled meetings.

Develop 

Project Team
NYSDOH AI HIVQUAL Workbook September 2006

102

Gantt Chart Example: Self Management Program

Activity Responsibility

Mary

Mary

Pat

Hal

Pat

Mary

Pat

Jack

Mary

Pat

weeks

1 2 3 4 5 6 7 8 9

Collect current written materials about self-management

Prepare presentation and draft educational materials

Discuss at quality management committee

Discuss with consumer advisory board

Establish team with providers and consumers

Formulate clear objectives and staff expectations

Present at staff meeting

Implement self-management program with one provider

Establish indicator to measure performance

Review new self-management program articles

x x

x x

x

x

x x

x x x

x

x x x x x

x x

x x

Implementation Plan of a Self-Management Program to empower patients to become informed and active 

participants in our health care delivery system.

Toolbox: 

 Gantt Chart

Description • Graphically displays activities (including roles and responsibilities) in sequential order plotted over time

• Depicts when each activity in a project must start and finish

• Shows which activities can be accomplished concurrently

• Shows the relative amount of time required to complete an activity 

• Presents the minimum time needed to complete an improvement project

use • To plan an improvement project

• To monitor the progress of an improvement project

Basic Construction 1. Identify all the activities that are necessary to complete an improvement project. 

(Note that these activities are not necessarily related.)

2. Identify the time required for each activity.

3. Identify the sequence of activities (e.g., which ones must be finished before another can begin and which 

can occur simultaneously).

4. Construct a horizontal timeline along the top axis; mark the appropriate scale for the project duration 

(e.g., days, weeks, or months).

5. List the project activities on the left vertical axis in order along with responsibility.

6. Mark the period of time from the planned beginning to the planned end for each activity.
NYSDOH AI HIVQUAL Workbook September 2006

103

The Big Picture

After defining and documenting the aspect of care under 

review, project team members review the process from 

which the problem originated in order to understand how 

the problem evolved and why it persists. Identification of 

these problem areas and their underlying causes provides 

team members with the necessary information to help 

solve the problem.

What to do

• Understand the process.

• Identify underlying causes.

Project Step 3: Investigate the Process.

Project Cycle

Program Cycle

Everyone’s a detective: “CSI: HIV Care Program”

After the area for improvement is identified and performance data 

collected, the project team sets off to identify where things may go 

wrong in the care delivery process. In this process, all team members can act as improvement detectives—critically investigating the 

current processes and identifying underlying causes. Techniques 

of investigation include developing flowcharts of process, identifying underlying causes, and brainstorming, but team members 

should be encouraged to be innovative in how they seek causes and 

possible solutions.

Snapshot of HIV Care

Using flowcharting to investigate the process for placing and reading PPDs identified two problems: HIV care providers were not 

sufficiently aware of the need to place PPDs and the patients were 

not returning to have their PPDs read, report JaneAnn Hall and 

Sandy Bergner of Hennepin County Medical Center, Minneapolis, 

MN. The project team then brainstormed the likely causes of these 

problems—a variety of scheduling problems with providers were 

identified. On the patient side, staff suspected the reason patients 

were not returning was because they did not fully understand the 

purpose of the PPD test or its importance. 

the Process 

Investigate 
NYSDOH AI HIVQUAL Workbook September 2006

104

Snapshot of HIV Care...Continued

In prioritizing the root causes, Hall and Bergner, say that HCMC 

staff determined that both issues were “integrally linked” and that 

both needed to be addressed by simultaneous improvement efforts. 

These efforts included the development of patient education materials and training nurses to place PPDs in the absence of physicians. 

These changes had been successful, with increases in both the 

number of patients receiving PPDs and having the read.

A quality improvement project team at the University of Pittsburgh 

Medical Center was formed to improve the process of prescription 

refills. Through their investigation, they found that many of their 

patients were receiving prescription refills but were not coming in 

on a consistent basis for medical visits or viral load monitoring. We 

found that “physicians were providing multiple refills on prescriptions, so patients didn’t need to come into the clinic to see their 

physician and have their lab work done,” reports Administrative 

Director Margaret Palumbo. Lacking adequate policies and procedures regarding refills exacerbated the problem.

The improvement effort included the development of a policy 

regarding refills (limited to no more than three), medical visits and 

lab monitoring frequency, and physician management of night and 

weekend call schedules. The new guidelines were communicated 

to patients through brochures and when they called in for their 

medication refills.

“While it was a difficult adjustment for patients and staff, after six 

months everyone seems to have adapted. As a result of this new 

policy, patient safety and retention in care has improved,” Palumbo 

reports. “The percentage of patients who kept their quarterly 

monitoring visit increased from 68 percent to 95 percent over nine 

months.” 

The cause or reason isn’t always so obvious. Often, project teams 

feel overwhelmed because they’ve identified too many possibilities, or systems that they can’t control. Sometimes, reducing the 

number of people involved in the effort can help focus the brainstorming effort. Sheila Boyle, Process Improvement Coordinator 

at the Albany Medical Center, Albany, NY reports that during one 

brainstorming activity “we started out with a very large group and 

got a lot of valuable ideas, but there was an overload of perspectives. There were so many ideas flying around, it because difficult 

to decide on the next steps. Once we limited to group to only 

four people that we felt adequately represented the process being 

investigated, we had more focused discussions and arrived at more 

concrete solutions and ideas.”

Conducting Fieldwork

Team members can also take their investigation out into the field to 

the places where care or services are actually delivered. One project 

team sent two team members to its clinic, pretending to be patients, 

to investigate the clinic visit process. Following the directions 

given to all patients, the team members actually got lost in their 

own facility because of errors in the instructions routinely given 

to patients. Only by walking through these instructions did they 

learn about a potential reason for patient visit delays. 

Frontline staff from different agencies and disciplines in the Dallas 

Family Access Network (Dallas FAN) traded places with each 

other to gain an understanding of the network’s member agencies 

and staff. “A nurse working with a case manager and an outreach 

worker going to an administrative office learned a lot because of the 

perspective of someone who is from another agency. They gained a 

great understanding of what goes on behind the scenes at the other 

member agencies. Now everyone has a new perspective of what 

it takes to walk in their colleagues’ shoes,” reposts Betty Cabrera, 

Dallas FAN’s Executive Director.
NYSDOH AI HIVQUAL Workbook September 2006

105

Snapshot of HIV Care...Continued

Other facility staff can become involved in the process through 

completing surveys or being interviewed by team members and 

reviewing the flowcharts and root cause analyses developed by 

the project team.

Project teams can also include consumers who can provide their 

perspective of being a client at the facility. The University of Miami 

School of Medicine, Miami, FL involved consumers in their investigation of low rates of follow-through with mental health referrals 

and appointments. They learned from consumers that because of 

the stigma associated with mental health services, consumers were 

reluctant to engage in these services. Their subsequent improvement project focused on dispelling this stigma.

Everything’s connected

Usually the process of investigation leads to new insight about 

how care is delivered. It can also uncover other aspects of facility 

operations. For example, one quality manager reports that during 

one brainstorming session they discovered a problem within their 

purchasing department. “It turned out that the requests for supplies for the lab were being sent to the wrong supplier. No wonder 

we never had what we needed!” This process led to improvement 

projects relating to both staff training and changes in purchasing 

policies as well as with lab operations, which was the initial focus 

of the project team. “During this process, it was important not 

to point fingers at individual staff and to continuously communicate to the team that we had to keep focused on the process, 

not the person.”

the Process 

Investigate 
NYSDOH AI HIVQUAL Workbook September 2006

106

Understand The Process.

Preliminary analysis of performance measurement data and 

other information helped to identify problems. The data 

point out deficiencies but do not necessarily explain their 

causes. Further analyses are necessary to refine the team’s 

understanding of the process and target potential problem 

areas that require improvement. 

By definition, a process is a series of steps or actions needed 

to produce something or to achieve an end. For example, a 

series of steps are repeated each time a patient comes for an 

exam—a patient registers, waits to be called, checks in with 

the nurse, meets with the provider, is examined, has blood 

drawn, if needed, then, schedules a follow-up appointment. 

A process is the basic level at which changes can be made 

to improve HIV care. Sometimes a relatively simple task 

requires several steps, any one or more of which may need 

to be changed.

Charting The Process

One of the best ways to understand a process is to draw a 

picture of it—and that’s basically what flowcharting is. A 

flowchart shows the steps of any process in sequential order. 

Flowcharts can be used to illustrate a sequence of events, 

activities or tasks for processes ranging from simple to complex. There are many styles of flowcharts but most are drawn 

using a few common symbols.

Flowcharts help staff visualize the process so that it is easier 

to understand and easier to improve. Teams should take the 

time to construct a flowchart because it:

• Provides a fast and efficient way to understand 

the process.

• Identifies potential sources of problems and provides 

a clear frame of reference for pinpointing the part of the 

process that requires change.

• Enables the team to communicate to others what they 

are doing and why.

The flowchart is completed by the project team since each 

member typically understands part of the process, but 

not necessarily the whole process. By working together the 

flowchart has a greater level of detail and accuracy, and 

agreement is reached more easily. It is important for team 

members to use information that is based on fact and 

not anecdote.
NYSDOH AI HIVQUAL Workbook September 2006

107

To round out the information gathered during flowcharting, 

a project team may interview those who are affected directly 

or indirectly by the process. Questions may focus on issues 

that might hinder the process, for example, waiting time, 

room availability and preparedness, computer problems, 

location of patient charts, patient access, and adequacy of 

the length of time scheduled. 

The Toolbox on page 108 provides a step-by-step guide 

for constructing a flowchart. An example of a flowchart 

is also included. 

Additional Resource

For guidance in teaching small groups about how flowcharts are used during process investigation, see the HIVQUAL Group Learning Guide 

"Flowchart" exercise. The exercise could also be used by project teams as a warm-up to creating their own flowchart. You can download this 

publication at www.hivqual.org.

The following steps are helpful in drafting and 

finalizing a flowchart:

• Brainstorm with those who are affected by 

the process under review.

• Create two flowcharts: one describing the current process and one outlining the desired improved process.

• Share a draft version of your flowchart openly with staff 

and gather their input; include graphic display to illustrate changes to staff.

• Interview patients or conduct focus groups with patients 

before finalizing flowchart.

• Measure the time for each step and indicate 

on flowchart.

Real-World Tip

Gather Input.

Notes

the Process 

Investigate 
NYSDOH AI HIVQUAL Workbook September 2006

108

Toolbox: 

 Flowcharting

 1. Decide on the beginning and ending points of the process. A team working on GYN care defines the beginning point when the 

patient registers at the clinic and the ending point when documentation of the results of the GYN exam are entered in the computer.

There can be more than one starting or ending point.

Patient registers

Results entered

into computer

 2. Identify each step of the process. Describe the steps of the process under review and write each step on a 3x5 card.

Patient signs

in with

medical

assistant

Health

maintenance

reviewed

Exam

performed

Need for Pap

identified by

PCP

 3. Use common symbols. Try out the following symbols. You can change them if you find a better way to illustrate the concept. 

 4. Place the cards in sequential order. Place the cards (with tape on the back) on a large piece of paper or a whiteboard and connect the 

steps with arrows. Usually on the first trial there are frequent changes. 

 5. Conduct a quick investigation, if needed. At times the team might not sufficiently understand the entire process and need to gather 

additional input.

 6. Identify and agree on key process areas. Step back and look at your diagram. Discuss what steps are problem areas based on team 

members’ experiences. Circle those steps. Time can be wasted on working on "surface" problems, i.e. those you see versus those that 

give rise to the problem. For each problem identified, discuss likely causes and write them next to each problem. This method combines the flowchart with a cause-and-effect analysis.

Oval shows START or END point on a process diagram

Rectangle shows an activity in the process

Diamond shows a "YES" or "NO" decision

Arrow connects steps in a process and directs the flow
NYSDOH AI HIVQUAL Workbook September 2006

109

Toolbox: 

 Flowcharting...Continued

Patient arrives at front desk.

Staff asks for the patient’s

name and searches the

database for his/her file.

Patient in system?

Yes

Staff asks patient to

complete paperwork for new

clients and return it to the

front desk.

Staff asks patient to be seated

in the waiting room.

Medical nurse takes patient

into the exam room.

Oval: shows beginning or ending 

step in a process

Flowchart Symbols

Rectangle: depicts particular step 

or task

Diamond: indicates a decision point

Arrow: shows direction of process flow

etc.

no

the Process 

Investigate 
NYSDOH AI HIVQUAL Workbook September 2006

110

Identify Underlying Causes.

Having completed the flowchart, the project team begins 

identifying potential barriers and the underlying causes of 

the problem(s). Underlying causes are the reasons a problem 

happens repeatedly. If the team can identify and eliminate 

underlying causes, they eliminate the problem. Conversely, 

if the team fails to correctly identify the underlying causes, 

they may only ‘cover up’ the problem and it will probably 

reoccur. In real life, a quality problem is embedded in a 

system with a myriad of interdependent steps. To fix just 

one cause might not fix the system barrier.

Identifying Potential Causes

During process investigation the project team generates 

ideas about potential causes of the problem, either through 

group discussions or brainstorming.

Brainstorming is defined as a technique to freely and uninhibitedly generate ideas using a group approach. It is a 

useful tool when a team needs to generate a large volume of 

ideas. The Toolbox on page 111 provides a step-by-step guide 

for brainstorming.

Mapping Underlying Causes

Once the underlying causes are identified, the project team 

maps the factors that influence the problem. The visual 

display is important to reach agreement among the team and 

to clearly communicate the causes of a problem to others at 

a care facility, especially the HIV quality committee, staff 

and stakeholders. 

The team can use flowcharts by adding the underlying 

causes at each step or a cause-and-effect diagram. The Toolbox on page 113 provides a step-by-step guide for creating a 

cause-and-effect diagram and an example.

Additional Resource

For guidance in teaching small groups about how brainstorming is used during process investigation, see the HIVQUAL Group Learning Guide 

"Brainstorming" exercise. The exercise could also be used to help team members adapt an "anything goes!" attitude before embarking on their own 

brainstorming session. You can download this publication at www.hivqual.org.
NYSDOH AI HIVQUAL Workbook September 2006

111

Toolbox: 

 Brainstorming

Basic Steps for Brainstorming

1. Write the topic statement or question in a central location. It should be clearly defined and written where everyone can see it.

2. Review general rules for brainstorming. Basic ground rules include:

• Go for quantity of ideas; do not censor your ideas or anyone else’s.

• Utilize free-association and building on previous ideas.

• Record ideas as stated; do not edit—only clarify, if necessary.

• Do not discuss or debate the merit of individual ideas.

3. Establish a time limit. 7 to 10 minutes is recommended.

4. Generate ideas with the group until time is up. Begin idea generation by going around the group, allowing one idea per person. 

Participants may pass if they do not have an idea. Ideas should be written down where everyone can see them. The process of 

generating ideas usually goes through several cycles. Later cycles tend to have a slower pace, but may result in the most innovative 

ideas. It is important not to rush the process.

5. Review and refine ideas. Discard any ideas that are virtually identical.

Brainstorming Example of Show-Rate for GYN Appointments

Background:

A project team investigates the show-rate for GYN appointments and presents their results to the quality

committee: 51 percent for GYN appointments.

Question for Brainstorming:

What are the reasons for a low show-rate for GYN appointments?

Idea List:

• Staff do not give patient printed appointment card

• Patients unaware of appointments

• Lack of childcare for patients

• Staff give patient wrong appointment information

• Difficult to reach patients directly by phone due to wrong contact information

• Computer system taken down for routine maintenance

• Appointment cards do not include exact clinic address

• Reminder calls placed by someone patient doesn’t know

• Computer can only print reminders for appointments within 3 months

• No procedure in place to reschedule broken appointments

• Remote location of clinic

• Only one appointment can be listed on appointment card

the Process 

Investigate 
NYSDOH AI HIVQUAL Workbook September 2006

112

Prioritizing Underlying Causes

Not all underlying causes are equal. Prioritization helps 

teams determine which one of the underlying causes has the 

greatest impact on the HIV system and potential for improving the aspect of care under review. Teams answer the following question: "Which underlying causes of the problem 

should we focus on first?"

Prioritization of underlying causes is a decision reached 

through team consensus. Possible criteria for 

prioritization are:

• Underlying cause within control of the team

• Impact to consumers (client inconvenience versus ‘pain’ 

caused by the problem)

• Difficulty in solving the underlying cause

• Resources required for addressing the underlying cause 

(e.g., staff time, money or space)

• Impact on delivery system

Additional Resource

For guidance in teaching small groups about how cause-and-effect diagrams are used during process investigation, see the HIVQUAL Group 

Learning Guide "Cause-and-Effect Diagram" exercise. You can download this publication at www.hivqual.org.

Consider the following tips when identifying and 

prioritizing underlying causes:

• Don’t wait for the perfect solution when you need to 

remove a barrier; try a solution as quickly as possible.

• Start by investigating ‘low-hanging fruit’—those problems with the most impact and value.

• Consult staff which underlying causes are the most 

important barriers and ask how to eliminate them. 

• Find out how similar HIV programs have successfully 

completed similar quality projects.

• Include consumers when investigating the causes.

Real-World Tip

Ask For Input And Eliminate Barriers.

The Pareto chart is a quality tool that can be particularly 

helpful during this step. The Toolbox on page 115 provides 

an overview of the Pareto charting process.

Notes
NYSDOH AI HIVQUAL Workbook September 2006

113

Toolbox: 

 Cause-and-Effect Diagram

The cause-and-effect diagram may also be referred to as an Ishikawa diagram, after the doctor who first developed it, or a 

fishbone diagram, after the diagram’s structure that resembles the skeleton of a fish. The "skeleton" of a cause-and-effect 

diagram is shown below.

Basic Construction of a Cause-and-Effect Diagram

1. Draw the diagram’s skeleton. Explain that the skeleton consists of a horizontal arrow pointing to the effect and additional arrows—representing causes—pointing to the horizontal arrow. Major causes can be separated into basic 

categories such as: Equipment, Environment, Procedures, People. These are only suggestions. A team should use 

categories that best fit their improvement needs. Others could be: Methods, Materials, Resources, and Measurement. 

Make them fit your problem.

2. Write the problem (or desired outcome) in the box at the end of the arrow.

3. Brainstorm potential causes and subcategories to fill in the "bones" of the skeleton if not done prior to this. Review the 

potential causes. Note how major causes typically have subcategories, identified by asking: Why does this happen?

4. Review and refine causes. This sets the stage to examine a few of the causes further and prioritize them.

Skeleton

the Process 

Investigate 
NYSDOH AI HIVQUAL Workbook September 2006

114

Toolbox: 

 Cause-and-Effect Diagram...Continued

PROCESS PATIENTS

EQUIPMENT

RESOURCES 

providers, space, 

services, data

Only 66 percent of

patients have had

pelvic exams

documented

• Follow-up procedure not followed

• GYN appointments not available for weeks

• 10 minute slot for GYN appointment too brief

• Waiting time for patient in clinic is too long

• GYN only available one day in the week

• AM session overflow delays PM session start

• Waiting time for GYN referral is too long

• Patients fail appointments for primary care 

• Patients fail appointments for GYN

• Patient are not aware of importance of GYN exam

• Contact information for patients is incorrect

• 2 GYN sessions per week availability

• Only one room in use for GYN

• Provider is overbooked

• Provider is not able to speak Spanish

• Primary care providers are not providing pelvic exams

• Providers are not referring HIV patients to GYN department

• No routine reports by GYN department about frequency of

 patient visits

• Due to new computer system, GYN appointments 

can not be made 

• Computers are frequently down

• Phone lines are busy at certain hours
NYSDOH AI HIVQUAL Workbook September 2006

115

Toolbox: 

 Pareto Chart

Description In any group of variables that contribute to a common effect, a relative few contributors (‘vital few’) will 

account for the majority of the effect while other contributors (‘useful many’) will have less impact. This principle is called Pareto or "80/20 Rule" which suggests that most effects are the result of relatively few causes, 

that is, approximately 80 percent of effects come from 20 percent of potential causes. To maximize your 

efforts, the team needs to identify first the ‘vital few’ and focus their efforts on those contributors.

use • Helps focus improvement efforts by ranking problems.

• Allows project improvement teams to identify the ‘vital few’ causes of problems on which to

 focus their improvement efforts.

Basic Construction 1. List response categories on the bottom horizontal axis in order of decreasing frequency, 

 starting on the left side of the chart.

2. Mark the units (e.g., frequency, percent, cost, or time) on the left vertical axis.

3. Mark the right vertical axis from 0 percent - 100 percent.

4. Record the ‘raw data’ on the chart.

5. Construct a line starting from the top of the tallest bar to show the cumulative percentages.

Pareto Chart

Example 200

160

120

80

40

Face-to-face 

conversation 

with clinician

Face-to-face 

conversation 

with nurse

Attendance 

of adherence 

workshop 

by patient

Attendance 

of peer 

adherence 

group

Handing

out of

consumer 

educational 

brochure

Other

81%

50

70

42

24 10 4

100 %

80 %

60 %

40 %

20 %

Which Adherence Interventions Work?

93%

98% 100%

60%

35%

the Process 

Investigate 
NYSDOH AI HIVQUAL Workbook September 2006

116

The Big Picture

Improvement changes are tried through ‘pilot tests’ on 

a small-scale before committing valuable time and resources 

to system wide implementation. Those pilot tests are often 

described as Plan-Do-Study-Act (PDSA) cycles. Selecting 

and planning pilot tests and evaluating pilot test results 

are probably the most critical activities in quality 

improvement projects.

What To Do

• Plan-Do-Study-Act (PDSA) cycles.

• Plan for pilot tests.

• Conduct pilot tests

Project Step 4: Plan and Test Changes.

Project Cycle

Program Cycle

Experimenting with changes to create 

lasting improvements

Investigating the process often generates a long list of ideas for 

changes. “One of our biggest challenges in planning a change was 

to get the improvement team to think small. Our investigation of 

the process identified so many possible underlying causes. The team 

wanted to take them all on,” reports a quality manager. “While 

they all seemed compelling, we finally agreed to focus on only a 

handful and to prioritize these for our first PDSA cycles.” 

Snapshot of HIV Care

Often, good ideas don’t work as planned. One clinic seeking to 

improve patient appointment keeping decided to implement a process for reminder telephone calls the day prior to the appointment. 

Through a PDSA cycle they discovered that over half of the phone 

numbers they had for their patients were no longer working and 

that much of the alternative contact information was incorrect. Implementing regular reminder calls with the currently documented 
NYSDOH AI HIVQUAL Workbook September 2006

117

Snapshot of HIV Care...Continued

contact information was not likely to improve appointment keeping 

and would use a lot of staff resources. They decided that they next 

needed to test and implement a process for updating patient contact 

information so that they would have useful telephone numbers to 

begin reminder phone calls. 

Creating a competition between teams has been used effectively to 

plan and test changes. The Albany Medical Center created some 

excitement about an improvement project to increase the number of 

annual comprehensive exams conducted. “We needed to find a way 

to jumpstart the improvement project and to create a commitment 

to quality improvement in the facility,” reports Sheila Boyle, Process 

Improvement Coordinator. “We created two teams who competed 

against each other. Each team had to develop and document an 

improvement process for identifying patients who were due for their 

annual comprehensive exam and getting these patients into the 

clinic.” The competition worked, doubling the number of exams 

usually provided. While the results were still less than the initial 

goal, it was still successful because “patients received high quality 

care, and staff became enthusiastic about their work and a commitment and interest in quality improvement took hold in the facility.”

Small or big, improvements need to be tested 

and studied

Improvement efforts can also be small and focused changes. 

Sometimes changes can be as simple as changing one item on a flow 

sheet, changing the order of who sees a patient first in the clinic, or 

placing a reminder sheet on top of a chart. 

“Regardless the size of the change—big or small—each cycle 

needs to be assessed against the predicted improvements and team 

members involved in deciding the next steps,” advices a quality consultant. “The team should learn from both successes and failures 

and use this knowledge to inform their process.”

Improvement efforts can sometime yield bigger results than expected. The quality team at St. John Riverside Hospital developed 

a simple graphic presentation of patient labs for CD4 and viral 

loads to help patients better understand their lab results and their 

relationship to care. The results from this change were better than 

expected. According to Kay Scott, “The patients were elated with 

the service. For the first time patients didn’t have ‘blank looks’ on 

their faces when providers reviewed their lab results with them. 

Patients were so excited about receiving the graphic printout that 

they told their case managers and other providers about them.” 

In Scott’s words, “Patient enthusiasm was the turning point—and 

it spread quickly throughout the hospital. Providers and staff 

were energized, and this was exactly the kind of buy-in needed to 

systematize the change.” According to Scott, “the initial boost in 

consumer satisfaction with the change provided a catalyst for making quality improvements.”

Plan and Test 

Changes 
NYSDOH AI HIVQUAL Workbook September 2006

118

Conduct PDSA (Plan-Do-Study-Act) Cycles.

Because not all changes tested in pilots result in improvements, a project team identifies promising ideas for changes, 

tests them on a small-scale and assesses the impact on the 

aspect of HIV care under review. A helpful framework for 

facilities to implement pilot test changes is the Plan-DoStudy-Act (PDSA) cycle, developed by Walter A. Shewhart. 

In other words, the PDSA cycle is a "trial-and-learning" 

method to test changes before system-wide implementation.

Four steps are included in the PDSA cycle:

• Plan (Plan a change). At this point the team identifies a 

change and plans its implementation on a small-scale; 

including the number of records, timeframe, responsibilities and predictions of results. 

• Do (Try it out on a small-scale). The project team members test the proposed change to see whether it results in 

an improvement. The goal is to keep the tests as small as 

possible. The shorter the test cycles, the more tests can 

be conducted and therefore, more opportunities for 

learning will emerge.

• Study (Observe the results). Once the results are analyzed and reviewed, the project team will need to find 

answers to the following questions: Did we meet our 

goal? What worked and what didn’t? Do we need additional test cycles?

• Act (Refine the change as necessary). The project team 

maximizes the impact of successful changes by increasing the sample size involving providers and expanding 

the test cycles. 

The following example should illustrate the process of the 

Plan-Do-Study-Act cycle: 

A project team that is charged to improve the performance 

on adherence assessments decides to pilot test a new flowchart 

that hopefully better documents the adherence discussions with 

patients. The team predicts a 20 percent increase. Initially, the 

team revises the clinical flowsheet and asks just one provider to 

test the revised flowsheet during one clinic session. Once feedback is received, the flowchart is revised and tested again with 

three providers over the course of the following week. The results 

are studied and more changes are made to the new flowsheet. 

After one more testing cycle, the revised flowsheet is implemented system-wide, ready for all providers for all clinic sessions. 

The completion of each PDSA cycle leads directly into the 

start of the next cycle. A team learns from the test and uses 

the new knowledge to plan the next pilot tests. The team 

continues linking PDSA cycles. Often, a team will test more 

than one change at a time, each change aimed at achieving 

the ultimate goal of the entire quality improvement project. 
NYSDOH AI HIVQUAL Workbook September 2006

119

Toolbox: 

PDSA (Plan-Do-Study-Act) Cycle

The PDSA cycle provides a framework for facilities to plan and test changes before systemwide implementation.

Step 4: Act

Refine the change

as necessary

Step 1: Plan

Plan a change

Step 3: Study

Observe the results

Step 2: Do

Try it out on 

a small-scale

ACT PLAN

STUDY DO

Plan and Test 

Changes 
NYSDOH AI HIVQUAL Workbook September 2006

120

Plan For Pilot Tests.

Pilot tests as PDSA cycles serve to try out potential solutions 

on a small-scale before system-wide implementation. 

The project team members initially select ideas for pilot 

tests that are most likely to succeed and plan their 

small-scale implementation. 

Selecting Ideas For Pilot Tests

A list of potential pilot tests are identified, narrowed down 

to a manageable number and then prioritized by the project 

team. The more time a team knows about the process, the 

easier it is to identify practical solutions for pilot tests.

The following criteria are helpful to make a quick determination with which pilot tests to start:

• Measurable: Can improvements be measured 

to track progress? 

• Immediate: Will the pilot test produce short 

term visible results?

• Focused: Is the pilot test targeted to the underlying cause? 

• Feasible: Is the pilot test feasible given 

available resources? 

• Supportable: If the test is successful, will staff and 

facility leaders support successful changes? 

The Toolbox on page 121 provides a selection grid that 

teams can use to score potential pilot tests and determine 

which are most likely to succeed.

A list of various strategies HIV care facilities have implemented to improve important quality of care aspects is provided in the Toolbox on page 123 and 124. It may be helpful 

to identify ideas for pilot tests in the following areas:

• Patient retention

• Annual GYN exam

• PPD screening

• Adherence to ARV therapy

Additional Resource

For guidance in teaching small groups about brainstorming potential pilot tests and the purpose of selection criteria in evaluating a potential pilot 

test, see the HIVQUAL Group Learning Guide "Selecting a Pilot Test" exercise. You can download this publication at www.hivqual.org.
NYSDOH AI HIVQUAL Workbook September 2006

121

Toolbox: 

Pilot Test Selection Grid: Improving the percentage of women 

receiving HAART therapy.

List the top three pilot tests and score each pilot test from 1 (lowest) to 5 (highest).

Criteria Pilot Test 1: 

Special education by clinician about benefits of

HAART therapy. 

Pilot Test 2: 

Establish peer support

group for women

to follow-up.

Pilot Test 3: 

Provide incentives for

women who continue regimen

during pregnancy.

measureable

immediate

focused

feasible

4 2 5

2 4 2

5 3 4

4 3 2

SUPPORTABLE

TOTAL 19 17 15

4 5 2

Plan and Test 

Changes 
NYSDOH AI HIVQUAL Workbook September 2006

122

Planning Pilot Tests

Project teams plan the implementation of pilot tests to better 

orchestrate and guide their efforts. A planning approach 

increases the likelihood of task completion. Team members 

strategize the following areas:

• Scope of pilot test. What is the working 

hypothesis for the pilot test? For how long do you want 

to test your changes before implementation?

• Timetable for pilot test implementation. What 

are the necessary steps and when will they be completed?

• Accountability for pilot test. Who will measure 

and follow-up on pilot test results? Who reports?

• Pilot test measurement. How do you measure 

the pilot success? What indicators are identified? 

• Sample Size. How many records will be measured?

The Toolbox on page 126 provides a written sample form 

for planning a pilot test. 

Consider the following suggestions when planning 

for pilot testing:

• 'Steal shamelessly, share senselessly!’ - contact other HIV 

programs, read best practices/success stories at www.

hivqual.org and NationalQualityCenter.org.

• Keep individual pilot tests simple—plan to collect 

only as much data as necessary.

• Reduce the sample size and/or test intervals 

to a minimum.

• Use the power of ‘1’: one clinic, one provider, 

one patient. 

• Predict results of pilot test results.

• Routinely inform staff about pilot test progress.

• Ask consumers and stakeholders for their input.

Real-World Tip

Plan for Successful Tests.

Notes
NYSDOH AI HIVQUAL Workbook September 2006

123

Toolbox: 

 Successful Improvement Strategies

Patient Retention:

• Update patient contact information at every visit.

• Ask for additional contacts to get in touch with patient ("Who knows where you are?").

• Create support systems for patients to keep appointments.

• Peer educators to introduce new patients to clinic.

• New patient luncheons and/or dinners to introduce clinics and staff.

• User-friendly orientation package.

• Providers to emphasize importance of making appointments.

• Combine services as much as possible into one visit (‘max-packing’).

• When scheduling appointments (sign-out, call), ask directly for barriers, responsive to patient (‘what time works best for you?’).

• Incentives for patients to come for appointment (transportation, etc.); patient can choose incentive.

• Phone script for staff to make appointments.

• Institute a reminder system, send letters, make reminder calls.

• Contact client within 1 week of missed appointment to reschedule.

• Set up a system to see walk-in patients who don’t keep appointments as scheduled.

• Make a provider available with flexible time to accommodate walk-ins.

• Identify and track lost patients.

• Focus groups with patients who were lost to follow-up.

• Use pharmacy to help (patients fill prescriptions but may not keep appointments).

• Use drug treatment providers to help locate patients, especially MMTP.

• Use case managers to ‘find’ patients.

Annual GYN Exam:

• Have the GYN exam performed by an HIV primary care provider (MD, PA, NP, or midwife).

• Implement a specific campaign to "Get Your GYN Exam" (e.g., 2 times/year during ‘low’ times in January and June).

• Have multidisciplinary staff implement preventative health education messages (e.g., the importance of routine GYN exams).

• GYN exam right after primary care appointment.

• Better access to female GYN provider.

• Dedicated sessions for HIV patients within the GYN department.

• Evening clinic for GYN exams.

• Provide child care during clinic hours.

• Hold patient care quarterly meetings so that each HIV patient is reviewed at least once per year to identify needs.

• Include the GYN exam during comprehensive annual physical exam.

• Use information systems to identify the list of patients needing GYN exams.

• Flag charts (e.g., use a sticky note) to identify who needs a GYN exam on the day of appointments.

• Provide transportation (e.g., Go get them!).

• Follow up release of information forms by CMs, nurse, etc. with calls and tracking logs.

• Dedicated sessions for HIV patients within the GYN department .

Plan and Test 

Changes 
NYSDOH AI HIVQUAL Workbook September 2006

124

Toolbox: 

 Successful Improvement Strategies...Continued

PPD Screening:

• Provide 24-hour access for reading (e.g., agreement with emergency room).

• Provide no waiting for PPD reading in clinics.

• Train medical assistants to plant and read PPDs.

• Use standing orders for PPDs.

• Improve coordination with external organizations that perform PPDs to get results.

• Make available self-read cards and telephone-read results.

• Do not plant on a Thursday unless the PPD can be read over the weekend.

• Implement vaccine campaigns for PPD twice per year (November and June).

• Make reminder calls 24 to 48 hours following planting.

• Use the PPD log for tracking.

• Schedule other services to coincide with the return for the PPD reading.

• Incorporate PPD into comprehensive annual physicals.

• Flag the chart in advance—use reminder stamp.

• Provide incentives (e.g., phone cards, gas cards, fast-food certificates) for patients who return for PPD reading.

Adherence to ARV Therapy:

• Designate a person to serve as an adherence counselor/educator.

• Hold multidisciplinary meetings between providers and CMs to review guidelines, develop messages, and decide who says what.

• Promote the use of adherence stamp to improve documentation.

• Increase the frequency of interaction between the patient and adherence counselor for non-adherent patients.

• Promote the use of adherence tools (e.g., pill boxes).

• Promote support group participation.

• Support peer advocacy.

• Provide peer outreach to deliver drugs.

• Ensure all staff involved with a patient delivers a consistent message reinforcing the importance of taking medication on time.

• Hold case conferences to share information.
NYSDOH AI HIVQUAL Workbook September 2006

125

Conduct Pilot Tests.

Pilot tests allow team members to assess the effectiveness 

of various solutions before system-wide implementation. A 

pilot test is intended to be a small-scale trial of a potential 

solution. The project team should consider the following 

strategies for successful pilot tests:

• Simplicity of pilot tests. Keep initial pilot tests 

simple and emphasize the following point: the more 

pilot tests can be conducted and more opportunities for 

learning will emerge.

• Series of pilot tests. Allow for multiple pilot tests and 

build on the success of previous pilot tests.

• Reduced sample size. Reduce the sample size to 

a minimum and collect as little data as necessary.

• Short-time approach. Reduce the test intervals to 

a minimum while increasing testing cycles.

• Clear accountability. Ensure that the responsibilities 

for conducting pilot tests are clearly defined 

and communicated.

Assess Impact

The improvement team evaluates the changes by answering 

the following questions:

• Did the changes help us reach our improvement 

project goal?

• Are additional pilot tests indicated?

• Are there other changes that can be implemented 

to exceed our goal?

• What prevented us from reaching our goal? 

Team members compile data collected during the pilot 

tests. Based on these results, the team reaches an agreement 

on how to move forward. Once a decision is made, the 

improvement team shares the pilot test results with the 

HIV quality committee.

Additional Resource

Langley Gerald J. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. Jossey-Bass, 1996. For guidance in 

teaching small groups about the basic steps of pilot test design and implementation, see the HIVQUAL Group Learning Guide "Planning 

a Pilot Test" exercise. You can download this publication at www.hivqual.org.

Simple ideas to get started: 

• Keep it simple and collect as little data as possible.

• Quick turnaround is key; conduct a series of pilot tests.

• Test initially with just one provider.

• Always plan two or three pilot tests ahead.

• React right away—if an improvement is very 

obvious, make a quick decision to implement. 

Do not wait for more data.

• Use original predictions to assess which solutions had 

impact and which needed additional follow-up.

Real-World Tip

Conduct Successful Tests.

Plan and Test 

Changes 
NYSDOH AI HIVQUAL Workbook September 2006

126

Toolbox: 

Pilot Test Planning Form

Goal:

• Increase patient knowledge about oral health and its importance as part of care.

• Increase the number of patients receiving oral health care.

a. Develop educational materials.

Plan the plot test implementation. who is responsible? when completed by? (week)

1 2 3 4 5 6 7 8

b. Make a 20-minute presentation to providers during the 

next staff meeting to discuss what they should tell patients 

about oral health. 

c. Put posters in waiting room.

X X

X

X X

Review results of the pilot test. 1 2 3 4 5 6 7 8

b. Count how many educational sessions are conducted 

between providers and patients.

c. Survey patients to assess their understanding of the 

importanceof oral health. X

X

d. Look at how many dental appointments are made.

X

X X

a. Count materials being given out to patients.

Decide whether/how to move forward. 1 2 3 4 5 6 7 8

Fran

Michael

a. Compile results in one report for final presentation. Catherine X

b. Present results to quality committee. X

c. Present results to staff. X X

define the plot test.

• Cause of problem: Although patients are screened for the 

dental exam and appropriate referrals are made, they do not 

follow upwith their dentists to receive the necessary services.

• Change to be tested: Oral Health Awareness Month.

• Evaluation method(s): Increase in dental appointments 

made by patients. Of the appointments made, how many 

are kept; Number of materials given out to patients; 

Survey to assess patients’ understanding of the importance 

of oral health. 
NYSDOH AI HIVQUAL Workbook September 2006

127

Project Step 5: Evaluate Results with Key 

Stakeholders.

The Big Picture

The project team’s next steps are to document the improvement project results and review them with the HIV quality 

committee. Once the quality committee determines that the 

improvement project has remained on track, decisions need 

to be made regarding how to effectively share the project’s 

success with staff and other stakeholders, and how to implement those improvements system-wide.

What To Do

• Summarize improvement project results.

• Evaluate results with HIV quality committee.

Project Cycle

Program Cycle

Sharing Results Of Improvement Projects To 

Create Interest In Quality Improvement.

“Did it work? Who can we tell?” After conducting an improvement project, the team should be excited about their results and 

will want to share them. The team should use effective methods 

of communication—storyboards, newsletters, websites, paycheck 

inserts—to convey the story of the improvement project and to create excitement about implementing the change. 

Snapshot of HIV Care

The improvement team “should not be afraid to share information,” 

counsels James Pratt of St. Joseph’s and Family Care, Reading, PA, 

“and when you do, the simplest solutions work best.” When reporting their improvement results, staff creates simple easy-to-read 

handouts to use during our discussions with stakeholders. “Using 

simple handouts makes the pilot test data user-friendly rather than 

intimidating,” says Pratt. The handouts give the meeting a clear diEvaluate Results 
NYSDOH AI HIVQUAL Workbook September 2006

128

Snapshot of HIV Care...Continued

rection and keep everyone focused on the same information. “The 

handouts also help to set the stage for a clear and open discussion 

about future tasks.”

The method of presentation should be tailored to the audience and 

purpose. For example, a presentation to the quality committee may 

detail the entire process used by the project team, while presentation to the facility’s board may use a graph to clearly display the 

results and provide a summary of what’s needed to implement the 

change facility-wide.

Displaying the data and improvement efforts in the public spaces 

also creates opportunities for patient education. Saneese Stephen, 

of Kings County Hospital Center, Center for H.O.P.E., Brooklyn, 

NY recounts, “When patients look at the storyboards displaying 

our improvement data, they can be educated on clinic performance 

on indicators. For example, I had one patient ask me ‘What does a 

PPD with 80 percent mean?’ They learned that PPDs and vaccinations are important because we’re displaying them on the walls and 

we’re measuring them. Hopefully, they’ll learn the importance of 

this care and will come in for this care. It lets them know that the 

clinic is dedicated to quality improvement and that we’re looking to 

make patient care better.”

The posters have also helped prompt staff to remind patients about 

screenings and exams. For example, Stephen reports, “if a social 

worker is seeing a patient and she sees that the patient hasn’t had 

a GYN exam and know it is important since it’s an indicator, they 

might ask the patient, ‘Why haven’t you kept your GYN appointment?’ Hopefully, by displaying our quality information to patients 

and staff, everyone will get involved in improving quality.”

The Center for H.O.P.E. takes an opposite approach when reporting the results of provider-specific data. “For each provider we look 

at their PPD, GYN and vaccination rates for the quarter. Then, we 

make a graphical representation showing how each provider does 

individually on each of the indicators and how they compare to the 

total,” Stephens reports. 

“At first, the providers were understandably hesitant about this approach, fearing that they would be micro-managed. However, now 

they are interested in how they’re doing and want to see themselves 

do better.” The provider-specific results are presented to each provider privately, but for public presentation, each provider is coded 

with a letter, with only Stephens and the facility’s Medical Director 

knowing the code.
NYSDOH AI HIVQUAL Workbook September 2006

129

Summarize Improvement Project Results.

The purpose of documenting the improvement project 

results is to communicate with others how problems in current processes have been solved and what results have been 

achieved. The final project write-up succinctly describes the 

efforts of the quality improvement team and allows future 

teams to learn from past projects.

Documenting Results

The data presented typically include baseline data and pilot 

test results. Graphic displays of data, such as charts and 

tables, help to convey results at-a-glance and should be used 

whenever possible. Refer to the Project Step 1 ‘Review, collect and analyze project data’ on page 90 for a more detailed 

discussion of the graphic presentation of data.

How the improvement team chooses to summarize project 

results—the choice of report format, length, and sophistication—may vary. Two potential formats are a written report 

or a storyboard (a poster display that includes a graphic 

display of data over time). 

To effectively present project results in either format, the 

"Four Cs" of effective communication should be applied:

• Clear: Use terms that committee members and staff 

understand and relate to.

• Concise: Be short and to the point.

• Complete: Include all relevant information.

• Correct: Ensure that all data are accurate.

Storyboards can help teams communicate the highlights of 

an improvement project to others. If a project team opts to 

prepare a storyboard, the following tips can help to make it an 

effective format:

• Construct the storyboard as a logical progression of 

‘boxed information.’

• Lead the reader through the main points and steps of 

the improvement project.

• Communicate with descriptive pictures and graphics 

more than words.

• Use color and keep any text simple.

• Following the HIV quality committee’s review of the 

storyboard, post it in a visible location to share the team’s 

results with consumers and staff.

• Present storyboard to the Consumer Advisory Committee.

Real-World Tip

Make Your Storyboard a Bill.

The Toolbox on page 130 shows an example of the type of 

information that should be included in either 

a written report or a storyboard. 

Evaluate Results 
NYSDOH AI HIVQUAL Workbook September 2006

130

Toolbox: 

Sample Improvement Project Data for Presentations

Site: AIDS Hospital Center

Contact Person: Phaedra Charman, AIDS Center Administrator

HIV Caseload: 700

Facility Type: Teaching Hospital

IMPROVEMENT STEPS

Improvement Area

Progress to Date/Results

Goal

Team Membership

Investigation of the Process

Pilot Test Interventions and Results

Systematized Change

HIV PPD Improvement Project

Between January 15 and June 30, 2006, increase the percentage of PPD chart documentation (placed 

and read) from 66 percent to 85 percent

• Thomas Oddou, MD, Chief of Infectious Diseases

• Phaedra Charman, AIDS Center Administrator

• Elvira Flores, Director of Ambulatory Care

• Gary Gross, Director of Quality Assurance

• Cynthia Ferraro, Director of Nursing

• Chris Feno, AIDS Center Nurse Practitioner

• Jorge Carrillo, Clinic Coordinator for Case Management

• Betsy Stelle, R.N., Senior Staff Nurse

Analytic tools used:

• Gantt chart

• Pareto chart

• Run chart

• Cause-and-effect diagram

• Brainstorming

Underlying causal analysis:

• No individual clearly assigned responsibility

• As a teaching hospital, house staff rotate through the AIDS Hospital Center, are unaware of 

 PPD requirement and unfamiliar with patients’ history of care

• Patients forget to return for PPD reading

Interventions on 1/06 and 2/06

• Beginning 1/06, clinic case managers were assigned responsibility for flagging charts of patients 

due for PPD

• Develop a written reminder card for patients

• Communicate with patients about the importance of PPD testing

• Track of PPD performance rates over time

• Beginning 2/06, senior staff nurse was assigned responsibility for planting PPD and giving patients 

a reminder card to return in 48 – 72 hours

Results on 2/06 and 3/06 - percentage of PPDs placed and read increased from 66 percent to:

• 75 percent by the end of 2/06

• 81 percent by the end of 3/06

• Results were reported at the regularly scheduled quarterly HIV Quality Committee 

meeting Won 4/06.

• A decision was made to make interventions permanent and system wide.
NYSDOH AI HIVQUAL Workbook September 2006

131

Toolbox: 

Sample Improvement Project Data for Presentations...Continued

IMPROVEMENT STEPS Progress to Date/Results

Next Steps

Program Infrastructure to Manage 

Ongoing Quality Improvement

Factors Influencing Improvement 

Projects

Lessons Learned from the 

Improvement Project

Additional interventions

• To achieve further improvement, the senior staff nurse was assigned responsibility for calling 

patients on day two to remind them to return on day three for PPD reading.

• Any patient not showing on day three for reading is called again by the senior staff nurse and 

urged to come in on day four.

Data collection

• Data were collected though focused survey on chart sample at the close of each month to monitor 

the impact of the interventions.

• The two additional interventions (above) increased performance to 88 percent by the end of 4/06.

Future planning

• Assessment of PPDs planted/read is now an indicator in ongoing quarterly 

chart review cycle.

• PPD and other performance data are reviewed quarterly by the center’s HIV 

Quality Committee.

Role of program leadership

• The medical director, administrator and staff nurse took the lead and were 

involved throughout.

Role of the improvement team

• Team met bimonthly at the close of clinic to review sample survey data, assess progress and determine next steps.

Communication mechanisms

• Dr. Oddou, MD, Chief of Infectious Diseases, serves on the hospital HIV Quality Committee 

and kept the committee informed.

• Phaedra Charman, AIDS Center Administrator, kept improvement team minutes and shared 

them with three quality hospital committees (HIV, internal medicine and ambulatory care).

Ongoing mechanisms

• PPDs placed/read now an ongoing indicator on chart review tool; applied to a random sample of 

45 charts quarterly.

• Data are reported at all HIV Quality Committee quarterly meetings.

Factors which helped us

• High degree of staff motivation and teamwork

• Excellent patient-staff relationships

Obstacles which hindered progress or needed to be overcome

• Some patients (due to poverty and/or other psychosocial factors) lack telephones and are hard to 

reach for follow-up.

As a program

• Patients appreciate high-quality comprehensive care and proactive staff who work in the interest of 

their health.

As a team

• Mutual support and teamwork

• Developing and supporting new leadership

Evaluate Results 
NYSDOH AI HIVQUAL Workbook September 2006

132

Evaluate Results With HIV Quality Committee.

Presenting the progress of the improvement project to the 

quality committee is important for several reasons. Sharing 

this information provides a feedback mechanism on the 

team’s present work and lays the groundwork for getting 

"buy-in" on how best to spread and systematize changes. It 

also promotes public relations and helps build future support 

for improvement activities from the HIV quality program.

Making A Decision

Based on the evaluation of the project, a decision can be 

made whether the improvement project should be continued 

for future gains and/or implemented system-wide.

Quality improvement project results are evaluated against 

the following criteria:

• Effectiveness against goals. Did the project reach its 

promised goal(s)?

• Range of impact. Should the project be further expanded 

to increase its impact? 

After reviewing the information with the HIV quality committee the final decision is made and the follow-up tasks 

required to pursue the chosen course of action are identified.

Additional Resource

For guidance in teaching team members about the purpose of presenting project data to specific audiences, see the HIVQUAL Group Learning 

Guide "Data Presentation" exercise. You can download this publication at www.hivqual.org.

Implementing Programs System-Wide

Spreading improvements into the wider system means 

implementing effective solutions based on the results of the 

pilot tests, where appropriate, throughout the HIV program. 

Perhaps the larger organization, of which the HIV program 

is a part, could also be included. A decision to implement 

the improvement project changes system-wide requires additional discussion and planning. 

Different system-wide implementation scenarios are:

• Expansion to the entire HIV program. A successful project could be implemented at the entire clinic or all clinics 

in the HIV program’s network.

• Expansion to non-HIV programs in a facility. A successful project in the HIV program of a hospital could be 

shared with other departments.

Sharing The Report

Sharing improvement project progress with the entire 

program has the additional benefit of educating other staff 

members on how changes were made and what quality 

improvement really means. Staff members can learn a great 

deal about planning resource allocation and prioritizing 

pilots for implementation. The report should also be shared 

with the HIV program’s leadership, board members, con-
NYSDOH AI HIVQUAL Workbook September 2006

133

The following actions can help in presenting 

improvement project results:

• Use different communication methods and channels.

• Tailor a presentation to the audience (e.g., formal 

or informal).

• Always include graphic displays that show data over time.

• Present results to consumers (e.g., the consumer advisory 

board) and use their input in systematizing gains.

Real-World Tip

Be User-Friendly.

sumers and advisory groups to create buy-in for upcoming 

steps to sustain the project’s improvements.

An improvement team may deliver two types of reports:

• An oral report to the quality committee and key stakeholders including the information shown in the Toolbox 

on page 130.

• An written report using descriptive pictures and graphics 

as noted in the real-world tip on storyboards.

Additional Resource

For guidance in teaching small groups about the basic criteria for evaluating pilot test results, see the HIVQUAL Group Learning Guide "Evaluating a Pilot Test" exercise. You can download this publication at www.hivqual.org.

Notes

Evaluate Results 
NYSDOH AI HIVQUAL Workbook September 2006

134

Project Step 6: Systematize Changes.

The Big Picture

Without a concerted effort to incorporate project changes 

into daily work procedures, the gains for which team 

members have invested significant time and resources are 

likely to recede over time. During this step, team members 

plan how to integrate their project improvements in daily 

routines and how to best sustain those improvements. 

They also evaluate how effectively the team performed in 

achieving the project goals.

What To Do

● Sustain improvements.

● Assess team effectiveness.

Project Cycle

Program Cycle

Creating Quality By Spreading And 

Integrating Change

Creating quality by implementing the successful changes is challenging. Resistance to change, no matter how small, should be 

anticipated by the quality committee. The quality committee can 

mitigate this resistance through a variety of techniques—from 

providing staff training and education about the change, providing continuous feedback and support to staff, standardizing the 

practice, to providing incentives to best performers or “champions 

of the change.” 

Snapshot of HIV Care

Spreading Quality Improvement Throughout 

The Facility

Quality improvement activities have been sustained at SUNY 

Downstate in Brooklyn, NY according to Kristina Brown, Quality 

Improvement Coordinator, because quality management is incorporated into the staff’s routine schedule. At the monthly, regularly 

scheduled committee meetings, “everyone has to be prepared to 

discuss whether they’ve worked on PDSA cycles, what’s worked and 

what hasn’t worked.” 
NYSDOH AI HIVQUAL Workbook September 2006

135

Snapshot of HIV Care...Continued

Deadlines also help keep staff focused—“everyone knows that 

by the 15th of the month, data have to go to the data collection 

person.” Quality management is also on the agenda of every staff 

meeting and there’s a scheduled quarterly presentation of the 

performance measurement data so everyone knows “how we’re doing, where we need to improve, and what we need to focus on 

in the next quarter.”

Brown believes that having this structure has been the key to 

successfully sustaining change. “There’s no month when quality 

improvement isn’t being addressed, even during those times when 

too many other things are going on. There’s no let up.”

Quality improvement has to be incorporated into facility’s routines, 

with leadership support. “If you let it go and consistency and 

accountability is not there, then all of the work you’ve done will 

quickly go by the wayside,” concludes Brown.

Don’t Rely On One Person To Sustain The 

Quality Program

One of the challenges facing quality programs, according to quality 

consultants, is the over-reliance of a facility’s efforts on one person. 

All too often, facility’s success with quality management has been 

do to hard and sustained work of one or a small group of committed staff. “If these driving forces leave the facility, or take on new 

duties, then the quality management program can fall apart. Leadership need to continuously ensure that the quality management 

program can be sustained through routine staff changes,” cautions 

a quality consultant. 

“Quality, like other services and functions, can’t be the responsibility of only one person. The institution must support quality by 

integrating it into all staff functions such as training, communications, job descriptions, and employee evaluations.”

While leadership is crucial to establishing, implementing and maintaining quality efforts at a facility, leaders also need to recognize 

their appropriate role. A quality manager reports that the founding 

physician at her facility, while expressing support for the quality 

management program, was undermining its impact because he 

continued to make changes at the agency on his own and ignored 

the stated role of the quality committee. 

“It was difficult for the staff to challenge him because he was the 

boss. Finally, during our annual quality management plan review 

that we were able to reach an agreement with him that he would use 

the quality committee to test and make changes. Unfortunately, 

had lost time and spent valuable resources because we spent a lot of 

time focusing on this problem, rather than on quality issues.”

Systematize 

Changes 
NYSDOH AI HIVQUAL Workbook September 2006

136

Sustain Improvements.

Every improvement project reaches an end point. Before the 

team completes its project work it is important to take the 

time to systematize the changes the team has meticulously 

tested and implemented. The goal is to institutionalize successful improvements so that they become the new ‘status 

quo’ and that the gains are sustained over time. 

The project team members who are most familiar with the 

processes and the facilities quality committee plan should 

work together on identifying how to sustain the new level of 

performance. Have a written action plan for systematizing 

and sustaining the improvements. The identified strategies 

are integrated into the overall quality management plan and 

annual workplan.

The following elements are helpful to maintain 

long-term effects of implemented gains:

● Re-measure performance level

● Educate staff to support improvements

● Identify a champion of change

● Ensure ownership of change

● Institutionalize changes

The Toolbox on page 137 shows a template for an action 

plan that may be used to document steps and timelines.

Re-measure Performance Level

The process that has been improved by the quality improvement team should be re-measured routinely to ensure that 

gains have been kept on time. A timetable for the re-collection 

of data (quarterly or bi-annually) should be created including a designating staff responsible for data collection and to 

whom the findings will be reported. The re-measurement of 

data should be integrated into the facility’s quality program 

and become part of the routine performance measurement 

process. The team should also decide on any necessary 

thresholds to trigger the attention of the quality committee or 

to reconvene the improvement team.

Additional Resource

For guidance in teaching small groups about the tasks required to sustain quality gains, see the HIVQUAL Group Learning Guide "Systematize 

Improvements" exercise. You can download this publication at www.hivqual.org.
NYSDOH AI HIVQUAL Workbook September 2006

137

Toolbox: 

Action Plan for Systematizing Improvements

Improvement Goal: Increase Annual Mental Health (MH) Screening for all HIV-infected Patients to 65 percent 

and Sustain Performance Level.

Task

Re-measure existing indicator

Educate providers using the 

new MH screening form

Identify staff person to lead 

MH screening efforts

Inform staff about the 

MH screening tool

Systematize MH screening tool 

into the fabric of the clinic

Re-measure performance level

Educate staff to 

support improvements

Identify a champion of change

Ensure ownership of change

Institutionalize change

Next Steps Timeline

• Decide at next meeting on 

 frequency of review 

• Work with MIS department to

 automate MH report 

• Invite outside speaker on 

this topic

• Arrange routine case conferences on specific cases

• Provide orientation to staff

• Formalize John as lead to 

sustain gains

• Send John to conference on 

this topic

• Explain the role of staff in the 

new process

• Showcase new tool and process 

on poster

• Review and revise all policies to 

integrate MH screening in job 

 descriptions

• Provide training on MH 

screening process during new 

employee orientation

April '06

May '06

April '06

May '06

April '06

STRATEGY

Systematize 

Changes 
NYSDOH AI HIVQUAL Workbook September 2006

138

Educate Staff To Support 

Improvements

Some level of training will be necessary to ensure that staff 

understands new tools and process changes, as well as their 

new roles and responsibilities in implementing the planned 

improvements. The specific nature of the quality improvement will dictate the type of training that is appropriate. For 

example, a printed worksheet of instructions or a laminated 

checklist hung near a workstation may suffice. Improvements that require more critical knowledge-based tasks or 

complex skills may require training incorporating some 

degree of problem-solving and decision-making. At a fundamental level, all staff involved in the improvement should 

receive consistent information.

Identify A Champion Of Change

A staff person who has intimate knowledge of the improvements is identified to become the internal champion of 

change. Staff or consumers can contact the person to ask 

follow-up questions or clarify certain details. This champion 

becomes the ‘human face’ of the new status quo and a visual 

reminder to sustain changes.

Concerted efforts are required to ensure gains achieved during quality improvement projects are sustained. The following tips can help sustain quality improvement gains:

• Assign one member of the improvement project team 

to monitor progress and routinely report back to the 

quality committee.

• Assign specific performance thresholds to trigger 

follow-up activities.

• Decide on a simple format that can be used to generate an 

on-going status report for the quality committee, 

staff and consumers.

• Provide simple incentives such as letters of recognition 

or small monetary incentives (e.g., gift cards or movie 

passes) to staff that support improvements.

• Highlight the advantages of the new process 

whenever possible.

Real-World Tip

Plan To Promote Quality Gains.
NYSDOH AI HIVQUAL Workbook September 2006

139

Ensure Ownership Of Changes

Steps taken to ensure the ownership of changes help make 

the improvement ‘part of the fabric’ of the HIV facility. Every opportunity is used to promote the new status quo to the 

entire staff either at staff meetings, through storyboards in 

hallways or in mini-presentations. Changes are also clearly 

communicated to new staff members to integrate these improvements in their daily work. These activities send a clear 

message that quality activities are everyone’s responsibility to 

keep the momentum of change going.

Institutionalization Of Changes

Finally, the project team reviews and revises, if necessary, 

the HIV program’s policies and procedures to ensure that 

new processes are documented. The team may also consider 

screening involved job descriptions and making appropriate 

changes. The process of sustaining the project gains is clearly 

communicated to the facility’s leaders and key stakeholders. 

System-wide buy-in also helps to overcome staff resistance 

and barriers in the implementation process. For more 

information on sustaining the quality program momentum, 

please see Program Step III on page 69. 

Systematize 

Changes 
NYSDOH AI HIVQUAL Workbook September 2006

140

Assess Team Effectiveness.

A team self-evaluation is the team’s final task. A team looks 

at lessons learned throughout the improvement project and 

identifies how future team efforts could be more effective 

and efficient in their efforts.

Evaluating team performance

Taking time to evaluate the improvement team’s performance will identify key learnings from the project. Team 

members can discuss the positive aspects of their group 

interactions along with the areas that need improvement. 

In this way, current project teams ‘sow the seeds’ for the 

success of future improvement teams. Evaluation also signals 

the end of an improvement project and provides a sense of 

closure for team members.

The Toolbox on page 141 provides two approaches to selfevaluation. Results of the team evaluation should be written 

up and shared with the overall quality committee and future 

project teams. How the quality committee should proceed 

and evaluate the project team performance is described in 

Program Step III on page 69.

Celebrate success

At this point it is important to celebrate the accomplishment of the project team. It stresses the significance of 

quality to the facility and creates some tangible excitement 

about successful changes. The quality committee can use 

this opportunity for staff recognition and announcements 

of future quality projects.

Additional Resource

For guidance in teaching small groups about the importance of post-improvement project team evaluation, see the HIVQUAL Group Learning 

Guide "Team Self Evaluation" exercise. You can download this publication at www.hivqual.org.

Every improvement project reaches an end point. Here are 

some suggestions to promote closure:

• Celebrate successes with a ‘lunch party.’

• Get leaders to acknowledge the hard work by the team. 

• Identify a spokesperson for the team who will present on 

the behalf of the team. 

• Summarize and archive the team’s efforts before the 

memories fade. 

• Promote the successes of this team to future 

team members. 

Real-World Tip

End Well.
NYSDOH AI HIVQUAL Workbook September 2006

141

Toolbox: 

Team Self-Evaluation Approaches

Method One

Write the following three questions about team effectiveness on flip chart paper and ask each team members to answer 

them from their own perspective:

• What worked well in the team?

• What needed to be improved?

• What do you suggest to improve future quality improvement teams?

Method Two

Add categories to brainstorm under each of the above questions to ensure that feedback is given on certain categories of 

experience, such as:

• Team composition and roles

• Project improvement process (e.g., goals, understanding of the process and problems, pilot tests)

• Length of time and resource allocation

• Program/organizational support

• Teamwork and distribution of tasks

Systematize 

Changes 
NYSDOH AI HIVQUAL Workbook September 2006

142

This additional resource is intended to supplement any 

documentation about performance measurement in this 

Workbook, including Project Step 1: Review, Collect and 

Analyze Project Data on page 82, and Program Step II: 

Establishing Performance Measurement Systems on page 57. 

Information and Toolbox examples in this section 

explain how to:

• Construct a population sample

• Design a data collection tool

• Assign and train abstractors

• Validate results

Sampling and Data Collection
NYSDOH AI HIVQUAL Workbook September 2006

143

Construct A Population Sample.

In many situations it is unrealistic and inefficient to collect 

data from every patient file. Data sampling allows teams to 

make inferences about a total patient population based on 

observations of a smaller subset of that group (the sample), 

saving both time and resources during data collection. Most 

facilities utilize random sampling, wherein medical records 

are drawn from the total population in such a way that each 

time a record is selected every record in the population has 

an equal opportunity to appear in the sample.

The goal of sampling is to reduce the amount of work 

involved in chart reviews for data collection, analysis and 

reporting. However, if the facility’s information system can 

access all the data that are needed for the indicators, a 

project team can skip ahead to the task: "Design a data 

collection tool," on page 147.

Defining Selection Criteria

Criteria determine who will be included in the sample. 

Examples of selection criteria include:

• Location. What facilities within the HIV care 

system will be included?

• Gender. Does the indicator apply to men, 

women, or both?

• Age. Does the indicator have particular age limits?

• Patient conditions. Does the indicator apply to all HIVinfected patients or is a specific diagnosis required? Is a 

confirmation of the diagnosis required, or is an empiric 

diagnosis acceptable? Do certain conditions make the 

patient ineligible?

• Treatment status. How many visits are required for eligibility? Must the patient currently be in treatment? Must 

the treatment have occurred within a specific time frame?

In the National HIVQUAL Project, the sample population 

includes male and female HIV patients who were active during a calendar year (for example, 1/1/2005 to 12/31/2005). 

Active cases are defined as those with at least two medical 

visits during the study period, with at least one visit during 

the last six months during the review period. Patients who 

expired before the end of the review period, but meet the 

visit criteria, are eligible for inclusion.

Additional Resource

For guidance in teaching small groups about how to define selection criteria, identify eligible cases and randomly select cases, see the HIVQUAL 

Group Learning Guide "Constructing a Sample" exercise. You can download this publication at www.hivqual.org. Other resource: Carey, R.G., 

and Lloyd, R.C. Measuring Quality Improvement in Healthcare. New York: Quality Resources, 1995.

Appendix 
NYSDOH AI HIVQUAL Workbook September 2006

144

Identifying Eligible Cases

Project teams will need to separate out the medical records 

that are eligible for measurement based on the selection 

criteria. The reporting capabilities of the facility’s computer system will determine how much work will need to 

be done by hand. When querying the facility management 

information system, the answers written for “defining your 

population focus” will need to be completed. For example, 

a system might specify all male and female HIV-infected 

patients over the age of 17 who made at least two visits 

between 1/1/2005 and 12/31/2005, with one of those visits 

in the last six months of the study period from 7/1/2005 to 

12/31/2005. If the system will provide only a list of names 

for at least two visits within the year, but not specify within 

the last six months of the study period, then the list is reviewed and those names that did not have a visit within the 

last six months crossed off. Likewise, if the system will provide only a list of patients seen during the study period, then 

the list is carefully reviewed to identify the eligible cases.

To calculate the total number of eligible patients, you need 

to first divide the entire case list into male and female lists to 

ensure sufficient numbers for the GYN indicator. These lists 

will now provide you with two numbers, the total number of 

eligible males and the total number of eligible females. (The 

total eligible population is the sum of the two.) 

Identifying Sample Size

The minimum sample size for an accurate measurement is 

based on the number of eligible cases. Some facilities use 

pre-existing sample tables to determine a project’s minimum 

sample size based on their own requirements.

The Toolbox on page 145 shows the HIVQUAL Sample Size 

Chart. It indicates the minimum number of records to be 

pulled for chart review for men and women based on the 

total number of eligible cases. The maximum number of 

records to be reviewed is 107 though it depends on the facility’s caseload. To ensure that women are properly represented in the GYN portion of the review, this table oversamples 

women. For example, if an agency has 55 eligible female 

patients and 90 eligible male patients (145 total eligible 

patients), then the minimum number of female records to be 

reviewed is 39. The number of male records to be reviewed 

is determined by subtracting the number of female records 

to be reviewed from the total number of records: 64 total 

records-39 female records=25 male records.
NYSDOH AI HIVQUAL Workbook September 2006

145

Toolbox: 

HIVQUAL Sample Size Chart

This chart is based on a 90 percent confidence interval with an error width of 16 percent when using the 

minimum number of records.

Up to 20

21 - 30

31 - 40

41 - 50

51 - 60

61 - 70

71 - 80

81 - 90

91 - 100

101 - 119

120 - 139

140 - 159

160 - 179

180 - 199

200 - 249

250 - 299

300 - 349

350 - 399

400 - 449

450 - 499

500 - 749

750 - 999

1000 - 4999

5000 or more

Minimum total

records

charts to

pull

All

24

30

35

39

43

46

49

52

57

61

64

67

70

75

79

82

85

87

88

94

97

105

107

All

31

39

46

51

56

60

64

68

74

79

83

87

91

98

103

107

111

113

114

122

126

137

139

Total Eligible

Population

TOTAL SAMPLE TABLE

Up to 20

21 - 30

31 - 40

41 - 50

51 - 60

61 - 70

71 - 80

81 - 90

91 - 100

101 - 119

120 - 139

140 - 159

160 - 179

180 - 199

200 - 249

250 - 299

300 - 349

350 - 399

400 - 449

450 - 499

500 - 749

750 - 999

1000 - 4999

5000 or more

Minimum FEMALE 

records

charts to

pull

All

24

30

35

39

43

46

49

52

57

61

64

67

70

75

79

82

85

87

88

94

97

105

107

All

31

39

46

51

56

60

64

68

74

79

83

87

91

98

103

107

111

113

114

122

126

137

139

Total Eligible

FEMALES

FEMALE SAMPLE TABLE

Appendix 
NYSDOH AI HIVQUAL Workbook September 2006

146

Constructing The Sample

Using the number of eligible female patients, determine 

the minimum number of female records needed from the 

HIVQUAL Sample Size Chart (page 145). To determine 

the number of male records you need, subtract the minimum number of female records from the total minimum 

number of records. 

Select the required number of records required from the 

total eligible records using a random process. Be aware that 

each medical record should have an equal change of being 

included in the sample. Most spread sheet programs offer 

random number tables.

The Tool box on page 148 provides a sample random number table and calculation instructions. 

Notes
NYSDOH AI HIVQUAL Workbook September 2006

147

Design A Data Collection Tool.

Data collection consists of the tools and techniques used 

to collect baseline and follow-up data, typically from 

patient medical records. For example, a project team may 

collect data on how many patients received PCP prophylaxis in the previous year. The final percentage is a starting 

point for process improvement and a baseline for 

future measurements.

The goal of a data collection tool is to help obtain the most 

current, complete and accurate information possible. If data 

are collected with little regard for accuracy, project results 

are jeopardized. 

Developing The Tool

To ensure that data are collected as intended, detailed instructions to guide abstractors through the data collection 

process needs to be written. Agreement needs to be found 

on what data are needed and why. Tools such as data entry 

forms and procedure checklists help lower the margin for 

collection error.

Those responsible for the review process should first create a 

paper or electronic document listing the review criteria. The

form should include eligibility criteria, and question/response parameters that define the ‘yes’ and ‘no’ responses, 

and N/A conditions. The form should be straightforward 

and concise to facilitate accurate data collection as well as 

any future re-measurement of data.

The following guidelines are useful when developing a data 

collection tool:

• Decide on the evaluation questions and 

information needed to answer those questions.

• Choose a format for collecting data such as 

questionnaire, discussion guide, interview, etc. consistent 

with the types of questions to be asked on the tool. For 

example, "Was a Pap smear performed on this female 

patient between 12/31/2005 and 1/1/2005?"

• Agree on the interpretation of what the responses mean 

(‘yes’, ‘no’, ‘NA’).

• Create results for each question. This will provide feedback on whether the results are meaningful. 

If not, revise the questions or responses or 

consider eliminating it.

• Develop and field test a data collection sheet. 

The Toolbox on page 152 illustrates a detailed example of a 

data collection tool.

Additional Resource

You can download a software program, called HIVQUAL3, at www.hivqual.org at no cost. The software that includes indicators based on clinical 

practice guidelines provides an efficient means of measuring and reporting clinical performance.

Appendix 
NYSDOH AI HIVQUAL Workbook September 2006

148

Toolbox: 

 Random Number Table 

To use random numbers for case selection, imagine that the list of eligible patients either male or female, not both, has 

been numbered sequentially, starting with one. Based on your sample size, select those cases whose assigned numbers correspond with the random numbers listed in the appropriate random number calculation. If a random number exceeds the 

final number on the list, continue counting sequentially from the top of the list, including only those cases that have not 

yet been chosen. An example follows on page 150.

RANDOM NUMBER table

Eligible Cases=21-30; Minimum Total Records=24

1, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19, 21, 22, 23, 

24, 25, 26, 27, 28, 29

Eligible Cases=31-40; Minimum Total Records=30

1, 2, 3, 4, 5, 6, 9, 11, 12, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 

26, 27, 28, 29, 31, 33, 34, 35, 37, 39, 40

Eligible Cases=41-50; Minimum Total Records=35

1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 14, 15, 17, 20, 22, 23, 24, 28, 

30, 31, 32, 34, 35, 37, 38, 39, 40, 42, 43, 45, 46, 47, 48, 50

Eligible Cases=51-60; Minimum Total Records=39

1, 3, 4, 5, 6, 8, 9, 12, 14, 16, 17, 19, 21, 26, 27, 28, 29, 30, 31, 

32, 34, 35, 36, 37, 38, 39, 41, 42, 43, 44, 45, 47, 48, 52, 54, 

55, 56, 57, 60

Eligible Cases=61-70; Minimum Total Records=43

1, 2, 4, 7, 9, 10, 12, 13, 14, 16, 19, 22, 24, 25, 26, 27, 29, 31, 

32, 33, 34, 35, 36, 37, 40, 41, 42, 44, 46, 48, 49, 51, 52, 54, 

56, 57, 60, 62, 64, 65, 66, 69, 70

Eligible Cases=71-80; Minimum Total Records=46

2, 3, 4, 5, 6, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 24, 

27, 30, 32, 33, 36, 37, 41, 44, 48, 49, 50, 53, 54, 55, 56, 57, 

59, 60, 62, 63, 66, 69, 70, 72, 76, 77, 79, 80

Eligible Cases=81-90; Minimum Total Records=49

1, 2, 3, 5, 6, 9, 10, 11, 12, 13, 16, 17, 19, 20, 23, 26, 29, 34, 

35, 38, 42, 43, 44, 46, 48, 50, 52, 54, 56, 57, 58, 59, 60, 61, 

63, 64, 65, 66, 68, 69, 70, 71, 73, 76, 77, 80, 81, 84, 86

Eligible Cases=91-100; Minimum Total Records=52

1, 2, 6, 9, 10, 13, 14, 15, 19, 22, 26, 28, 30, 32, 33, 34, 35, 

37, 38, 40, 42, 43, 44, 45, 46, 47, 48, 51, 54, 55, 56, 58, 59, 

60, 61, 70, 71, 74, 75, 79, 80, 81, 85, 86, 88, 92, 93, 94, 96, 

97, 98, 99

Eligible Cases=101-119; Minimum Total Records=57

1, 2, 5, 6, 8, 11, 12, 13, 16, 18, 19, 20, 21, 22, 26, 35, 36, 39, 

41, 43, 45, 47, 48, 49, 50, 51, 52, 57, 60, 62, 65, 72, 74, 76, 

77, 78, 80, 81, 82, 83, 84, 85, 86, 88, 89, 92, 93, 94, 97, 102, 

103, 104, 106, 111, 114, 118, 119

Eligible Cases=120-139; Minimum Total Records=61

3, 4, 5, 7, 16, 17, 19, 23, 24, 27, 29, 30, 36, 38, 39, 45, 48, 

50, 53, 54, 59, 61, 62, 64, 65, 66, 67, 69, 71, 76, 77, 79, 82, 

83, 85, 86, 87, 88, 91, 92, 96, 100, 101, 103, 105, 106, 108, 

112, 113, 114, 116, 124, 125, 126, 128, 129, 130, 133, 134, 

135, 138

Eligible Cases=140-159; Minimum Total Records=64

2, 5, 14, 16, 18, 21, 25, 26, 27, 28, 30, 31, 32, 33, 37, 40, 41, 

42, 43, 44, 48, 49, 51, 55, 56, 59, 61, 63, 67, 74, 75, 81, 82, 

85, 86, 87, 88, 89, 90, 91, 93, 98, 102, 105, 107, 114, 118, 

121, 122, 129, 130, 131, 136, 140, 145, 146, 147, 149, 151, 

152, 155, 157, 158, 159

Eligible Cases=160-179; Minimum Total Records=67

2, 5, 9, 11, 14, 16, 18, 27, 29, 30, 34, 38, 39, 41, 42, 48, 52, 

53, 57, 59, 60, 72, 73, 77, 83, 88, 93, 96, 99, 100, 102, 104, 

105, 107, 108, 109, 110, 111, 114, 116, 118, 119, 120, 121, 

124, 126, 131, 134, 138, 141, 147, 149, 150, 152, 153, 154, 

155, 156, 157, 163, 170, 171, 172, 173, 177, 178, 179

Eligible Cases=180-199; Minimum Total Records=70

2, 4, 5, 7, 12, 20, 21, 26, 29, 32, 37, 40, 41, 44, 45, 50, 51, 53, 

55, 60, 61, 62, 65, 74, 76, 78, 80, 86, 88, 90, 91, 92, 95, 101, 

103, 104, 110, 116, 122, 123, 124, 125, 128, 130, 131, 136, 

140, 141, 143, 144, 149, 155, 159, 162, 163, 168, 169, 170, 

173, 177, 180, 182, 185, 186, 190, 193, 194, 195, 197, 198

Eligible Cases=200-249; Minimum Total Records=75

1, 3, 5, 6, 10, 13, 16, 18, 20, 26, 30, 31, 35, 36, 44, 48, 51, 

52, 53, 55, 60, 72, 73, 76, 77, 81, 85, 87, 89, 99, 109, 112, 

118, 122, 123, 124, 127, 130, 134, 139, 142, 143, 150, 151, 

155, 160, 165, 172, 177, 180, 181, 192, 195, 196, 199, 200, 

203, 205, 208, 210, 211, 212, 218, 219, 220, 226, 227, 228, 

230, 231, 232, 237, 239, 244, 249

RANDOM NUMBER table
NYSDOH AI HIVQUAL Workbook September 2006

149

Toolbox: 

 Random Number Table...Continued 

Eligible Cases=250-299; Minimum Total Records=79

4, 5, 6, 7, 17, 20, 28, 29, 34, 38, 39, 46, 47, 49, 51, 52, 53, 56, 

58, 67, 72, 75, 79, 81, 82, 87, 99, 106, 112, 125, 134, 136, 

137, 138, 147, 153, 155, 165, 166, 167, 169, 170, 171, 172, 

175, 182, 184, 189, 190, 191, 193, 195, 200, 203, 204, 211, 

216, 220, 223, 226, 231, 234, 242, 243, 251, 255, 256, 262, 

263, 266, 269, 270, 272, 274, 281, 290, 293, 294, 296

Eligible Cases=300-349; Minimum Total Records=82

8, 16, 17, 26, 39, 47, 49, 52, 54, 55, 57, 61, 66, 70, 80, 81, 84, 

86, 93, 95, 96, 110, 113, 116, 123, 124, 132, 139, 140, 141, 

142, 154, 167, 171, 173, 176, 177, 178, 179, 180, 182, 183, 

195, 197, 203, 207, 208, 211, 213, 219, 223, 229, 233, 237, 

238, 248, 251, 253, 258, 260, 266, 272, 278, 283, 284, 289, 

292, 296, 297, 305, 308, 309, 314, 316, 318, 319, 323, 325, 

327, 335, 343, 347

Eligible Cases=350-399; Minimum Total Records=85

2, 4, 6, 9, 13, 14, 34, 41, 44, 54, 57, 58, 66, 85, 87, 90, 92, 

94, 96, 100, 103, 107, 115, 118, 124, 125, 129, 137, 139, 154, 

155, 159, 162, 165, 172, 175, 178, 184, 195, 203, 211, 217, 

220, 225, 226, 228, 230, 241, 242, 250, 251, 252, 257, 262, 

265, 270, 277, 279, 284, 293, 296, 298, 300, 302, 304, 311, 

312, 315, 323, 332, 336, 337, 338, 343, 345, 363, 364, 368, 

372, 373, 375, 376, 390, 393, 398

Eligible Cases=400-449; Minimum Total Records=87

6, 29, 31, 33, 37, 41, 43, 49, 61, 67, 71, 74, 75, 86, 94, 102, 

105, 111, 112, 118, 120, 124, 128, 133, 137, 139, 141, 152, 

159, 163, 171, 185, 187, 191, 193, 195, 196, 200, 214, 220, 

221, 223, 225, 227, 228, 230, 237, 239, 242, 245, 255, 259, 

261, 272, 276, 279, 282, 294, 297, 299, 300, 302, 306, 318, 

319, 320, 324, 347, 352, 362, 366, 373, 374, 380, 381, 387, 

393, 395, 399, 404, 407, 408, 421, 426, 436, 442, 446

Eligible Cases=450-499; Minimum Total Records=88

15, 21, 25, 29, 45, 46, 50, 51, 60, 64, 69, 73, 83, 102, 105, 

109, 115, 118, 127, 137, 142, 145, 149, 157, 160, 162, 169, 

171, 172, 178, 187, 190, 194, 206, 218, 229, 236, 249, 252, 

253, 256, 263, 291, 310, 319, 321, 326, 329, 332, 344, 

349, 355, 374, 377, 378, 381, 382, 384, 385, 389, 393, 394, 

395, 399, 403, 412, 421, 425, 426, 434, 440, 441, 453, 454, 

459, 462, 464, 465, 469, 475, 476, 480, 481, 484, 486, 487, 

497, 498

RANDOM NUMBER table

Eligible Cases=500-749; Minimum Total Records=94

1, 15, 20, 30, 47, 64, 77, 81, 93, 99, 125, 127, 171, 182, 190, 

221, 233, 235, 247, 253, 264, 267, 273, 281, 284, 293, 298, 

306, 310, 319, 325, 329, 335, 341, 345, 357, 359, 366, 370, 

377, 385, 393, 395, 401, 408, 409, 412, 416, 421, 426, 430, 

440, 468, 472, 475, 496, 497, 501, 504, 505, 510, 518, 528, 

536, 540, 552, 574, 576, 580, 581, 582, 587, 588, 598, 602, 

611, 618, 628, 640, 665, 667, 674, 675, 676, 687, 691, 717, 

730, 732, 737, 738, 742, 744, 747

Eligible Cases=750-999; Minimum Total Records=97

21, 22, 46, 49, 53, 54, 57, 59, 61, 84, 91, 100, 109, 140, 176, 

180, 204, 229, 234, 240, 249, 275, 277, 287, 297, 311, 313, 

324, 330, 346, 354, 355, 358, 380, 385, 399, 401, 407, 414, 

434, 443, 449, 461, 466, 479, 491, 499, 512, 530, 542, 543, 

557, 562, 580, 595, 616, 653, 666, 676, 687, 694, 701, 702, 

708, 711, 719, 721, 746, 756, 778, 781, 791, 795, 802, 813, 

817, 824, 834, 843, 844, 864, 877, 878, 884, 897, 898, 901, 

913, 915, 933, 957, 973, 980, 986, 993, 994, 997

Eligible Cases=1000-4999; Minimum Total Records=105

40, 100, 300, 322, 349, 406, 489, 496, 541, 581, 649, 656, 

660, 707, 745, 778, 779, 800, 902, 917, 955, 1030, 1083, 

1105, 1173, 1179, 1202, 1214, 1296, 1344, 1373, 1442, 1501, 

1527, 1570, 1578, 1608, 1705, 1742, 1757, 1771, 1774, 1778, 

1800, 1843, 1872, 1880, 2040, 2112, 2426, 2494, 2530, 

2558, 2611, 2790, 2960, 3048, 3076, 3117, 3159, 3225, 3235, 

3324, 3331, 3351, 3403, 3450, 3463, 3525, 3529, 3555, 3605, 

3685, 3752, 3758, 3835, 3916, 3919, 3920, 3934, 3935, 3941, 

4045, 4093, 4125, 4145, 4170, 4240, 4396, 4436, 4467, 4522, 

4537, 4560, 4590, 4642, 4668, 4670, 4747, 4811, 4830, 4868, 

4903, 4937, 4986

Eligible Cases=5000 or more; Minimum Total Records=107

(visit www.randomizer.org)

RANDOM NUMBER table

Appendix 
NYSDOH AI HIVQUAL Workbook September 2006

150

Toolbox: 

 Random Number Table Calculations 

A team pulls a sample out of a total of 46 eligible records (20 male and 26 female patients).

The random number calculation for females is based on the random number table as follows:

Total eligible female records=26; Minimum female records=24

Random number calculations: 1, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 27, 28, 29

To begin, the team numbers the list of eligible female records from 1 to 26 and selects the first round of records as indicated by the random number table calculations: 1, 4, 5, 6, 7, 9, 10, 11, 13, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, and 26. 

Since the next random number is 27, the team continues counting from the top of the record list—but this time excludes 

those records that have already been selected.

FIRST ROUND

1. Record 1

4. Record 4

5. Record 5

6. Record 6

7. Record 7

8. Record 8

9. Record 9

10. Record 10

11. Record 11

12. Record 12

13. Record 13

14. Record 14

15. Record 15

16. Record 16

17. Record 17

18. Record 18

19. Record 19

20. Record 20

2. Record 2

3. Record 3

21. Record 21

22. Record 22

23. Record 23

24. Record 24

25. Record 25

26. Record 26

second ROUND

27. Record 2

28. Record 3

29. Record 8
NYSDOH AI HIVQUAL Workbook September 2006

151

Assign And Train Abstractors.

Data abstractors are those HIV staff that collect measurement data from the sample. 

Assigning Abstractors

At a minimum, staff members chosen to be abstractors 

should be familiar with medical records and relevant terminology. If possible, providers should be engaged in a peerreview process to broaden the sense of ownership and learn 

from each other. Other factors to consider are availability 

and comfort level working with data.

The collection process should have clearly defined roles. A 

single staff member might oversee the data collection process 

while others remain available to answer questions that arise.

Since data extraction from charts can be tedious for one 

person, many HIV facilities find it helpful to assign a few 

HIV clinical staff do this activity together for a few hours 

at a time. The benefits of this approach are:

• Completing the task in a timely manner.

• Applying the same interpretation to answering 

the questions.

• Expanding understanding of both the process 

of data extraction and identification of issues involved.

Training Abstractors

Whether the project team collects the data or other staff is 

involved, some time for an informal training session should 

be scheduled as close to the actual time of data collection as 

possible. A designated staff member should begin by reviewing the basic purpose of performance measurement and the 

specific clinical aspects of the subject under review. Abstractors should be given a ‘walk through’ of the data collection 

process. Patient eligibility criteria and where to find required 

information in patient charts should be clearly indicated. 

Additional Resource

For guidance in teaching small groups about the characteristics of a well-designed data collection system, see the HIVQUAL Group Learning 

Guide "Data Collection" exercise. The exercise could also be used to assess the strengths and weaknesses of an existing data collection system. You 

can download this publication at www.hivqual.org.

Appendix 
NYSDOH AI HIVQUAL Workbook September 2006

152

Toolbox: 

 Data Collection Tool 

ARV Management (Eligibility: All Patients on ARV Therapy)

For each trimester (every 4 months), how do you assess the patient’s stability?

o Stable: Was a viral load performed within each trimester? o Yes o No

o Unstable:

If ARV medication was changed, was a viral load performed within 8 weeks? o Yes o No

If ARV medication was stopped, were decision and clinical follow up documented within three months? o Yes o No

If ARV medication unchanged, was justification documented? o Yes o No

If medication was started or changed in this trimester, did the patient get treatment education? o Yes o No

Adherence to ARV Therapy (Eligibility: All Patients on ARV Therapy)

Was adherence discussed with the patient each trimester (every 4 months)?

o	 Yes: Adherence discussion documented

Was an adherence problem identified? oYes (Was the adherence problem addressed? o Yes o No)

 oNo 

o	 No: No documentation of adherence discussion

HIV Specialist Care

Was the patient seen by an HIV Specialist at minimum every 4 months?

o	 Yes: Patient was seen at least every 4 months by an HIV Specialist. 

o	 No: Patient was not seen by an HIV Specialist.

CD4 Cell Count 

 Was a CD4 count performed within each trimester (at minimum 3 times/12 months)?

o	 Yes: A CD4 count was documented in each trimester. 

o	 No: A CD4 count was not documented in each trimester.

Viral Load 

 Was a viral load performed within each trimester (at minimum 3 times/12 months)?

o	 Yes: A viral load was documented in each trimester. 

o	 No: A viral load was not documented in each trimester.

Lipid Screening (Eligibility: All Patients on ARV Therapy)

Was a Lipid Profile Done? 

o	 Yes: Lipid profile was done within the last 12 months. 

o	 No: Lipid profile was not done within the last 12 months.

PCP Prophylaxis 

 Did the patient with fewer than 200 CD4 cells and no sustained CD4 cell increase > 200 during the last 6 months of the review period 

 receive PCP prophylaxis?

o	 Yes: The patient received PCP prophylaxis. 

o	 No: The patient did not receive PCP prophylaxis.

MAC Prophylaxis (Eligibility: CD4 cells <_50 and no sustained CD4 cell increase to >_50 during the last 3 months of review period)

Did the patient receive MAC prophylaxis?

o	 Yes: The patient received MAC prophylaxis. (CBC done within 6 months? o Yes o No)

o	 No: The patient did not receive MAC prophylaxis.
NYSDOH AI HIVQUAL Workbook September 2006

153

Toolbox: 

 Data Collection Tool...Continued 

Annual Discussion of Substance Use 

Was substance use discussed with the patient during the past 12 months?

o	 Yes: Substance use was discussed. 

o	 Current user (within 6 months) 

If a substance was injected, was safer injection/needle exchange addressed? o Yes o No 

Patient in treatment during review period? o Yes o No If no: o Treatment discussed; no referral made 

 o Treatment discussed; referral made

 o Treatment not discussed

o	 Past user only (last use over 6 months); Prevention/ongoing treatment discussed? o Yes o No o No current use (within 6 

months) or past use (over 6 months) identified

o	 No: Substance use was not discussed with the patient during the past 12 months. 

Annual Syphilis Serology 

Was a syphilis serology performed within the past 12 months?

o	 Yes: A syphilis serology was performed. (If yes, was a confirmation test performed? o Yes o No)

o	 No: A syphilis serology was not performed within the past 12 months.

Hepatitis C (HCV) Screening

Was the patient’s hepatitis C status known?

o	 Yes: HCV screening positive

Was patient’s hepatitis A status known? o Yes, seropositive

 o Yes, seronegative (Was patient offered hepatitis A vaccination? o Yes o No)

 o No

Was alcohol counseling documented within the last 12 months? o Yes o No

Was hepatitis C education documented within the last 12 months? o Yes o No

o	 Yes: HCV screening negative

o	 No: No screening performed

Ophthalmology Exam (Eligibility: CD4 cells < 50)

Was an ophthalmology exam documented within the last 12 months?

o	 Yes: An exam was done within the last 12 months. o	 No: An exam was not done within the last 12 months.

Annual Pelvic Exam 

Was a pelvic exam performed within the past 12 months?

o	 Yes: A pelvic exam was recorded. 

o Pap done (Abnormal? o Yes o No If abnormal, 2nd Pap or GYN referral? o Yes o No)

o Gonorrhea culture done? 

o Chlamydia screening done? 

o	 No: A pelvic exam was not recorded within the past 12 months.

TB Screening (Eligibility: HIV- infected patients without a history of previous TB treatment or a history of a positive PPD) 

 Was PPD testing performed (placed and read within 72 hours) in the past 12 months?

o	 Yes: PPD screening was performed (placed and read). 

o	 No: PPD screening was not performed (or it was placed but results not documented). 

Appendix 
NYSDOH AI HIVQUAL Workbook September 2006

154

Toolbox: 

 Data Collection Tool...Continued 

Case Management Coordination of Services

Is coordination of services documented (i.e., case conference or progress notes, other activities involving coordination of services) on at least

 a quarterly basis? o Yes o No

Case Management Follow-Up on Service Plan Goals and Referrals

For each identified client need area, is there follow-up regarding service plan goals every 120 days until achieved? o Yes o No

For each identified client need area, is there follow-up regarding service plan referrals every 120 days until achieved? o Yes o No

Case Management Service Plan Development

Was a case management service plan developed within 45 days of initial client contact? 

o	 Yes: Service plan developed within 45 days of initial client contact.

 Was service plan developed with client participation (i.e., did client sign service plan)? o Yes o No

o	 No: No documentation that service plan was developed within 45 days of initial client contact.

Comprehensive Case Management Assessment

Was a comprehensive case management assessment of client needs performed within 30 days of initial client contact? o Yes o No

Annual Dental Exam 

Was a dental exam performed within the last 12 months?

o	 Yes: A dental exam was performed on __ /__ /__. 

o	 No: A dental exam was not performed. 

Annual Discussion of Tobacco Use 

Was tobacco use discussed with the patient during the past 12 months?

o	 Yes: Tobacco use was discussed. o No: Tobacco use was not discussed. 

Mental Health Screening

Was a mental health screening performed within the last 12 months? o Yes o No

Was a need for mental health referral identified? o Yes o No

Was referral to a mental health provider made? o Yes o No 
NYSDOH AI HIVQUAL Workbook September 2006

155

Abstractors should be allowed to do a "dry run" using a 

sample medical record at the end of the training session. 

This may help reveal any points of confusion and surface 

general questions about the process. In addition, the importance of flagging problem charts and forwarding them 

to the appropriate providers immediately for further review 

and follow-up should be emphasized. Deficient patient care 

always takes precedence over data collection. If reviewers 

discover negligent care in a medical record, they have an 

obligation to immediately alert a person who can investigate 

the issue further. The data from this medical record should 

still be included in the collected data.

Conducting A Dry Run

If possible, a dry run of the measurement process should 

be conducted to assess its efficiency and to ensure that the 

abstractors are able to get the information they need from 

the sample before starting the full-scale measurement. Using two to three records from the total sample, abstractors 

should collect the required information as explained during 

the training session. The project team or quality committee 

should meet with the abstractors to review the results and 

collect feedback about the process. Based on this debriefing, 

the measurement can be tweaked as needed. An additional 

dry run of the revised process may be conducted depending 

on the changes made.

Notes

Appendix 
NYSDOH AI HIVQUAL Workbook September 2006

156

Validate Results.

It is good practice to validate the data collection results to 

ensure the results are correct. An inter-rater reliability testing 

procedure can accomplish this.

Reliability is defined as the consistency of an instrument’s 

measurements when used under the same conditions with 

the same subjects. The reliability of a measure is important 

for ensuring comparability of results over time. Frequently 

used in medical record abstraction, inter-rater reliability is 

defined as the reliability between two or more abstractors 

reviewing the same records. 

Consider the following simple examples of inter-rater reliability testing. If one person is responsible for chart review, 

identify another person to look at approximately 8-10 charts 

to assess the chart abstraction process using consistent 

project data collection tools. If multiple people are responsible for chart review, then each can check a small sample 

(2-4) of the others’ charts. Apply the same process for 

the data entry task.

Based on abstractor feedback and reliability data, the quality 

management committee and/or improvement project team 

should make any necessary changes to the data collection 

process before the next measurement effort. This may require 

that team members modify the process and its related forms, 

retrain abstractors, and/or reconsider the chosen indicator as 

a critical aspect of care.

Additional Resource

The following resource will provide you additional information on reliability testing and statistical analysis beyond the scope of this 

document: Agresti, Alan. An Introduction to Categorical Data Analysis, Wiley Series in Probability and Statistics. Applied Probability 

and Statistics.

Notes
NYSDOH AI HIVQUAL Workbook September 2006

157

Notes

Appendix 
NYSDOH AI HIVQUAL Workbook September 2006

158

Notes
NYSDOH AI HIVQUAL Workbook September 2006

159

Notes
09/2006 HIVQUAL Workbook # 9385