**Evaluation of Written Prescription Information Provided in Community
Pharmacies, 2001**

**Final Report to the **

**U.S. Department of Health and Human Services and **

**the Food and Drug Administration**

**Bonnie L. Svarstad, Ph.D.^a^**

**Principal Investigator **

**Jeanine K. Mount, Ph.D., R.Ph.^b^**

**Co-investigator**

**University of Wisconsin - Madison**

**School of Pharmacy**

**December 21, 2001**

**^a^William S. Apple Professor of Pharmacy and Professor of Sociology,
University of Wisconsin-Madison, School of Pharmacy, 777 Highland Ave,
Madison, WI 53705; Tel: 608-265-2128; FAX: 608-262-5262; Email: **

**^b^Associate Professor of Social and Administrative Pharmacy,
University of Wisconsin-Madison, School of Pharmacy, Madison, WI 53705;
Tel: 608-262-8678; **

**Acknowledgments: We gratefully acknowledge the assistance of Dr.
Nathan Kanous, the clinical consultant who helped develop and refine the
expert evaluation forms for this project. We also thank Mr. Pil Ho Kim
for assisting with expert and consumer evaluations, data processing, and
statistical analysis. We appreciate the contributions of expert and
consumer panelists, consumer facilitators, the University of Wisconsin
Survey Laboratory, the National Association of Boards of Pharmacy, and
Dr. Ellen Tabak and colleagues at the U.S. Food and Drug Administration.
**

**A. INTRODUCTION**

This report describes the methods and findings of a national study to
assess the quality of written prescription information provided
voluntarily to persons receiving new prescription medicines in community
pharmacies. The study was contracted by the U.S. Department of Health
and Human Services and the Food and Drug Administration (FDA) and
conducted in collaboration with the National Association of Boards of
Pharmacy (NABP) and a national panel of experts in pharmacy practice,
pharmacotherapy, drug information, and health communications (Appendix
A). Below is a brief overview of the study and questions to be addressed
in this report. A more detailed description of study methodology,
results, and discussion of findings follows.

The present evaluation had several advantages over the 1999 interim
evaluation of patient information materials provided by community
pharmacies in eight states. First, information materials were obtained
from community pharmacies that were selected randomly from a national
electronic list of licensed pharmacies in the United States. Data based
on a national sample allow decision-makers to make generalizations
having national implications. Second, patient information materials were
collected by trained shoppers who were identified and supervised by a
professional shopper research firm (Second-to-None, Inc.). The shoppers
acted as patients using a standard protocol that involved four new
prescriptions for atenolol, atorvastatin, glyburide, and nitroglycerin.
Shoppers (referred to as "patient-observers") followed a standard
scenario and script, accepted any information that was offered, and paid
cash for the prescriptions. Using trained shoppers and a standard
scenario to collect information minimizes biases that can occur when
using other data collection techniques such as observation by state
inspectors and pharmacy students in different states and/or other
methodologies such as patient or pharmacist report of information
practices.

A third advantage of the study was that patient information materials
were evaluated by both expert and consumer raters. Expert raters focused
on the extent to which the information was sufficiently specific,
comprehensive, scientifically accurate, unbiased, up-to-date, and
consistent with principles known to facilitate legibility and
comprehension of written materials. Consumer raters focused on the
extent to which the information was legible, comprehensible, and useful
from a consumer's perspective. Patient information materials also were
evaluated using an objective method of assessing readability or the
approximate level of education needed to read various materials.

Finally, the expert evaluation forms incorporated criteria recommended
in the 1996 Long-Range Action Plan for the Provision of Useful
Prescription Medicine Information^1^ and

included more explicit check lists for evaluating the adequacy of drug
information provided to patients. These explicit check lists allow more
precise evaluation and description of patient information and may
provide more useful information to health professionals, information
providers, and the public.

The specific research questions addressed in this report were the
following:

• abWhat percentage of patient-observers were given *any written patient
information* with the four study drugs (atenolol, atorvastatin,
glyburide, and nitrogylcerin)?

• abWhat percentage of patient-observers were given *written patient
information that adhered to various criteria used by expert panelists? *

• abWhat percentage of patient-observers were given *written information
that adhered to various criteria used by consumer panelists*?

**B. METHODS **

**B.1. Selection of Pharmacies **

A simple random sample of 384 community pharmacies was selected by the
contractor (FDA) from a national electronic list of pharmacies certified
by the National Council for Prescription Drug Programs (NCPDP). A sample
size of 384 was needed to obtain an estimate that would fall within
[+]{.underline} 5% of the true population value 95% of the time.
Pharmacies were eligible if they were identified as an independent,
chain, or franchise pharmacy within a retail, grocery, or department
store setting. Pharmacies were ineligible if they were identified as an
hospital, clinic, long-term care, mail order, IV infusion, dispensing
physician, Indian Health Service, Veterans Administration hospital, or
other government/federal setting. Pharmacies also were ineligible if
they were located in Alaska, Hawaii, Puerto Rico, the U.S. possessions,
or a state that prohibited the filling of prescriptions for research
purposes. A total of 57,157 pharmacies were eligible using these
criteria. A computer then assigned a sequence number to each eligible
pharmacy, matched a list of randomly generated sequence numbers to
eligible pharmacies with the same sequence numbers, and generated a
final list of 384 pharmacies (with replacements). The list included the
pharmacy's name, address, and telephone number.

The contractor provided the final list of 384 pharmacies with
replacements to Second-to-None, Inc., a professional shopper research
firm responsible for visiting the pharmacies. Before visiting a
pharmacy, the assigned shopper (patient-observer) called the pharmacy to
verify location and hours. A pharmacy was replaced if the original
pharmacy had no telephone listing or had closed (n=27), had disconnected
the telephone or failed to answer it (n=11), no longer filled
prescriptions or only filled prescriptions for certain populations or
members (n=10), or only carried a limited number of medications (n=3).
Several pharmacies also were replaced due to shopper error, scheduling
problems, or no available shopper within an 100 mile radius (n=7). A
total of 384 pharmacies were visited in 44 states. According to
information from NCPDP, 36% of the visited pharmacies were independent
pharmacies, 60% were chain pharmacies, and 4% were franchise pharmacies.

**B.2. Protocol for Collecting Written Patient Information **

Each study pharmacy was visited once by a shopper acting as a patient
(called "patient-observers") with four new prescriptions for atenolol,
atorvastatin, glyburide, and nitroglycerin. The patient-observers were
hired and supervised by the professional shopper research firm
(Second-to-None, Inc), and prescriptions were written by medical
consultants in the pharmacy's region. Of 384 pharmacy visits, 72% were
made by female patient-observers and 66% were made by patient-observers
who were 45 years or older (mean=50 years old).

Patient-observers were instructed to use a standard scenario as the
script when acting as patient-observers. The standard scenario was
developed by University of Wisconsin researchers and their clinical
consultant (Nathan Kanous, Pharm.D). The scenario defined the
patient-observer's name, address, reason for being in the area, health
and medication history and other information that might be needed while
acting as a patient-observer. Instructions were written out and provided
to the shopper research firm for distribution to patient-observers. The
standard patient - observer scenario follows:

You are to assume the following scenario. You are from (city and state
identified by project manager). You are [visiting or driving
through]{.underline} (pharmacy site area). You received four
prescriptions from your doctor but forgot to fill them before leaving.
You have some heart disease and diabetes, but [no other medical
conditions]{.underline}. [You take no other medications]{.underline} and
[have no drug allergies]{.underline}. You [have not taken these
medications before]{.underline}. You don't have insurance for medication
and therefore [pay with cash]{.underline}.

The reason you now have four new prescriptions is that you were "short
of breath" and had "chest pain" so had to go to the hospital for some
tests. They told you that you had "some heart disease" and that your
cholesterol and blood sugar were too high. The doctor prescribed
medications for the heart (atenolol), chest pain (nitroglycerin),
diabetes (glyburide), and cholesterol (Lipitor). The [doctor was busy
and ]{.underline}did not give any other information. He/she said the
directions would be on the prescription bottles. You have a [follow-up
appointment with the doctor in two weeks]{.underline}.

Patient-observers were instructed to make their approach and
presentation in each pharmacy as uniform as possible. They were asked to
be polite, to act interested, to answer questions briefly and to accept
any oral counseling or written information offered by pharmacy staff. In
addition, patient-observers were asked to avoid volunteering any
information unless asked, to not ask questions, to not initiate "small
talk" and to politely leave the pharmacy if the legitimacy of the
prescriptions was questioned. If patient-observers recognized or were
recognized by any pharmacy personnel, they were instructed to exit the
pharmacy prior to presenting the prescriptions. Patient-observers were
encouraged to role-play or practice the script and learn how to answer
commonly asked questions prior to data collection. See Appendix B for
commonly asked questions and suggested patient-observer responses and
other things to do or avoid in the pharmacies.

Patient-observers were provided with study prescriptions and the name
and address of their assigned pharmacy. They traveled to the pharmacy,
presented the prescriptions to be filled, picked up and paid for the
prescriptions with cash, answered any questions according to the
standard scenario, accepted any information materials that were offered,
and exited the pharmacy. All pharmacy visits were conducted during the
year 2001. After receiving the materials, the shopper research firm
removed pharmacy and pharmacist names from all materials and forwarded
them to the University of Wisconsin for evaluation.

**B.3. Expert Evaluation Procedures **

***B.3.1 Expert Panel.* ** A national panel of 16 experts was nominated
by NABP, investigators, and six professional pharmacy organizations
(Academy of Managed Care Pharmacy, American College of Clinical
Pharmacy, American Pharmaceutical Association, American Society of
Health-System Pharmacists, National Association of Chain Drug Stores,
National Community Pharmacists Association). The 16 expert panelists
include experienced pharmacy practitioners and pharmacy faculty members
specializing in pharmacotherapy, drug information, and health
communications. The panelists' role included nominating drugs to be
included in the study, reviewing and approving the expert evaluation
forms, and rating all patient information sheets collected by
patient-observers.

Before beginning their work, panelists reviewed sections of the 1996
Action Plan for the Provision of Useful Prescription Medicine
Information, including Chapter 3 ("Guidelines for Useful Prescription
Medicine Information") and Appendix G ("Specific Language and Format
Guidelines, with Samples").^1^ Panelists also reviewed FDA approved
labeling information for each study drug and the first drafts of four
Patient Information Evaluation Forms (PIEF) prepared by the clinical
consultant and investigators. Panelists were asked to review the
evaluation forms for consistency with criteria outlined in the 1996
Action Plan and to offer other suggestions for improving the technical
accuracy, clarity, and reliability of the forms as well as the format
and scoring procedures. Their comments were incorporated into revised
drafts of the evaluation forms. This review and feedback process was
repeated until all expert panelists approved the final Patient
Information Evaluation Forms.

***B.3.2. Expert Evaluation Forms and Criteria. ***The final expert
evaluation forms incorporated eight general criteria suggested in the
1996 Action Plan (Table 1). Criterion 1-6 indicated that patient
information must be sufficiently specific and comprehensive; criterion 7
indicated that patient information must be scientifically accurate,
unbiased, and up-to-date; and criterion 8 indicated that patient
information must be readily comprehensible and legible to consumers. To
insure good reliability among raters, the forms also listed 5-10
sub-criteria or operationalizations under each general criterion. The
sub-criteria defined or illustrated what kinds of information had to be
present to be considered acceptable. The total number of sub-criteria
was 61 for atenolol, 62 for atorvastatin, 62 for glyburide, and 63 for
nitroglycerin. Copies of the final evaluation forms listing each
criterion and sub-criterion are provided in Appendices C1 to C4.

***B.3.3. Scoring Procedures.* **For each sub-criterion, raters
indicated whether there was full adherence (2 points), partial adherence
(1 point), or no adherence (0 points). A computer later calculated the
percentage of points obtained for each general criterion and for all
criteria combined. Six levels of adherence were defined for each
criterion:

• abLevel 0: no information

• abLevel 1: information with 0-19% of relevant points

• abLevel 2: information with 20-39% of relevant points

• abLevel 3: information with 40-59% of relevant points

• abLevel 4: information with 60-79% of relevant points

• abLevel 5: information with 80-100% of relevant points

***B.3.4. Assessment of Readability*. **Expert panelists rated all
sub-criteria except sub-criteria 8.7 to 8.10 under Criterion 8.
Panelists suggested that project staff evaluate these particular
sub-criteria as they involved relatively objective assessments of font
size, line spacing, and level of reading difficulty. Level of reading
difficulty was assessed using an objective formula called the Gunning
Fog test^2^. The Gunning Fog test was selected because it is considered
one of the simplest methods of estimating the grade level needed to read
informational materials^3^. The method involves selecting a sample of
100 consecutive words (W), counting the number of sentences (S) in the
sample, and counting the number of words having three or more syllables
(T) that are not combination words, capitalized words, or verbs ending
in "ed" or "es". The final step involves calculating the grade level
needed to read the material by using the Gunning Fog formula: (W/S + T)
x 0.4. In the present study, staff selected a 100-word sample starting
with the paragraph that describes how to use or take the medication.
Since long drug names may result in overestimation of reading skill
required^3^, drug names were not included when counting the number of
words having three or more syllables. Appendix D describes the
procedures used to assess readability using a modified Gunning Fog
formula.

***B.3.5. Inter-rater Reliability.*** After the evaluation forms were
approved by all panelists, the 16 panelists were divided into four
sub-groups based on their expertise and experience with the four study
drugs. Each sub-group was given responsibility for evaluating the
information sheets for their assigned drug. Before rating the
information sheets, inter-rater reliability was tested using 48 patient
information sheets. Researchers assigned and mailed a sub-set of
information sheets to various panelist pairs in each sub-group.
Panelists independently evaluated their assigned materials and mailed
rating forms back to researchers. A computer was used to calculate the
percentage of points obtained for each general criterion and the
percentage of points obtained overall. Inter-rater reliability was
assessed using Pearson correlation coefficients. Pearson r for overall
adherence was 0.93 for atenolol raters, 0.90 for atorvastatin raters,
0.97 for glyburide raters, 0.87 for nitrogylcerin raters, and 0.90 for
all raters combined. Pearson r was 0.42 for Criterion 7 and ranged from
0.62 to 0.96 for remaining criteria. Further analysis of Criterion 7
showed exact agreement among panelists in 73% of the cases; however,
there was some confusion about whether this section should be completed
for cases where no benefit-risk information was provided. This problem
was clarified through further discussion with panelists.

After establishing acceptable inter-rater reliability, researchers
sorted all information sheets by drug and mailed one-fourth of the
sheets to each panelist in the sub-group responsible for reviewing that
drug. Expert panelists rated a total of 1,367 pharmacy-generated patient
information leaflets and 31 "patient information booklets" published and
included in nitroglycerin boxes by drug manufacturers. This means that
each expert panelist rated an average of 87 informational items. After
reviewing their assigned materials, panelists returned the rating forms
to researchers who submitted all forms to the University of Wisconsin
Survey Laboratory for data entry.

**B.4 Consumer Evaluation Procedures **

***B.4.1. Consumer Evaluation Form and Scoring Procedures.*
**Researchers developed and pre-tested a 1-page Consumer Evaluation Form
(CEF) for measuring the perceived comprehensibility, legibility, and
usefulness of patient information sheets from a consumer's perspective.
(A copy is provided in Appendix E). The Consumer Evaluation Form
incorporated criteria suggested by the 1996 Action^1^ and several items
that were validated in a previous study by Krass, Svarstad, and
Bultman^4^. The first section included nine items believed to facilitate
legibility and comprehension by consumers. For each item, consumers were
asked to: "Please **[circle one number]{.underline}** that best
describes how **[YOU]{.underline}** would feel if you were taking this
medicine for the first time and received this information sheet from the
pharmacy". Each item was given with a semantic differential scale scored
from "1\" (poor) to "5\" (good). The relevant items included: poor -
good print size, poor - good print quality, poor - good spacing between
lines, poorly organized - well organized, poor - good length, unclear -
clear, unhelpful - helpful, incomplete - complete, and hard - easy to
find important information.

The second section of the Consumer Evaluation Form asked for an overall
assessment of three variables: readability, comprehensibility, and
usefulness: "**[Overall]{.underline}**, what is your opinion about this
information sheet. Please **[circle]{.underline}** one number that best
describes how **[you]{.underline}** would feel if you received this
information sheet". Each item was given with a semantic differential
scale scored from "1\" (poor) to "5\" (good): hard - easy to read,
hard - easy to understand, and not useful - useful. Item scores were
summed for the first section (range=9-45), second section (range=3-15),
and combined sections (range=12-60). Five levels of adherence were
defined:

• abLevel 1: information with 0-19% of relevant points

• abLevel 2: information with 20-39% of relevant points

• abLevel 3: information with 40-59% of relevant points

• abLevel 4: information with 60-79% of relevant points

• abLevel 5: information with 80-100% of relevant points

***B.4.2. Test - retest Reliability.*** Researchers analyzed the
test-retest reliability of the Consumer Evaluation Form using 18 patient
information sheets and nine consumer raters who were asked to
independently evaluate the same information sheets at two group sessions
held at least three days apart. Pearson r was 0.81 for the first
section, 0.78 for the second section, and 0.82 for the combined
sections. These results suggest that the Consumer Evaluation Form had
good test - retest reliability.

***B.4.3. Recruitment of Consumer Panelists.*** After establishing
reliability, researchers identified 14 consumer facilitators with the
help of pharmacy faculty in different geographic areas. Each consumer
facilitator was asked to recruit 8-15 consumer panelists willing to
attend a 2-hour group session where each panelist would be asked to read
and evaluate a sample of patient information sheets using the 1-page
Consumer Evaluation Form. Facilitators were provided a packet of
materials that included a sample announcement and sign-up sheet,
guidelines for selecting consumer panelists, guidelines for leading the
group sessions, and one packet of materials for each consumer panelist.
Facilitators used a variety of techniques for recruiting consumers,
including: contacting employees or members of various organizations and
social groups, posting announcements and sign-up sheets in clinics,
senior centers, or apartment buildings, and other "snowball" techniques.

Facilitators were encouraged to recruit a diverse group of male and
female consumer panelists who: were able to read and understand English;
had no training as a doctor, nurse, or pharmacist; did not reside in a
nursing home or other institution; and had diverse race or ethnic
backgrounds. Since the study drugs generally are prescribed for older
adults, recruitment guidelines suggested that a majority of consumer
panelists should be 50 years or older. Panelists were limited to English
speakers, because it was not feasible to gather information sheets
printed in other languages.

Facilitators recruited 154 consumer panelists from 11 states. Panelist
age ranged from 20 to 89 years old (mean=61.4, sd=16.7), with
three-fourths of all panelists being 52 years old or older. Two-thirds
(68%) of the panelists were female, 89% were white, and 97% reported
English as their native language. Approximately 8% had not completed
high school, 38% had completed high school, and 54% had more than a high
school education. Three-fourths of the panelists (77%) typically used
one or more prescribed medications each day (range=0-16 drugs, mean=3.0,
sd=3.3). If a consumer panelist currently used a study drug, he/she did
not evaluate patient information sheets for that drug.

***B.4.4. Consumer Group Sessions. ***Each consumer panelist attended
one small group session led by a consumer facilitator. The purpose of
the small group session was to evaluate a sample of patient information
sheets using Consumer Evaluation Forms. The facilitator began the
session by providing a brief background of the study, gathering
demographic information from each panelist, distributing one packet of
materials to each panelist, and answering any questions. Each panelist
was asked to read and evaluate independently 10-12 patient information
sheets during the session. They were asked to rate one information sheet
at a time, to place each information sheet and evaluation form into
their envelope after finishing it, and to avoid talking with other
panelists until everyone had completed their evaluations. Each consumer
panelist was given an honorarium of \$50 for his/her efforts.

**B. 5. Data Entry and Processing**

All expert and consumer evaluation forms were submitted to the
University of Wisconsin Survey Laboratory for data entry and merging
into an electronic data file. Researchers checked the data file for
missing or inaccurate entries and created frequency tables using SPSS.

**C. RESULTS**

**C.1. Frequency of written information transmission**

Patient-observers presented new prescriptions in 384 community
pharmacies located in 44 states. The overwhelming majority of these
prescriptions were dispensed with computer-generated "patient
information leaflets" beyond container labels. The percent of
patient-observers who were given a patient information leaflet was 89.6%
for atenolol, 88.8% for glyburide, 89.3% for atorvastatin, and 88.3% for
nitrogylcerin (Table 2). A total of 1,367 patient information leaflets
were rated by expert panelists. Of these, 62.1% were at least 5.6 inches
long and 37.9% were 5.5 inches long or less. Nearly 53.6% of the
leaflets were based on information from a single publisher, 45.6% were
based on information from unknown publishers, and less than 1% were
based on information from other publishers. In addition to these
pharmacy-generated patient information leaflets, 31 patients received a
pre-printed "patient information booklet" included in the package by the
nitroglycerin manufacturer. These manufacturers' booklets will be
evaluated and described later.

**C.2. Overall quality of patient information leaflets according to
expert panelists**

Table 3 presents the distribution of expert panelists' ratings of
patient information leaflets by drug, including cases where no written
information was given. In the first row, we see the distribution of
ratings for atenolol information. Only 20.1% of the patient-observers
received a leaflet that met the Level 4 threshold and none received
leaflets that met the Level 5 threshold. Approximately 56.5% of the
patient-observers received a leaflet that met Level 3 threshold (40-59%
of points). The average percentage of points for all atenolol
prescriptions was 51.4.

Expert panelists' ratings of leaflets for glyburide and atorvastatin
were similar, with only 24.5% and 16.9% of patient-observers receiving a
leaflet that met the Level 4 threshold for glyburide and atorvastatin,
respectively. None received a glyburide or atorvastatin leaflet that met
the Level 5 threshold. For both drugs, the average percentage of points
received was approximately 51%. The ratings of nitroglycerin leaflets
were somewhat higher, with 41.7% receiving information that met Level 4
threshold. Again, none of the patient-observers received a nitroglycerin
leaflet that met Level 5 threshold. The average percentage of points
received was 55.3%.

**C.3. Expert panelists' ratings of patient information leaflets by
criterion **

Table 4 shows the distribution of expert panelists' ratings of patient
information leaflets by criterion (excluding cases in which no leaflet
was given). Level of adherence to criteria varied across the eight
criteria, with the highest means obtained on Criterion 7 (scientific
accuracy, unbiased, up-to-date). The mean percentage of points on
Criterion 7 ranged from 90-99% for the four study drugs, as seen in the
final column for Criterion 7. In contrast, the mean percentage of points
ranged from 38-66% for Criterion 1 (drug name and indications for use),
33-51% for Criterion 2 (contraindications and what to do if applicable),
54-75% for Criterion 3 (specific directions about how to use, monitor,
and get maximum benefits), 33-45% for Criterion 4 (specific precautions
and how to avoid harm while using the medication), 40-53% for Criterion
5 (symptoms of serious or frequent adverse reactions and what to do),
36-38% for Criterion 6 (general information and encouragement to ask
questions), and 44-56% for Criterion 8 (legibility and
comprehensibility).

The distribution of ratings showed that the expert ratings generally
fell below the Level 4 threshold on five of eight general criteria,
including Criterion 2 (contraindications), Criterion 4 (precautions),
Criterion 5 (adverse reactions), Criterion 6 (general information), and
Criterion 8 (legibility and comprehensibility). While there were some
variation among study drugs for Criteria 1-5, there were few drug
differences for Criteria 6-8. The most striking pattern was the low
level of adherence overall.

**C.4. Expert panelists' ratings of patient information leaflets by
sub-criterion **

Tables 5-8 show the percentage of pharmacy-generated patient information
leaflets with full or partial adherence on each sub-criterion. The
percentage of leaflets with "no adherence" is omitted from these tables
to simplify data presentation. The tables indicate more clearly why
patient information leaflets did not fully meet various criteria used by
expert panalists.

Criterion 1 requires inclusion of generic and brand names, phonetic
spelling of generic names, and other information about the drug and its
indications for use. Tables 5-6 show that most atenolol and glyburide
leaflets included the generic name and a common indication for use.
However, nearly one-half of the leaflets omitted the phonetic spelling
of generic names and most did not mention any brand names. Leaflets for
atorvastatin and nitroglycerin provided more information about brand
names, but also lacked phonetic spellings of generic names (Tables 7-8).
Most glyburide and atorvastatin leaflets failed to mention that these
medications should be used in persons whose conditions cannot be
controlled by other means such as proper diet.

Criterion 2 requires specific information about contraindications and
what to do if applicable. Results show that 75% or more of all leaflets
fully adhered to sub-criteria regarding pregnancy and breast-feeding.
However, a majority of leaflets failed to adhere fully to other
sub-criteria in this area. For example, many atorvastatin leaflets did
not encourage patients to tell their provider if they drink large
amounts of alcohol or have a history of liver disease (Table 7: 2.4,
2.5).

Criterion 3 requires specific directions about how to use, monitor, and
get the most benefit from the medication. Over 80% of the nitroglycerin
leaflets fully adhered to seven of the nine sub-criteria in this area
(Table 8). Atenolol, glyburide, and atorvastatin leaflets fully adhered
to only 2-3 sub-criteria related to missed doses and use with regard to
food. Lower ratings were obtained for other sub-criteria because these
leaflets did not always emphasize the importance of regular use, taking
the medicine at the same time(s) each day, and regular testing of blood
glucose or cholesterol levels (Tables 5-7).

Criterion 4 requires specific precautions and information about how to
avoid harm while using the medication. A majority of the patient
information leaflets encouraged patients to tell their providers if they
take any other medications; however, leaflets often lacked specific
information in this area. For example, only one-fourth of leaflets
mentioned the interaction between atentolol and calcium channel blockers
such as verapamil and diltiazem and only one-fourth of atentolol
leaflets mentioned the potential interaction with immunosuppressants
such cyclosporine (Sandimmune). While 80% of glyburide leaflets
mentioned sensitivity to sun, many glyburide leaflets did not mention
potential interactions with MAO inhibitors, aspirin products, or
anticoagulants. One in three nitroglycerin leaflets failed to mention
any interaction with sildenafil or Viagra.

Nearly all atenolol leaflets included a warning about stopping atenolol;
however, only 36% included a warning that was sufficiently specific and
clear according to panelists (Table 5: 4.5). Over 80% of the leaflets
advised patients to tell their provider or dentist about their use of
atenolol before surgery, but patients rarely were informed that atenolol
may cause serious reaction to allergy shots or worsen allergic reactions
to foods, medicines, or stings. Only one in four atorvastatin leaflets
included any statement about the need for liver function tests (Table 7:
4.8).

Criterion 5 requires information about the symptoms of serious or
frequent adverse reactions and what to do about them. Panelists rated
the atenolol leaflets favorably with regard to certain adverse reactions
(Table 5: 5.1, 5.5); however, they noted insufficient information
regarding other adverse reactions such as feeling depressed and trouble
having sex or sleeping (Table 5: 5.7-5.9). The quality of information
regarding adverse reactions also varied considerably for the glyburide,
atorvastatin, and nitroglycerin leaflets, as shown in Tables 5-8.

Criterion 6 requires several items of general information and
encouragement to ask questions. Table 5 illustrates why
pharmacy-generated patient information leaflets often obtained low
ratings in this area. Only 48-57% of atenolol leaflets included a
statement about keeping medicines away from children and not giving this
medicine to others (6-1-6.2), a disclaimer statement that the leaflet
does not include all uses or effects (6.3), publisher name and date of
publication (6.4-6.5), and encouragement to ask questions (6.6).
Atenolol leaflets rarely informed patients about the availability of
longer leaflets written for professionals (6.7). Similar problems were
found with glyburide, atorvastatin, and nitroglycerin leaflets.

Criterion 7 requires that information be scientifically accurate,
unbiased, and up-to-date. The overwhelming majority of
pharmacy-generated leaflets adhered fully to all sub-criteria in this
area. The exception was that some atenolol leaflets did not adhere fully
to sub-criterion 7.2 (no unapproved uses are listed).

Criterion 8 requires that information be readily comprehensible and
legible. Results indicated several reasons for low ratings in this area.
For example, only 19% of atenolol leaflets adhered fully to
sub-criterion 8.1 (black box warning information printed in bold-face
type or box). Most leaflets received favorable ratings due to minimal
use of italics or ornate typefaces (8.2) and use of upper and lower case
lettering (8.3). However, they did not adhere fully to other principles
for enhancing legibility and comprehensibility. Headings generally were
not placed on separate lines (8.4). Bullets were not used to enhance
readability (8.5). Information was not always well organized and easy to
find (8.6). Spacing between lines was inadequate (8.7). Print size often
was smaller than 10-point type (8.8). Some leaflets also lacked good
ink-paper contrast (8.9). Finally, atenolol, glyburide, and atorvastatin
leaflets often exceeded the recommended 8^th^ grade level of reading
difficulty. According to the Gunning Fog test, mean grade levels were
9.6 for atenolol, 8.7 for glyburide, 9.4 for atorvastatin, and 7.6 for
nitroglycerin leaflets.

**C.5. Consumer panelists' ratings of patient information leaflets **

Table 9 summarizes the ratings of patient information leaflets with all
criteria rated by a consumer panelist. Of 318 atenolol leaflets rated by
consumer panelists, 24.2% met the Level 5 threshold and 30.2% met the
Level 4 threshold. Similar trends were noted for consumer ratings of
other study drugs. The percentage of leaflets meeting Level 5 threshold
was 20.4%, 27.9%, and 28.6% for glyburide, atorvastatin, and
nitroglycerin leaflets, respectively. The percentage of glyburide,
atorvastatin, and nitroglycerin leaflets meeting Level 4 threshold was
an additional 27.5%, 31.8%, and 34.1%, respectively. The mean percentage
of points obtained ranged from 57.0% (glyburide) to 64.6%
(nitroglycerin).

Table 10 summarizes consumer panelists' ratings of written information
using 12 relevant sub-criteria and 5-point semantic differential scales
(1=poor, 5=good). Several results are noteworthy. First, consumer
panelists were most critical of the leaflets' legibility. One of every
three leaflets received a low score of "1\" or "2\" on print size, print
quality, line spacing, and ease of reading. Consumer raters were
especially critical of print size, spacing, and ease of reading. For
example, print size received a score of "1\" (poor) for 23% of atenolol
leaflets, 24% of glyburide leaflets, 20% of atorvastatin leaflets, and
21% of nitroglycerin leaflets and line spacing received a score of "1\"
for 24% of atenolol leaflets, 21% of glyburide leaflets, 19% of
atorvastatin leaflets, and 17% of nitroglycerin leaflets. The mean
rating scores ranged from 2.9 to 3.1 on print size, 2.8 to 3.1 on line
spacing, 3.1 to 3.4 on print quality, and 2.9 to 3.2 on the ease of
reading overall.

Consumer raters were more positive about the leaflets' comprehensibility
and usefulness; however, none of the mean rating scores exceeded 3.9 in
these areas. For example, the mean rating scores ranged from 3.5 to 3.8
on clarity, 3.6 to 3.8 on completeness, 3.4 to 3.7 on organization, 3.3
to 3.6 on length, 3.4 to 3.6 on ease of finding important information,
and 3.3 to 3.7 on the ease of understanding overall. Results show
similar trends on helpfulness and usefulness: mean rating scores ranged
from 3.6 to 3.9 on helpfulness and 3.5 to 3.8 on usefulness overall.

Finally, it is important to note the similarity in consumer panelists'
ratings from one study drug to another. While nitroglycerin ratings were
slightly higher than those for other study drugs, all mean rating scores
ranged between 2.8 and 3.9. On the other hand, consumer panelists
appeared to be quite discriminating. Like expert panelists, the consumer
panelists rated some leaflets more favorably than others and appeared to
have little or no difficulty evaluating leaflets on these criteria.
Further analysis is required to determine whether and how expert and
consumer panelist ratings are correlated with each other and how these
ratings should be combined when determining the overall usefulness of
written prescription information.

**C.6. Expert and consumer ratings of patient information booklets
published by nitroglycerin manufacturers **

In addition to rating pharmacy-generated patient information leaflets,
31 patient-observers received a pre-printed "patient information
booklet" that was published and included in the nitroglycerin box by the
manufacturer. Table 11 shows the distribution of expert panelists'
ratings of manufacturers' booklets. Overall, manufacturer's booklets
only reached the Level 3 threshold of adherence. Booklet quality varied
by criterion, with relatively high ratings given on Criterion 3
(directions) and relatively low ratings given on Criteria 4, 5, and 6
(precautions, adverse reactions, and general information). Concerns also
were raised about booklet accuracy, bias, and timeliness: 48% of the
manufacturers' booklets received only 20-39% of the points on Criterion
7.

Table 12 shows the percent of booklets with partial or full adherence on
each sub-criterion. These data help explain why these booklets did not
receive higher ratings on certain criteria. While excellent ratings were
given on directions for use (Criterion 3), the majority of nitroglycerin
booklets did not adhere fully to most sub-criteria under Criterion 4
(precautions). In fact, none of the booklets mentioned a potential
interaction with sildenafil (Viagra). One possible reason is that many
booklets were published 3-6 years before they were given to the
patient-observers in spring 2001. Expert panelists also commented that
the booklets were not as neutral as they could have been, largely
because they appeared to minimize or down-play adverse reactions and
precautions (as reflected in panelists'ratings on Criterion 7). Expert
panelists also were gave low ratings on booklet legibility (Criterion
8). None of the booklets adhered partially or fully to sub-criteria 8.7
and 8.8 (adequate line spacing and font size).

Consumer panelists' ratings were consistent with expert panelists in
several respects. First, they gave relatively low ratings on print size,
print quality, line spacing, and overall ease of reading. Higher ratings
were given on booklet completeness, possibly because the directions for
use were perceived to be clear, complete, well organized, and easy to
understand.

**D. STUDY LIMITATIONS AND CONCLUSIONS**

It is important to note several study limitations when drawing
conclusions from this evaluation. First, data were collected from a
simple random sample of community pharmacies in 44 states. While the
national sample is a major strength of this study, we cannot generalize
to mail order pharmacies and other excluded settings. Second, pharmacies
were visited by shoppers who followed a standard protocol and scenario
involving four new prescriptions. We cannot generalize to patients who
pick up refill prescriptions. Third, the study evaluated the quantity
and quality of information published in English. Patient access to
prescription information in Spanish and other languages was not
addressed in this study. Fourth, we obtained good test-retest
reliability using the Consumer Evaluation Form and obtained useful
information from a diverse group of 154 consumer panelists. It is not
known whether similar results would be obtained using a random sample of
consumers and other methodologies for assessing the consumer's
perspective. Fifth, this study was focused on the provision of written
prescription information. Further analysis is needed to determine
whether the written information was provided alone or in combination
with oral information and how these different methods of presentation
affect consumers' evaluation of and behavioral response to written
prescription information. This is an important issue, because consumers
may be more likely to read and use written prescription information if
they also receive oral counseling and encouragement to read this
information.

Despite these limitations, this study yielded several important
findings. First, it is clear that most community pharmacies now provide
a computer-generated patient information leaflet with every new
prescription. However, leaflet length varies considerably. Approximately
38% of the leaflets were relatively short or abbreviated (less than 5.6
inches in length) and 62% were standard length (5.6 inches long or
longer). Why some pharmacies are more likely than others to provide
relatively short or abbreviated leaflets is not known at this time.

Second, the findings show that leaflet quality varies widely. Expert
panelists evaluated 1,367 pharmacy-generated patient information
leaflets and found relatively high adherence on Criterion 7 (accuracy,
unbiased, up-to-date). In contrast, expert ratings generally fell below
the Level 4 threshold on five of eight remaining criteria. Ratings were
especially low on criteria dealing with the risks of drug treatment and
general information (Criteria 2, 4, 6). Consumer panelists also noted
variability in leaflet quality and were especially critical of print
size, print quality, line spacing, and overall ease of reading.

Third, panelists found a number of problems with the patient information
booklets published by nitroglycerin manufacturers. These booklets
received high ratings on directions for use. However, expert raters
expressed concern about the lack of sufficient, accurate, and up-to-date
information on adverse reactions and precautions. They noted that the
majority of manufacturers' booklets were not fully neutral or unbiased
in content and tone. In sharp contrast, most pharmacy-generated leaflets
were considered neutral in content and tone.

**E. REFERENCES **

1\. Steering Committee for the Collaborative Development of a Long-Range
Action Plan for the Provision of Useful Prescription Medicine
Information, [Action Plan for the Provision of Useful Prescription
Medicine Information]{.underline}, Unpublished report submitted to The
Honorable Donne E. Shalala, Secretary of the U. S. Department of Health
and Human Services, December 1996.

2\. Gunning R. The Technique of Clear Writing (Rev Ed). New York:
McGraw-Hill, 1968.

3\. Buck, ML. Proving patients with written medication information. The
Annals of Pharmacotherapy. 1998; 32:962-969.

4\. Krass I, Svarstad BL, Bultman D. Using alternative methodologies for
evaluating patient medication leaflets. Patient Education and Counseling
(in press)