Ruth S. Day

Duke University

4/27/00

**[WRITTEN PRESCRIPTION INFORMATION]{.underline}**

**Current Data and Future Studies**

# OVERALL APPROACH OF THE STUDY

The overall approach of the Svarstad and Bultman (1999) study is very
good, especially in its geographic breadth, random sampling of
pharmacies, training of patient-observers, and modifications of the
rating form to fit each drug. Analyses concerning the percentage of
patients who received leaflets are interesting and complete. Analyses of
expert panel ratings do a good initial job of portraying the findings,
given the available time and amount of data collected. However,
additional analyses are needed, both to help interpret the reported
findings and to provide other types of information. Since the authors
presented additional analyses at the CDER Workshop in February, 2000 and
are evidently continuing to work on their data, a complete list of
needed analyses is not presented here. Instead, just a few especially
interesting ones are discussed.

Comments are presented below on five general topics: 1) The 10 Criteria;
2) Characterization of Leaflets; 3) Consumer Comprehension; 4)
Comprehension vs. Cognitive Accessibility; and

5\) Recommendations. These comments are not exhaustive, but are intended
to highlight some important issues concerning interpretation of the
current study and implications for future studies.

# 

# THE 10 CRITERIA 

**[Overall Results.]{.underline}**

The 10 general criteria selected to evaluate leaflet quality are good
reflections of the 1996 Action Plan concerns. Overall ratings for each
criterion are presented in Table 3 of the Svarstad and Bultman report,
separately for each drug. While the table is helpful, it is difficult to
get an overall view of how well the criteria fared. It would help to
average over the three drugs, then present the criteria in rank order,
from best to worst, as shown here:

[Criterion \#]{.underline} [Criterion Topic]{.underline} [Mean
Rating]{.underline}

\# 9 Scientific Accuracy 8.1

\# 7 Unbiased 8.0

\# 8 \*\* Legible/Comprehensible 7.3

\# 5 Adverse Reactions 7.2

\# 1 Identify/Benefits 6.9

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

#10 Up-to-Date 6.1

\# 2 Directions 5.6

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

\# 6 Storage/General Information 3.7

\# 3 Contraindications 3.6

\# 4 Precautions 3.6

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

\*\* *Note: this high rating is misleading, as discussed below*

This representation of the data shows that the criteria fell roughly
into three groups. The top group fared very well, with mean ratings in
the 7-9 range, defined by the authors in Appendix C as "100% of boxes
\[i.e., subcriteria\] checked, partial or full \[credit\]." This is a
very positive finding, since half of the criteria fell into this group.
However, the high rating for Criterion #8 (Legible/Comprehensible) is
misleading, as discussed below (see section on Consumer Comprehension).

The middle group achieved intermediate ratings, defined as "50-99% of
boxes checked, partial or full." Fortunately, it is relatively easy to
improve leaflets on Criterion #10 (Up-To-Date). For example, to improve
adherence to Criterion #10a (the letter "a" designates the first
subcriterion), vendors can just include their name and publication date
on all leaflets. Criterion #2 (Directions) is harder to improve, since
it involves different directions for each drug.

The bottom group of criteria received unacceptably low ratings. Vendors
should be alerted to these problems soon, and perhaps guidelines for
improving them should be developed. Some of the subcriteria involved
will be relatively easy to fix (e.g., Criterion 6c \-- include a "boiler
plate" disclaimer on all leaflets that the medicine is intended only for
this patient), while others will be more difficult (e.g., Criterion 4a
-- potential drug interactions), since they are different for each drug.

Additional analysis and interpretation of all subcriteria will help
elucidate the nature of the obtained ratings for the general criteria,
identify those, which are easy to fix, and suggest strategies for those
that are more difficult to resolve. However, there are some potential
problems with the specific subcriteria used, as discussed next.

**[Subcriteria.]{.underline}**

The use of just 10 general criteria is efficient and the use of
subcriteria to "operationalize" each also makes sense. However there are
two potential problems \-- how raters used this system and the nature of
the subcriteria used for each general criterion.

[Raters Task.]{.underline} The raters' first task was to "check whether
there was partial or full adherence" to each of 2-5 subcriteria within a
general criterion. They made a 3-choice decision for each subcriterion
-- an item could be "addressed fully," "partially," or not at all.
Raters may have varied in the *degree* of adherence they used to decide
what constituted full or partial adherence. Were they given guidelines
about this? Future studies might give raters explicit guidelines for
these adherence levels, then award partial credit (e.g., ½ point) for
partial adherence. At present, the scoring method lumps all partial- and
full-credit items together, thereby inflating adherence ratings overall
and making it impossible to distinguish basically full-credit items from
partial-credit items.

[Nature of the Subcriteria.]{.underline} It is difficult to generate
appropriate subcriteria for each general criterion, and inevitably some
will fit better than others. However there are many cases where a set of
subcriteria reflect quite different types of information, so caution
must be used in interpreting results for the general criterion. For
example, Criterion #8 (Legibility/Comprehensibility) contains the
following subcriteria: 8a) three components of linguistic simplicity;
8b) four components of spatial layout; 8c) four components of visual
legibility. Although the components within each subcriterion are
internally consistent, the three subcriteria are quite different and
should not be lumped together. Furthermore, none of the subcriteria
directly address "comprehensibility." To assess this crucial concept,
laboratory tests with consumers are needed (see below). Therefore
calling the general criterion Legibility/Comprehensibility is
misleading. Future studies should reconsider the nature of all
subcriteria, and keep/delete/add/modify/relocate them as appropriate.
They should also reconsider the names of the general criteria, to make
sure they communicate in a valid and clear way.

Sometimes the components of a given subcriterion are related, but could
still cause problems for raters. For example, results for Criterion #10a
show that only about half of the leaflets "include publisher name and
date of publication;" does that mean that 50% provided both pieces of
information, or 100% provided publisher name and 0% publication date, or
something else? Since raters had to consider such individual pieces of
information anyway, why not have raters evaluate each piece separately
to begin with? The results could still be analyzed as in current report.
They could also be subjected to some global analyses of all subcriteria
to determine which ones cluster together and perhaps form "factors" that
define a somewhat different set of general criteria. As with all
methods, there are advantages and disadvantages to this alternative
approach. Future studies should at least reconsider the nature of the
initial decisions that raters make, especially the size of the
information units they evaluate.

**[CHARACTERIZATION OF LEAFLETS]{.underline}**

More information is needed about the types of leaflets collected. About
how many "general types" are there? What is the relative proportion of
leaflets for each type? Without this information, it is very difficult
to interpret the results aggregated across all leaflets for a given
drug. A given adherence rate could reflect considerable consistency over
many types of leaflets, or a heavy weighting of just 1-2 types of
leaflets, and so forth. Leaflet "types" could be defined by vendor and
by overall similarity of features, as described next.

**[Vendors]{.underline}**

Evidently it is difficult/impossible to identify vendor information on
many of the leaflets. However results for Criterion #10a show that
approximately 50% of the leaflets (across all three drugs) have the
"publisher name and publication date." Therefore data from this sizeable
subset of leaflets can be analyzed further. The overall results can be
compared with those from the vendor-unknown subset; if the results are
comparable, then concerns about vendor variability will be lessened.
Data from each vendor can also be averaged separately, to determine the
range of adherence they achieve. These scores can also be weighted
according to the number of leaflets from each vendor, then combined to
form a weighted measure of adherence. The weighted and unweighted
measures provide different information about adherence to the various
criteria, and both would be useful to know.

**[Leaflet Similarity]{.underline}**

More important than vendor identification is the issue of similarity
across leaflets. Some leaflets must be highly similar, in content and/or
format. Clusters of highly similar leaflets (in terms of any of the
criteria or subcriteria) can be obtained using a variety of methods,
such as an object sorting task. How many clusters of similar leaflets
are there, how well does each cluster adhere to the various criteria,
and what is the relative proportion of leaflets in each cluster? The
rating results can be reanalyzed in terms of these similarity
assessments, to better characterize the nature of the results. Again,
both weighted and unweighted measures should prove useful.

## 

## CONSUMER COMPREHENSION 

**[Experts vs. Consumers.]{.underline}**

The experts in this study were well qualified to evaluate most of the
leaflet criteria, especially those concerning scientific accuracy and
the *presence* of various types of information. However consumers are
essential to evaluate the extent to which the information is
"comprehensible" -- for *they* are the intended recipients of the
leaflets. Research in cognitive psychology shows that prior knowledge of
a content domain (such as drugs) makes it easier to understand new
information in that domain. Therefore the expert panelists probably
found the leaflets more "comprehensible" than consumers would, which
means that the high rating for Criterion #8 (Legible/Comprehensible) is
very misleading. Furthermore, the inter-rater reliability was only *r* =
.49 for this criterion -- the lowest of all ten criteria. Even these
health care experts could not agree concerning what constitutes
comprehensibility for their own domain of expertise. Perhaps experts in
cognitive psychology and/or psycholinguistics might agree more.
Nevertheless, consumers are mandatory for assessing this crucial
criterion.

## Comprehension vs. Metacomprehension. 

## Comprehension can be tested in various ways, such as asking questions that require people to draw inferences from information presented to them. However, asking them only to judge "how well" they understand something is called metacomprehension, and is a very different measure. Empirical research demonstrates repeatedly that there is often a sizeable gap between metacomprehension and comprehension -- people often think they understand more than they actually do. The only way to find out what and how much they actually understand is to test comprehension directly, preferably in a laboratory situation. My own research (e.g., Day, 1999) shows that consumers dramatically overestimate their comprehension of drug leaflets, compared to their actual comprehension scores in laboratory experiments.

**[Consumer Testing Methods.]{.underline}**

Given that consumers should be included in future studies, what role
should they play? Various methodological approaches could be used, and
each has its strengths and drawbacks.

[1) Consumer Panel.]{.underline} It would be easy to re-run the current
study using a panel of consumers, perhaps with minor changes in the
leaflet criteria used, (e.g., eliminate Criterion #9a, on "consistency
with FDA labeling"). It would permit a direct comparison of experts and
consumers. However, this approach only tests participants' ability to
find requested information in the leaflet and to give *opinions* about
how well they understand it; therefore it examines their
metacomprehension, [not]{.underline} how well they actually understand
it. Also, because of the amount and demanding nature of the work
required, only a relatively small set of consumers would participate,
and they probably would not be sufficiently representative of consumers
at large.

[2) Focus Groups.]{.underline} Focus Groups could ask consumers a
variety of questions and get open-ended replies that might be missed
with other methods. Follow-up on unexpected points could be pursued
further. However, it would be difficult to quantify the results of these
discussions. Again, Focus Groups assess metacomprehension, not
comprehension.

[3) Survey.]{.underline} Various types of surveys could be used, by
telephone or mail, and reach a large number of representative consumers.
A well-designed survey would make quantification of the data quite easy.
While most conducive to testing metacomprehension, some types of
comprehension questions could also be asked. However, there are too many
uncontrolled variables in this situation that could bias the outcomes --
e.g., how long consumers spent reading the leaflet, the presence of
distracting situations during the testing period, etc.

[4) Laboratory Experiments.]{.underline} Carefully designed experiments
in a laboratory setting enable us to control confounding variables, use
a variety of cognitive tasks (e.g., search, memory, comprehension,
problem solving), test both cognition and metacognition, and use both
closed- and open-ended response methods. A modest number of consumers
can be tested to achieve appropriate statistical power, yet still be
representative of the general population. Although experiments require a
laboratory setting and expertise, they provide the widest variety of
information in the most controlled situation.

The advantages and disadvantages of all these methods are summarized
below:

###  [METHOD]{.underline} [ADVANTAGES]{.underline} [DRAWBACKS]{.underline}

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

Panel Ratings \--reuse current methods \--tests metacomprehension only

> \--allows direct comparison to experts \--participants = very few,
> unrepresentative

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

Focus Groups \--open-ended replies \--tests metacomprehension only

\--follow-up possible \--participants = relatively few

\--difficult to quantify data

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

Survey \--easy to quantify data \--tests metacomprehension mostly

\--participants = many, representative \--uncontrollable test variables

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

Laboratory \--test both cognition & metacognition \--requires laboratory
setting & expertise

Experiment \--control confounding variables

\--use multiple cognitive tasks

\--use closed- and open-ended responses

\--participants = modest number, representative

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

# 

# COMPREHENSION vs. COGNITIVE ACCESSIBILITY

# Comprehension vs. Other Cognitive Processes 

A drug information leaflet could contain all required information, but
still not guarantee that consumers will understand it. It could meet all
regulatory and legal goals, yet still fail in its main purpose.
Therefore consumer comprehension is at the core of leaflet evaluation,
as argued above. However "comprehension" (being able to understand
something) is not the only important cognitive process that leaflets
should facilitate. For example, consumers should be able to [search and
find]{.underline} information when they need it, [remember]{.underline}
some of it (at least what types of information are provided), and be
able to [use]{.underline} it accurately.

# Cognitive Experiments

Research in the Cognition Laboratory at Duke University examines how
people acquire, store, retrieve, and use drug information. To do so, we
use a variety of cognitive tasks, including perception, attention,
memory, comprehension, and problem solving. We also study the nature of
the information itself and how it can be displayed to enhance memory,
comprehension, and problem solving. For example, in a recent study (Day,
1999) funded by the U.S. Pharmacopeia, we found that patients could not
recall many of the side effects from their leaflets, but when asked
whether specific side effects could occur, they were moderately
accurate. Nevertheless, in a problem solving task, they could not
determine what to do if they experienced specific side effects -- e.g.,
whether they should seek immediate medical treatment or not. In
subsequent experiments, we presented the same side effect information in
a new display format and found a dramatic increase in performance -- the
same patients showed an 83% improvement in knowing what action to take.

# Cognitive Accessibility

We have developed guidelines for evaluating and increasing the
"cognitive accessiblity" of medical information, not only for drugs, but
also for medical devices (currently funded by CDRH at FDA). Cognitive
accessibility is the ease with which people can acquire, store,
retrieve, and use medical information in a variety of cognitive and
behavioral tasks, such as search-and-find, memory, problem solving, and
everyday action. It is based on well-known phenomena and principles in
cognitive psychology as well as empirical results from our lab. It would
be very interesting to evaluate the leaflets in the current study in
terms of their cognitive accessibility. Also, if this study is expanded
in the future (e.g., to examine more states, pharmacies, drugs, etc.),
it would be very useful to conduct subsequent laboratory experiments to
test consumer cognition of the leaflets obtained.

#### RECOMMENDATIONS

Based on the comments above, the following recommendations are offered.
Again, they are not exhaustive, but are intended as a partial summary of
the discussion here plus suggestions for working with the current
dataset, improving leaflets, and conducting future studies.

1)  [Overview of the 10 criteria]{.underline}

> \--Rank order criterion ratings from most to least adherence, averaged
> across all drugs.
>
> \--Determine cutoffs for acceptable adherence.
>
> \--For criteria with unacceptable ratings, develop guidelines for
> improving them and alert vendors to
>
> the need for change.

2)  [Criterion #8 (Legible/Comprehensible)]{.underline}

\--Do not accept the high rating; consider it only as experts'
*opinions*, biased by prior knowledge.

\--Test consumers to evaluate comprehensibility.

\--Do not lump legibility and comprehensibility together.

3)  [Subcritera]{.underline}

> \--Identify subcriteria with unacceptable adherence levels.
>
> \--Identify those that are easy vs. hard to fix.
>
> \--Develop strategies for improving adherence.
>
> \--Determine the extent to which subcriteria are consistent with
> respect to each other, their

own components, and the general criterion they are designed to
operationalize.

> \--For each general criterion, reconsider which subcriteria to
> keep/delete/add/modify/relocate in future

studies.

\--Give raters more guidelines concerning adherence levels for full- vs.
partial-credit items.

\--To score ratings, give partial credit (e.g., ½ point) for partial
credit items -- instead of lumping them

together with full-credit items.

4)  [Criterion Names]{.underline}

> \--Reconsider criterion names. Are they valid with respect to their
> subcriteria?

5)  [Initial Ratings]{.underline}

> \--Continue to have raters consider subcriteria first.
>
> \--However, rate *one* piece of information at a time (without lumping
> several together).

6)  [Leaflet Vendor]{.underline}

> \--Perform separate analyses on leaflets with known vendor
> (approximately 50% of those obtained).
>
> \--Compare results to vendor-unknown leaflets.

7)  [Leaflet Similarity]{.underline}

> \--Determine clusters of similar leaflets in terms of content and/or
> format.
>
> \--Reanalyze data in terms of these clusters.
>
> \--Obtain mean ratings weighted by the relative proportion of each
> type of leaflet.

8)  [Consumer Experiments]{.underline}

> \--Test comprehension as well as metacomprehension.
>
> \--Test other cognitive processes as well (e.g., search, memory,
> problem solving).
>
> \--Use closed- and open-ended questions.
>
> \--Control for confounding variables in the context of a laboratory
> setting.
>
> \--Conduct this testing on leaflets from the current study and/or
> future studies

9)  [Cognitive Accessibility]{.underline}

> \--Evaluate the cognitive accessibility of leaflets.
>
> \--Reanalyze ratings data in terms of cognitive accessibility scores.
>
> \--Conduct consumer experiments on leaflets with different cognitive
> accessibility scores.

10) [Multiple Approaches]{.underline}

> \--All research approaches have advantages and limitations, and
> provide different types of information.
>
> \--Therefore in future studies:
>
> \--use multiple approaches ([both]{.underline} panel ratings and
> laboratory experiments)
>
> \--include [both]{.underline} experts and consumers

**Butler, Jennie C**

**From:** Ruth S. Day \[ruthday@acpub.duke.edu\]

**Sent:** Saturday, April 29, 2000 8:24 PM

**To:** Butler, Jennie C; ruthday@acpub.duke.edu

**Subject:** Re: Docket-00N-0352

Thanks for letting me know about the problem with my attachment. I had

technical problems with a graph on the first page, but have converted it
into

a table, so it should transmit all right now. Let me know if there are
any

further problems.

Thanks,

Ruth Day

==================

\"Butler, Jennie C\" wrote:

\> Your attached comment was not complete.

\>

\> Please resubmit.

\>

\> Jennie Butler

\>

\> \> \-\-\-\--Original Message\-\-\-\--

\> \> From: Ruth S. Day \[SMTP:ruthday@acpub.duke.edu\]

\> \> Sent: Friday, April 28, 2000 3:52 PM

\> \> To: fdadockets@oc.fda.gov

\> \> Subject: Docket-00N-0352

\> \>

\> \> Dockets Management Branch:

\> \>

\> \> Attached please find my comments concerning Docket 00N-0352,
concerning

\> \> the study on \"Evaluation of Written Prescription Information
\....\" My

\> \> comments include an evaluation of the study and suggestions future
work.

\> \>

\> \> Thank you for this opportunity.

\> \>

\> \> Sincerely,

\> \> Ruth S. Day

\> \> =====================

\> \>

\> \> Professor Ruth S. Day

\> \> Psychology Department

\> \> Duke University

\> \> Box 90085

\> \> Durham, NC 27708-0085

\> \> (919) 660-5694

\> \[Butler, Jennie C\] \<\<Docket-00N-0352-RSDay.doc\>\>

\> \> \<\< File: Docket-00N-0352-RSDay.doc \>\> \<\< File: Card for Ruth
S. Day \>\>

\>

\>![](media/image1.wmf)![](media/image2.wmf)