![](media/image1.png){width="1.25in" height="1.0416666666666667in"}

***Draft Criteria for Assessing ISO Research Topic Priorities***

***Revised February 21, 2009***

***Background***

On February 20-21, 2009, The Keystone Center convened a small writing
group to consider key issues and prepare written materials for review by
a larger stakeholder group on March 16, 2009. The writing group included
individuals and members of groups whose primary interests are focused on
vaccine-related issues, state and local public health and immunization
officials, pediatricians, federal agency officials, and members of the
National Vaccine Advisory Committee (NVAC).

This document reflects the writing group's views about the
identification of draft criteria and weighting for the purpose of
prioritizing the Immunization Safety Office's Scientific Agenda. The
draft agenda can be accessed at:
<http://www.cdc.gov/vaccinesafety/agenda.htm>. The group agreed that a
prioritization system using objective criteria would be most useful for
NVAC to decide which research projects should be conducted first. Over
two days of discussions, the group reached consensus on a broad set of
criteria, additional issues to consider and contextual comments for the
criteria, and an initial approach to weighting the criteria. The writing
group also identified several general principles that they found to be
important in setting the stage for how they hope the criteria will be
applied.

*Note: This draft summary will be reviewed in detail at the March 16
meeting.  To the extent that interested stakeholders have questions or
would like additional information regarding this draft, please contact
Heather Bergman of The Keystone Center at 303-531-5511 or
hbergman@keystone.org. *

***General Principles***

-   In order to limit the need to prioritize scientific research,
    resources for vaccine safety studies should be increased as much as
    possible.

-   Resources should be allocated to achieve maximum impact.

-   Criteria should be applied in a transparent process with stakeholder
    input throughout the process to enhance accountability and enhance
    public trust and confidence.

-   The following criteria are proposed so that prioritization decisions
    are made in a consistent and fair fashion.

-   In order to achieve accountability, NVAC will provide an explanation
    of how they applied the criteria to the issues on the research
    agenda.

-   It is understood that ISO will be agile and responsive to scientific
    and policy questions and issues that emerge within the 5-year
    scientific agenda window.

-   After each research study is prioritized, ISO will assess whether
    the study is within its scope. If it is not, ISO will refer the
    issue to NVPO for action.

***Writing Group Prioritization Process***

-   Writing group participants were asked to indicate the relative
    importance of the six criteria using colored dots. Each participant
    placed one dot on each of the six criteria (which were printed on
    chart paper). Green dots indicated that a criterion was of the
    highest importance; yellow dots indicated that a criterion was very
    important; and red dots indicated that a criterion was relatively
    less important but still important.

-   The criteria below have been listed from left to right in the order
    of importance that resulted from the dots exercise (based on the
    number of green dots each criterion received).

```{=html}
<!-- -->
```
-   Although all six criteria are very important, the prioritization
    exercise indicated that there is a sequence reflected in how the
    criteria should be applied. The first five criteria relate to the
    substantive question of what studies should be done. These five
    criteria should be applied to the ISO Research Agenda first, then
    the sixth criterion (which relates to whether/how a study can be
    done) should be applied.

-   ***Note:*** Federal agency officials did not participate in the
    weighting exercise.

*Note: Several of the criteria below were identified as important during
the community meetings in Birmingham, Ashland, and Indianapolis. These
are indicated with an asterisk (\*). (The criteria without asterisks
were not included in the community discussions, so community members did
not indicate whether they are important or not.)*

+------+----------+------------------------+--------------------------+
| **   |          |                        |                          |
| Spec |          |                        |                          |
| ific |          |                        |                          |
| Vac  |          |                        |                          |
| cine |          |                        |                          |
| Sa   |          |                        |                          |
| fety |          |                        |                          |
| Hy   |          |                        |                          |
| poth |          |                        |                          |
| eses |          |                        |                          |
| (    |          |                        |                          |
| Cate |          |                        |                          |
| gory |          |                        |                          |
| A)** |          |                        |                          |
+------+----------+------------------------+--------------------------+
| > ** | **Cr     | **Issues to Consider** | **Comments**             |
| Step | iteria** |                        |                          |
| >    |          |                        |                          |
| 1:** |          |                        |                          |
| >    |          |                        |                          |
| > ** |          |                        |                          |
| What |          |                        |                          |
| > to |          |                        |                          |
| >    |          |                        |                          |
| Do** |          |                        |                          |
+------+----------+------------------------+--------------------------+
|      | **Sign   | 1.  Number of people   | -   Communities          |
|      | ificance |     who receive the    |     indicated that       |
|      | of the   |     vaccine(s).\*      |     whether a vaccine is |
|      | Ex       |                        |     given to children is |
|      | posure** | 2.  Receipt of vaccine |     very important.\*    |
|      |          |     by infants or      |                          |
|      |          |     children.\*        | -   Communities          |
|      |          |                        |     indicated that       |
|      |          | 3.  Receipt of vaccine |     whether a vaccine is |
|      |          |     by other           |     mandatory is very    |
|      |          |     vulnerable         |     important.\*         |
|      |          |     populations.\*     |                          |
|      |          |                        | -   Communities          |
|      |          | 4.  The vaccine(s)     |     indicated that       |
|      |          |     is/are mandatory   |     number of people who |
|      |          |     or universally     |     received the vaccine |
|      |          |     recommended.\*     |     is important.\*      |
|      |          |                        |                          |
|      |          |                        | -   Vulnerable           |
|      |          |                        |     populations include  |
|      |          |                        |     pregnant women,      |
|      |          |                        |     chronically ill, and |
|      |          |                        |     the underserved.     |
+------+----------+------------------------+--------------------------+
|      | **Burden | 1.  Severity of the    | -   To extent possible,  |
|      | of the   |     health event       |     use a standard       |
|      | Health   |     including acute    |     framework (e.g.,     |
|      | Event**  |     and chronic        |     QALY) to             |
|      |          |     disability,        |     characterize         |
|      |          |     treatability, and  |     severity.            |
|      |          |     preventability.\*  |                          |
|      |          |                        | -   Communities          |
|      |          | 2.  Frequency of the   |     indicated that       |
|      |          |     health event.\*    |     severity is more     |
|      |          |                        |     important than       |
|      |          | 3.  Increasing         |     frequency.\*         |
|      |          |     incidence of the   |                          |
|      |          |     health event       |                          |
+------+----------+------------------------+--------------------------+
|      | **Public | 1.  Strength of public | -   May include          |
|      | Con      |     concern about a    |     frequency of the     |
|      | cern\*** |     possible link      |     concern (e.g.,       |
|      |          |     between            |     determined by a      |
|      |          |     vaccination and    |     survey) or the       |
|      |          |     the adverse health |     passion with which   |
|      |          |     event.\*           |     the concern is       |
|      |          |                        |     expressed.           |
|      |          |                        |                          |
|      |          |                        | -   Communities          |
|      |          |                        |     indicated public and |
|      |          |                        |     parental concerns    |
|      |          |                        |     are important.\*     |
+------+----------+------------------------+--------------------------+
|      | **Status | 1.  Temporal sequence  | -   Prior data may       |
|      | of       |     of the vaccine(s)  |     include information  |
|      | Existing |     and the health     |     from surveillance or |
|      | Sc       |     event              |     animal studies.      |
|      | ientific |                        |                          |
|      | Knowl    | 2.  Prior data provide | -   Communities          |
|      | edge**\* |     information that   |     indicated scientific |
|      |          |     would support the  |     concern is           |
|      |          |     hypothesis         |     important.\*         |
|      |          |                        |                          |
|      |          | 3.  A plausible        | -   These considerations |
|      |          |     biological         |     are based on the     |
|      |          |     mechanism exists   |     Bradford Hill        |
|      |          |     linking the        |     criteria; other      |
|      |          |     vaccine(s) and the |     Bradford Hill        |
|      |          |     health event.      |     criteria may be      |
|      |          |                        |     useful in assessing  |
|      |          |                        |     the existing         |
|      |          |                        |     scientific knowledge |
|      |          |                        |     as well.             |
+------+----------+------------------------+--------------------------+
|      | **Impact | A study is likely to   |                          |
|      | and      | have a significant     |                          |
|      | Public   | impact on the          |                          |
|      | Policy** | understanding of this  |                          |
|      |          | issue.                 |                          |
+------+----------+------------------------+--------------------------+
| > ** | **Feasi  | 1.  Methodological and |                          |
| Step | bility** |     ethical            |                          |
| >    |          |     feasibility        |                          |
| 2:** |          |                        |                          |
| >    |          | 2.  Cost of the study  |                          |
| > *  |          |     and impact on the  |                          |
| *How |          |     ability to do      |                          |
| > to |          |     other studies      |                          |
| > Do |          |     (including         |                          |
| >    |          |     opportunity costs) |                          |
| It** |          |                        |                          |
|      |          | 3.  Optimal sequencing |                          |
|      |          |     with other         |                          |
|      |          |     potential studies  |                          |
|      |          |     that may be done   |                          |
|      |          |     by groups other    |                          |
|      |          |     than ISO.          |                          |
+------+----------+------------------------+--------------------------+

**Prioritization of Categories B -- D**

> **[General Principles]{.underline}**

-   Categories B -- D include topic areas but not specific testable
    hypotheses that can be studied.

-   Application of criteria to Categories B -- D will lead to
    prioritization of the topics within each category.

-   Following prioritization, ISO should convene working groups of
    internal and external experts to identify specific testable
    hypotheses that can be studied. Working groups should be convened
    first for topics that rank higher.

+---+-------+----------------+-------------------+-------------------+
|   | **    | **Category     | **Category C:**   | **Category D:**   |
|   | Crite | B:**           |                   |                   |
|   | ria** |                | **Special         | **Clinical        |
|   |       | **Vaccines and | Populations**     | Outcomes**        |
|   |       | Vaccination    |                   |                   |
|   |       | Practices**    |                   |                   |
+---+-------+----------------+-------------------+-------------------+
|   | **Si  | 1.  Number of  | 1.  Number of     | 1.  The           |
|   | gnifi |     people who |     people in the |                   |
|   | cance |     receive    |     special       |  disease/syndrome |
|   | of    |     the        |     population\*  |     occurs among  |
|   | the   |     vaccine or |                   |     infants and   |
|   | Expos |                | 2.  Levels of     |     children\*    |
|   | ure** |    vaccination |     vaccine       |                   |
|   |       |     practice\* |     coverage in   | 2.  The           |
|   |       |                |     the special   |                   |
|   |       | 2.  Exposure   |     population    |  disease/syndrome |
|   |       |     of infants |                   |     occurs among  |
|   |       |     and        | 3.  Vaccine(s)    |     vulnerable    |
|   |       |     children\* |     is/are        |     populations   |
|   |       |                |     mandatory or  |                   |
|   |       | 3.  Exposure   |     universally   |                   |
|   |       |     of         |     recommended   |                   |
|   |       |     vulnerable |     for the       |                   |
|   |       |                |     special       |                   |
|   |       |    populations |     population.\* |                   |
|   |       |                |                   |                   |
|   |       | 4.  The        | 4.  The special   |                   |
|   |       |     vaccine(s) |     population is |                   |
|   |       |     is/are     |     of particular |                   |
|   |       |     mandatory  |     concern to    |                   |
|   |       |     or         |     our society   |                   |
|   |       |                |     or is         |                   |
|   |       |    universally |     believed to   |                   |
|   |       |                |     merit a       |                   |
|   |       | recommended.\* |     higher degree |                   |
|   |       |                |     of care or    |                   |
|   |       |                |     attention\*   |                   |
+---+-------+----------------+-------------------+-------------------+
|   | **B   | 1.  Severity   | NA                | 1.  Severity of   |
|   | urden |     of the     |                   |     the           |
|   | of    |     health     |                   |                   |
|   | the   |     event(s)   |                   |  disease/syndrome |
|   | H     |                |                   |     including     |
|   | ealth |   hypothesized |                   |     acute and     |
|   | Ev    |     to be      |                   |     chronic       |
|   | ent** |     associated |                   |     disability,   |
|   |       |     with this  |                   |     treatability, |
|   |       |     vaccine or |                   |     and           |
|   |       |                |                   |                   |
|   |       |    vaccination |                   |  preventability\* |
|   |       |     practice   |                   |                   |
|   |       |     in         |                   | 2.  Frequency of  |
|   |       |     including  |                   |     the disease/  |
|   |       |     acute and  |                   |     syndrome\*    |
|   |       |     chronic    |                   |                   |
|   |       |                |                   | 3.  Increasing    |
|   |       |    disability, |                   |     incidence of  |
|   |       |                |                   |     the disease/  |
|   |       |  treatability, |                   |     syndrome      |
|   |       |     and        |                   |                   |
|   |       |     pr         |                   |                   |
|   |       | eventability\* |                   |                   |
|   |       |                |                   |                   |
|   |       | 2.  Frequency  |                   |                   |
|   |       |     of the     |                   |                   |
|   |       |     health     |                   |                   |
|   |       |     event(s)\* |                   |                   |
|   |       |                |                   |                   |
|   |       | 3.  Increasing |                   |                   |
|   |       |     incidence  |                   |                   |
|   |       |     of the     |                   |                   |
|   |       |     health     |                   |                   |
|   |       |     event(s)   |                   |                   |
+---+-------+----------------+-------------------+-------------------+
|   | **P   | Strength of    | Strength of       | Strength of       |
|   | ublic | public concern | scientific        | public concern    |
|   | C     | about a        | concern that the  | that the          |
|   | oncer | possible link  | special           | disease/syndrome  |
|   | n\*** | between a      | population may    | may be associated |
|   |       | vaccine or     | experience        | with a vaccine(s) |
|   |       | vaccination    | greater risk of   | or vaccination    |
|   |       | and one or     | injury from       | practice\*        |
|   |       | more adverse   | vaccination\*     |                   |
|   |       | health         |                   |                   |
|   |       | events\*       |                   |                   |
+---+-------+----------------+-------------------+-------------------+
|   | **S   | Strength of    | Strength of       | Strength of       |
|   | tatus | scientific     | scientific        | scientific        |
|   | of    | concern about  | concern that the  | concern that the  |
|   | Exi   | a possible     | special           | disease/syndrome  |
|   | sting | link between a | population may    | may be associated |
|   | Scien | vaccine or     | experience        | with a vaccine(s) |
|   | tific | vaccination    | greater risk of   | or vaccination    |
|   | Kno   | practice and   | injury from       | practice\*        |
|   | wledg | one or more    | vaccination\*     |                   |
|   | e**\* | adverse health |                   |                   |
|   |       | events\*       |                   |                   |
+---+-------+----------------+-------------------+-------------------+
|   | **I   | 1.  Importance | 3.  Importance of | 1.  Importance of |
|   | mpact |     of study   |     study         |     study         |
|   | and   |     studying   |     studying this |     studying this |
|   | P     |     this       |     special       |     clinical      |
|   | ublic |     vaccine or |     population to |     outcome       |
|   | Pol   |                |     an ongoing or |     process       |
|   | icy** |    vaccination |     imminent      |                   |
|   |       |     practice   |     policy        | 2.  Importance of |
|   |       |     to an      |     process       |     studying this |
|   |       |     ongoing or |                   |     clinical      |
|   |       |     imminent   | 4.  Importance of |     outcome to    |
|   |       |     policy     |     studying this |     confidence    |
|   |       |     process    |     special       |     and trust in  |
|   |       |                |     population to |     the vaccine   |
|   |       | 2.  Importance |     confidence    |     program       |
|   |       |     of         |     and trust in  |                   |
|   |       |     studying   |     the vaccine   |                   |
|   |       |     this       |     program       |                   |
|   |       |     vaccine or |                   |                   |
|   |       |                |                   |                   |
|   |       |    vaccination |                   |                   |
|   |       |     practice   |                   |                   |
|   |       |     to         |                   |                   |
|   |       |     confidence |                   |                   |
|   |       |     and trust  |                   |                   |
|   |       |     in the     |                   |                   |
|   |       |     vaccine    |                   |                   |
|   |       |     program    |                   |                   |
+---+-------+----------------+-------------------+-------------------+
| > | **Fea | NA until       |                   |                   |
|   | sibil | development of |                   |                   |
| * | ity** | a specific     |                   |                   |
| * |       | hypothesis or  |                   |                   |
| S |       | hypotheses     |                   |                   |
| t |       |                |                   |                   |
| e |       |                |                   |                   |
| p |       |                |                   |                   |
| > |       |                |                   |                   |
|   |       |                |                   |                   |
| 2 |       |                |                   |                   |
| : |       |                |                   |                   |
| * |       |                |                   |                   |
| * |       |                |                   |                   |
| > |       |                |                   |                   |
| > |       |                |                   |                   |
|   |       |                |                   |                   |
| * |       |                |                   |                   |
| * |       |                |                   |                   |
| H |       |                |                   |                   |
| o |       |                |                   |                   |
| w |       |                |                   |                   |
| > |       |                |                   |                   |
|   |       |                |                   |                   |
| t |       |                |                   |                   |
| o |       |                |                   |                   |
| > |       |                |                   |                   |
|   |       |                |                   |                   |
| D |       |                |                   |                   |
| o |       |                |                   |                   |
| > |       |                |                   |                   |
|   |       |                |                   |                   |
| I |       |                |                   |                   |
| t |       |                |                   |                   |
| * |       |                |                   |                   |
| * |       |                |                   |                   |
+---+-------+----------------+-------------------+-------------------+