**Science of Science Management**

**October 2-3, 2008**

**Work Group Breakout Summaries**

**Priority Questions for Breakout Discussion Groups**

A core element of the meeting was four simultaneous, closed session
breakout discussions, which align with the conceptual model of the
meeting. The four constructs are:

-   Current State of Knowledge Assessment

-   Knowledge Generation/Advancement

-   Knowledge Utilization/Dissemination/Diffusion

-   Public Health Impact

**Figure 1. Conceptual Model**

![](media/image1.png){width="6.25in" height="2.875in"}

 

Each breakout discussion focused on a single priority question in order
to create a feasible cross-disciplinary assessment model. The
discussions were led by NIH Institute Directors with the participation
of invited experts and a cross-section of key NIH staff. This
collaborative effort was aimed to result in a proposed assessment model
that can be tested. The results can provide evidence-based information
for decisionmakers to plan or to create policy. Introductions to the
four concept areas were provided, as follows. The four groups approached
their tasks in slightly different ways. Summaries of their discussions
follow.

**Current State of Knowledge Assessment Work Group**

**Participants**

-   *Lawrence Tabak, Director, NIDCR, Chair*

```{=html}
<!-- -->
```
-   David Wilson, Associate Professor, Dept of Public and International
    Affairs, George Mason University

-   Mary Kane, President, Concept Systems Incorporated

-   Adam Jaffe, Dean of Arts and Sciences and Fred C. Hecht Professor in
    Economics, Brandeis University

-   Michael Darby, Professor of Money and Financial Markets, University
    of California-Los Angeles

-   James Wong, Senior Product Strategist, Hitachi Global Storage
    Technologies

-   Lenworth Johnson, Professor of Ophthalmology and Neurology,
    University of Missouri

-   Kathie Reed, NIA, Director, Office of Planning, Analysis, and
    Evaluation

-   Alan Koretsky, NINDS, Senior Investigator, Laboratory of Functional
    and Molecular Imaging

-   Robert Star, NIDDK, Director, Division of Kidney, Urologic and
    Hematologic Diseases

-   Richard Suzman, NIA, Director, Division of Behavioral and Social
    Research

-   Christie Drew, NIEHS, Health Scientist Administrator, Program
    Analysis Branch

-   Luci Roberts, OPASI

**Charge**

The group was asked to focus on how existing knowledge and needs can be
assessed to identify possible gap areas and research opportunities.

> ***Overarching Guiding Question:*** What components should be included
> in a comprehensive framework of processes, analytic tools and methods
> that can be used to assess and prioritize the state of knowledge in a
> basic, clinical, or population-based research field to encourage
> innovation and advancement?
>
> ***Priority Question:*** How can the current state of knowledge be
> assessed to identify science opportunity for innovative research?

**Themes**

Key questions raised include:

1.  At what points in the process is research enhanced?

2.  Does funding individuals versus projects lead to better research?

3.  How do program directors make prioritization decisions?

4.  How is a decision reached that the science is ready?

5.  How is innovation sensed? What criteria can identify innovation?

6.  Is there a technological opportunity to replace what is there and is
    it appropriate?

7.  Should a database tool be created to investigate
    science-of-science-management questions?

8.  Which funding schemes produce the most innovative research and how
    is innovation measured?

The group identified four elements that drive all scientific endeavors:
people, data, decisions and decision frameworks, and scientific or
theoretical frameworks. These elements could provide an organizing
framework on which to identify gaps.

*Identifying Stars.* Who are the star scientists? They write frequently
cited papers, but are papers cited because they are "classic", methods
papers or review articles, or because they present a new area of
research which could flag opportunities? Frequently cited papers are not
always innovative; thus, paper citations might identify an area but not
a star scientist. And, "just because it's hot doesn't mean it's a
spark." Part of defining the current state of science is identifying
star scientists and the qualities that make them so. The goal is to be
better able to identify stars and recruit them at the an early stage of
their careers.

*Discerning "Mediocre" and "Bad" Science and Scientists.* The
"charlatans and crooks" must be identified because they can point to
systemic problems that need to be fixed, and they consume resources that
could be used productively. The mediocre also must be better identified.
Although there will never be a system where every researcher is 100%
productive; knowledge is cumulative and everything builds on increments
of knowledge, no matter how miniscule. The challenge to evaluation is in
knowing that a path is leading nowhere.

*Counting Citations.* One must be able to read a paper critically to
evaluate bibliometric data and assess why people might cite it. Citation
research may only reveal what is already known. Innovative research is
not always recognized by the number of times an article is cited.

*Identifying Gaps.* Literature review often reveals gaps, but not all
gaps are worth filling. The challenge is to determine the reason for the
gaps and to decide which gaps need to be filled. One gap that must be
filled is the reporting of negative research results (methodologies and
robustness).

*Disclosing Serendipity.* Most innovation occurs through serendipity;
yet scientists rarely report this phenomenon. A database could capture
this, or case studies of innovative products could be conducted (e.g.,
CAT scan preceding MRI) to better understand the factors influencing
their development. It is important to remember, however, that "Chance
favors the prepared mind." (Pasteur)

*Creating Databases to Study Productivity.* There are few analyses of
productivity. A reliable and valid searchable NIH database containing
common longitudinal data on the careers of people and their ideas,
funding (or funding rejections), institutions, and products would assist
research in this area. Similarly, an analysis of 500 cases of innovation
might shed light on "where the breaks came."

**Factors to Consider in Studying and Measuring Constructs**

-   The more general a model, the better. But different approaches may
    be needed depending on whether one is assessing basic, clinical, or
    population science.

```{=html}
<!-- -->
```
-   Common definitions of terms are needed.

-   Identify areas where control can be influenced over a structure and
    where it cannot.

-   Work from the current state to the desired state and define the
    components that would comprise a framework for assessment.

```{=html}
<!-- -->
```
-   Some assessment tools are available, but knowing how to apply them
    to all areas is challenging.

-   What sort of processing systems are needed? Decisions have to be
    focused on knowledge, prioritization, and connection to decisions
    that NIH has the power to make (i.e., what NIH is authorized to
    fund?).

-   Recognize that change may influence the incentive system under which
    researchers function.

-   Recognizing opportunity and where scientists direct resources may be
    very different from doing innovative work. Scientists follow the
    money. It is not possible to assess where science is going; only
    where it has been.

-   Scientists may avoid research pathways with substantial regulatory /
    IRB requirements, complex patient recruitment or special collection.

-   It is critical to get input from all stakeholders.

-   It is important to recognize the phenomenon of simultaneous
    innovation.

-   How much do NIH funding decisions influence the model and how is it
    assessed?

-   Develop a method for scientists to record how they make decisions
    and set priorities.

-   Define the horizon for assessment and determining success---30 to 40
    years or 1 or 2 years?

-   Are new investigators, in fact, more innovative?

**Proposed Model for Study**

1.  Define current status (using bibliometrics, wiki, expert opinion).

2.  Identify gaps and prioritize those that should be pursued (involving
    all stakeholders).

3.  Identify areas or people who can pursue filling those gaps.

4.  Determine whether the science is ready.

Inherent in each step is a series of questions amenable to research.
Stakeholders should have input into prioritization.

**Relationships among the Constructs Depicting the Components of the
Priority Question**

**Factors Needed to Study and Measure Constructs**

+-----------------------+-----------------------+-----------------------+
| **                    | **Ways To Measure**   | > **Source**          |
| Construct/Component** |                       |                       |
+-----------------------+-----------------------+-----------------------+
| Define current state  | Create a database     | -   Data NIH develops |
| of research           | tool that is          |     internally        |
|                       | reliable, valid, and  |                       |
|                       | available for         | -   Data NIH can      |
|                       | investigating SoS     |     mandate from PIs  |
|                       | management questions? |                       |
|                       |                       | -   External data     |
|                       | Unfunded              |     (e.g.,            |
|                       | applications: fate of |     bibliometrics)    |
|                       | applicants? (not      |                       |
|                       | publicly available)   |                       |
+-----------------------+-----------------------+-----------------------+
| Define investigator   | Whether funding       |                       |
| quality               | individuals or        |                       |
|                       | funding projects      |                       |
|                       | yields more           |                       |
|                       | innovation?           |                       |
+-----------------------+-----------------------+-----------------------+
| Gaps---go / no-go     | How is the current    | -   Ask program       |
|                       | state of knowledge    |     directors and PIs |
|                       | assessed at NIH?      |     (new and senior)  |
|                       |                       |                       |
|                       | Develop criteria by   |                       |
|                       | using the             |                       |
|                       | characteristic        |                       |
|                       | elements of the       |                       |
|                       | actors (i.e., new PIs |                       |
|                       | more innovative?)     |                       |
+-----------------------+-----------------------+-----------------------+
| Decisions             | How do different      | -   Domain scores     |
|                       | people approach       |                       |
|                       | prioritization and    | -   Priority scores   |
|                       | how does it relate to |                       |
|                       | success?              | -   Percentiles       |
|                       |                       |                       |
|                       | Is the science ready? | -   Other inputs into |
|                       |                       |     funding decisions |
+-----------------------+-----------------------+-----------------------+

**Current State of Knowledge Assessment Work Group**

**Participants**

-   *Nora Volkow, Director, NIDA, Chair*

```{=html}
<!-- -->
```
-   Scott Stern, Associate Professor, Kellogg School of Management
    Northwestern University

-   Katy Börner, Associate Professor of Information Science and
    Informatics, Indiana University

-   Susan Cozzens, Director Technology Policy and Assessment Center,
    Georgia Institute of Technology

-   Edward Roberts, Professor of Management of Technology/Founder and
    Chair MIT Entrepreneurship Center, Massachusetts Institute of
    Technology

-   Gilbert Omenn, Professor of Internal Medicine, Human Genetics and
    Public Health, University of Michigan

-   Arthur Kleinman, Professor of Medical Anthropology Harvard
    University

-   Kevin Callahan, NIAID, Director, Office of Strategic Planning and
    Financial Management

-   Susan Gottesman, NCI, Senior Investigator, Biochemical Genetics

-   Mark Guyer, NHGRI, Director, Division of Extramural Research

-   Richard Fabsitz, NHLBI, Deputy Chief, Epidemiology Branch

-   Nancy Jones, NIAID, Planning and Evaluation Specialist, Strategic
    Planning and Evaluation Branch

-   Joni Rutter, NIDA

-   Kerry Goetz, OPASI

**Charge**

The group was charged to address the need to develop more appropriate
methodologies for assessing the products of scientific research, and for
understanding the types of knowledge generated (in addition to
publication) especially discovery, innovation, and across large systems.

> ***Overarching Guiding Question:*** What is needed for a comprehensive
> assessment of NIH knowledge generation and advancement?
>
> ***Priority Question***: What is needed for the assessment of NIH
> knowledge generation?

**Themes**

*Terminology, Assumptions, and Goals*: Should we be focused on the
"science of science" or on the "science of science management"? Is it
correct to assume that better science management will mean better and
more useful science? What are the untoward effects of science
management? Is it possible that it will create an "iron cage" or
bureaucracy that leads to worse outcomes? On the other hand, we cannot
continue to manage science based merely on hunches and anecdotes.
Perhaps we need a better definition of "knowledge" and an appreciation
of the many types of knowledge generated in addition to the accumulation
of research findings (e.g., skills, experience, wisdom). Ideas are not
the same as knowledge. Do we care about ideas as much we care about
knowledge? Should we focus on processes and metrics as part of science
management?

*What We Don't Know*: We don't understand scientific creativity, that
is, how do important ideas come about? Better understanding of what
constitutes scientific creativity would allow us to design research to
increase and foster idea development.

*Ways of Knowing*: How can we better recognize new ideas and/or
stimulate them, beyond the typical program announcements? We need more
real-time measures of science as it is being conducted and knowledge is
being generated instead of always looking back (e.g., through
publications). We need better tools to capture knowledge that is being
generated, in addition to bibliometrics, e.g., serendipitous results,
unanticipated results, and knowledge of how to manage science. We need
to expand our concept of knowledge outputs, e.g., education of other
scientists, better processes (new methods and tools),
research/scientific expertise and competencies, hypotheses confirmed,
refuted, or shown inappropriate negative results (alternative
knowledge), , ideas that stimulate more research/research questions,
patents, productive workshops.

*Approaches to Knowledge Generation*: Science can come from individuals
or collaborations. How do we weigh individual creativity versus team
approaches? What techniques encourage "out of the box"
cross-stimulation? Research on entrepreneurs has shown that the most
successful ones work in teams---productivity is related to size and time
together of the founding team. Research questions to be answered
include: 1) what kinds of teams and what kinds of teamwork are most
likely to generate different types of knowledge; 2) what is the
productivity of teams, and 3) can program management influence the
productivity of teams? How are answers influenced by the nature of the
question or problem? How do managers identify which projects require
interdisciplinary approaches, ideas, tools/platforms, so that a higher
weight can be given to teamwork?

*The Limits of Current Tools*: Old bibliometric models do not reflect
the aggregate, dynamic, transient, and collective efforts of teams of
scientists. They do not capture streaming data. The paradigm of
knowledge generation has changed yet the metrics have not (e.g.,
insufficient focus on interactions, collaborations).

**Factors to Consider in Studying and Measuring Constructs**

-   The assessment process should be connected to the mission---to
    advance human health.

-   What rewards can be developed to encourage the translation of highly
    specialized science advances into advances in health care and public
    health?

-   There is a gap between the knowledge that gets produced in the
    conduct of science and what gets reported. We don't disclose
    failure, which dramatically inhibits the productivity of future
    generations of scientists.

-   How do we know what knowledge is valid? Within a field people sense
    what is valid but this is poorly captured by metrics.

-   How does "conventional wisdom" inhibit innovative ideas?

-   How much redundancy is needed in science, and how can appropriate
    replication be managed?

-   Publications are still the central reward structure. This has to be
    recognized and more effort should be focused on more insightful and
    innovative use of bibliometrics. For example, using publications to
    capture hypotheses confirmed, refuted, modified, or discarded, as
    well as new methods and concepts, etc.

**Proposed Models for Study**

-   Study management and organizational models across NIH, looking at
    organizational range of management and organizational styles and
    structures.

-   Study review process to identify patterns associated with innovative
    ideas

-   Conduct case studies of different management and organizational
    styles.

-   Conduct natural experiments.

-   Create models of how science and science management currently
    function to assist in hypothesis development and testing

-   Study individual versus group science.

-   Develop infrastructure to allow and facilitate fast access to
    information (e.g., searchable databases, scholarly marketplace).

-   Study how NIH funding mechanisms and NIH policies (e.g. conflicts of
    interest reporting, data base sharing etc.) affect science
    conduct/practice (demographics of scientists, teams versus
    individuals or research types) and productivity such as knowledge
    generation.

-   Clinicaltrials.gov is a model of recorded results, whether successes
    or failures. Start asking grantees to report on three things learned
    but not published.

**Relationships among the Constructs Depicting the Components of the
Priority Question**

![](media/image2.wmf)

**Factors Needed to Study and Measure Constructs**

![](media/image3.wmf)

**Knowledge Utilization/Dissemination/Diffusion Working Group**

**Participants**

-   *Thomas Insel, Director, NIMH, Chair*

-   William Trochim, Professor, Department of Policy Analysis and
    Management, Cornell University

-   Jason Owen-Smith, Assistant Professor, Sociology and Organizational
    Studies, University of Michigan

-   Lynne Zucker, Professor of Sociology and Policy Studies, University
    of California-Los Angeles

-   Fiona Murray, Associate Professor, Management of Technological
    Innovation and Entrepreneurship, Massachusetts Institute of
    Technology

-   Michelle McMurry, Director, Health, Biomedical Science and Society
    Initiative, Aspen Institute

-   Edwin Flores, Founder, Chalker Flores LLP

-   Della Hann, NIMH, Director, Office of Science Policy and Program
    Planning

-   David Lipman, NLM, Director, NCBI; Senior Investigator

-   Anita Linde, NIAMS, Director, Office of Science Policy and Planning

-   Stephen Marcus, NCI, Scientist, Tobacco Control Research Branch

-   Patty Mabry, OD, Office of Behavioral and Social Sciences Research

-   Christina Clark, COPR

-   Evelyn Botchway, OPASI

**Charge**

The group was asked to focus on how to assess how knowledge and
resources generated from research results are communicated, distributed,
utilized, and adopted into behaviors, standards of care, science
policies, and the next generation of research.

> ***Overarching Guiding Question*:** How can social networks and
> collaborations among constituents/stakeholders facilitate the exchange
> and use of relevant knowledge to enhance learning and innovation and
> to facilitate the utilization of the information in practical
> applications and at key decision points?
>
> ***Priority Question*:** How can we best leverage social networks to
> facilitate information utilization?

**Themes**

Key questions raised include:

-   How do we understand the formation and dissolution of
    social/knowledge relationships?

-   What is the landscape of all the social networks? Who are the
    players, and what differing activities do they engage in together?

-   Where is the research knowledge being disseminated, and who are the
    final users within and outside of NIH?

-   How do biomedical and behavioral research breakthroughs ultimately
    get disseminated/go viral?

-   How does NIH and biomedical science find new ways to disseminate
    information in a Facebook-influenced society? What matrix should be
    created to crystallize NIH's role in this process?

-   What kind of information do stakeholders need for different
    purposes? How can NIH focus the public health community to use
    information dissemination to promote healthy choices?

-   How is the effectiveness of information dissemination measured? What
    metrics exist to measure "downstream" applications / translation of
    science findings to public health?

*Social Networks---How They Develop and Function:* Discussion involved
the various players and members of social networks, including the
producers and consumers of biomedical research, how they are identified,
and what promotes these relationships. There is a need to define the
landscape of all social networks related to biomedical science and what
an optimized product would look like. However, a range of different
pressures, constraints, and opportunities for forming different types of
interactions are in play involving a number of factors, including shared
materials and reagents, co-authorship, and licensing.

*How Relationships Could Develop in Science:* In social network terms,
one could envision relationships that can change fluidly---for example,
for two groups that simultaneously publish on reprogramming stem cells,
the acknowledgments and supplemental materials can help identify nodes
and connections (such as cited papers and shared materials) that can
delineate how each led to those outcomes. The evaluation of
collaborations in biotechnology and nanotechnology suggests that the
greater the number of articles published that list industry and academic
partners, the greater the success of the collaboration, although the
effect is greater with bigger "name" researchers than with lesser-known
colleagues. Commercialization is important, but it is not the sole or
even most important factor. Information flow is crucial to ensure that
state-of-the-art treatment guidelines are implemented, and a feedback
loop is important in identifying remaining barriers and deficiencies at
each phase of translation.

*Barriers to Social Networking in Biomedicine:* For the general U.S.
population, websites such as Amazon.com offer informal rating systems,
and Facebook promotes social networking. However, in biomedicine, such
web-based networking is unpopular because it does not offer standard
incentives, such as the "point system" of publishing and presenting that
is used in academia and at NIH. Other reasons social network systems are
not used include intellectual property/commercial concerns and delays in
the acquisition of relevant information.

Regarding barriers, put simply, the culture of traditional biomedical
science tends to impede the rapid communication of research results.
Intellectual property and patent issues also serve to undermine
information access. In addition, the broader public often has less
access to information than researchers do, even though the
public/consumer is often motivated and willing to do the necessary
background work to be able to understand most of this information.
However, different actors will need different kinds of information and
the methods of transmitting that information will vary.

The conduct of biomedical and behavioral research in the United States
is still "a proprietary enterprise" that rewards the individual and
inhibits broad collaboration. One result is the resistance to "big
science" and a preference for R01-oriented projects. It also is based on
a culture of single-person entrepreneurship that impedes broad
collaboration. The jargon and culture of differing branches of science
lead to difficulties between both individuals and groups of scientists
(i.e., basic versus clinical scientists, principal investigators versus
postdoctoral students). Other barriers include time, money, the
political environment, and disciplinary egocentrism. Also, different
standards of responsibility can create dampers on the flow of
information. NIH rightly takes responsibility for the research it
supports, but other players may not have the same standards. That may be
one reason why NIH and senior researchers have hesitated to embrace such
a completely open tool, but may be more willing to use semi-open,
controlled systems.

*Progress in Breaking Through Some Barriers:* However, there are
exceptions. Howard Hughes Medical Institute and the Wellcome Trust want
to develop an invitation-only web system to create a "rating and
commenting service" to rank scientific papers. This may represent a good
"middle ground" between the use of a completely open networking system
and the use of the current information dissemination method of journals
and scientific meetings. Also, some website/listservs for targeted
groups of researchers have started posting key papers as they are
published and are encouraging comments and discussions. They have grown
to include presentations and are being archived. As another example,
PubMed policy is to maximize access to information rapidly.

The focus in overcoming these barriers, it was agreed, should be on
creating a testable model of social networks, with practical measures of
those interactions.

*Conceptual Framework for Developing a Model**. ***The questions
involved in developing such a model include the following:

-   What kinds of knowledge are needed?

-   Who and what kinds of players want that knowledge?

-   For what purposes do they want that knowledge?

-   From whom do they want the knowledge?

-   What are the facilitating and impeding factors?

-   Through what kinds of channels can/should knowledge be disseminated?

-   Can this knowledge be tapped to help promote the marketing of
    healthy choices?

-   Once the communications networks are identified, how does one
    measure change and the impact of that change?

*Vectors of Dissemination**. ***These include publishing, both
traditional print and web based; online search engines; listservs;
training; gatekeepers---scientific, financial, legal, and journalistic;
integration through meta-analysis, review articles, and guidelines that
translate disparate information into useable formats; conferences and
presentations; informal face-to-face communications; community and
personal relationships; and disease groups.

*Sources of Public Input Back to Scientists**. ***These include
legislative bodies; funding priorities; patients and healthcare
providers; and NIH, industry, and clinical trial advisory bodies at
various levels.

*Measuring Transactions.* Various transactions can be measured, such as
**w**hen an investigator acknowledges others in a paper; time from
publication to first other use; and facilitators and barriers in each
interaction.

*Metrics: The Rate of Flow from A to B**. ***The best collaborations
occur when there is a high degree of openness and confidence in the
other partner; however, this also might produce the least amount of
documentation (e.g., conversations, sharing of materials, reagents). The
best records may, in fact, reflect an unwillingness to work with others,
or grudging collaboration. Citation measures do not work very well, and
there is a need to distinguish between citations within the research
niche and across disciplines and to distinguish between process measures
that tell how well the intermediate markers may function and the
long-term effect.

*Research Measures and Questions.* These include looking at sources of
variation across rows and down columns in the matrix; how
producer-to-producer outputs become the inputs to producer-to-user ties;
how outcomes of producer-to-user ties get bled into user-to-user ties;
when new information becomes integrated into classroom settings; the
extent of mechanisms by which researchers gather feedback, such as
community advisory groups, as part of their studies; organized feedback
from disease/consumer groups on how responsiveness has changed over
time; the impact of FDA regulatory approaches: strict on drugs and
biologics, nonexistent on surgical procedures; measurement of the
quality and context of information sharing; and negative and positive
framing of feedback.

Participants decided that the best way to understand the opportunities
and challenges in these research utilization and dissemination efforts
would be through a systems approach, using a matrix of producers and
users in which each relationship (producer to producer etc.) has the
potential for bidirectional interactions (see below). The group explored
the four possible interactions in the matrix and for each sought to
identify key partners in utilization and dissemination, the information
and other types of knowledge they share, methods of communication,
impediments to their communication and knowledge utilization, and,
ultimately, methods of measuring the effectiveness of dissemination.

![](media/image4.wmf)

General precepts:

-   Nothing in this matrix is prescriptive.

-   All matrix rows can be drilled down and particular columns can be
    compared across rows

-   The outputs of one row often are the inputs of another.

-   What makes this a system are the interactions among rows.

Issues/measures that are generic and cut across all categories include:

-   Time to use or adoption.

-   Rates of use or saturation.

-   Established infrastructure so that outcomes can be studied.

All players/stakeholders can be both producers and consumers of
information. Key players include:

-   Researchers---basic, translational, clinical.

-   Healthcare practitioners.

-   Patients and families.

-   Policy makers.

-   Congress and staff.

-   Federal agencies.

-   Advocates.

-   Disease advocacy groups.

-   High-income individuals---through direct funding and impact on
    policy makers.

-   Foundations.

-   Communications mediators such as health, science, and policy
    journalists.

**Public Health Impact Working Group**

**Participants**

-   *Paul Sieving, Director, NEI, Chair*

-   Doris Rubio, Associate Professor of Medicine, Biostatistics, and
    Nursing, University of Pittsburgh

-   Nate Osgood, Assistant Professor, Department of Computer Science,
    University of Saskatchewan

-   Daniel Sarewitz, Director of the Consortium for Science, Policy and
    Outcomes, Arizona State University

-   Harold Pincus, Professor, Department of Psychiatry, Columbia
    University

-   Ernst Berndt, Professor of Applied Economics, Massachusetts
    Institute of Technology

-   Phyllis Wise, Provost and Executive Vice President, University of
    Washington\
    Lori Mulligan, NCRR, Director, Office of Science Policy

-   Ronald Germain, NIAID, Senior Investigator, Immunology

-   Clifford Lane, NIAID, Senior Investigator, Division of Clinical
    Research

-   Richard Fisher, NEI, Associate Director for Science Policy and
    Legislation

-   Susan Daniels, NIAID, Health Scientist Administrator, Office of
    Scientific Coordination and Program Operations

-   Genevieve deAlmeida-Morris, NIDA

-   Kathryn Law, OPASI

**Charge**

The group was asked to focus on how public health impact can be assessed
to identify possible relationships with biomedical research.

> ***Overarching Guiding Question**:* What systemic models for improved
> public health, including pathways and contexts, could be useful for
> informing multiple NIH decision making processes?
>
> ***Priority Question**:* How do we measure the impact of NIH research
> on public health?

**Themes**

Key questions raised include:

1.  What is the actual \"product\" of NIH research?

2.  What are the facilitators and barriers to success in research?

3.  What should be assessed?

4.  Should NIH influence the research process and extrapolate forward in
    an attempt to reach for goals OR focus on assessing what is
    currently available?

5.  Should NIH identify a need, and then address how it would organize
    to address the need?

6.  Should the burden of disease be linked with NIH expenditures?

7.  How can NIH maximize its impact on public health?

8.  How can we better advocate to stakeholders?

*Factors to Consider in Studying and Measuring Construct***s**. Three
concepts underlie the priority question: It would be useful to know what
percentage of NIH research could be converted to research on public
health; NIH could take more of a microview or a macroview in
evaluation---in other words, tracking could be accomplished by
individual project or by research area; and useful models and metrics
would then give rise to questions related to the impact of NIH research
on public health that could be incorporated into an RFA.

*Approaches to Creating a Model as a Starting Point.* NIH directors\'
congressional testimonies could be reviewed to correlate the goals that
were delineated in such testimony with the actual accomplishments. In
addition, retrospective analysis could be conducted by selecting major
clinical advances in the last two decades and assessing NIH's role in
their development. There was discussion about how it may be advantageous
to project both backward and forward to identify the facilitators and
barriers to success in public health research. By way of a biological
science analogy, such approaches could to be likened to starting at the
receptor, at the surface of the cell, and then moving to the gene of
interest OR starting at the gene of interest and working backward.

*Assessment.* Maintaining a degree of sophistication about the
complexity of the system that is the subject of the research is
essential. For example, clinical research in HIV started with the
premise that a vaccine approach would be faster and easier than a
pharmaceutical approach, and investment in vaccines---in terms of
infrastructure and funding---was made before there was a vaccine
candidate. This approach differed from the one that was taken to
microbicide development. Also, the change in treatment guidelines was
driven by research and drug discovery. There is a need for systems-based
mapping of the complex phenomena that influence improvements in public
health, as well as a need for comparative assessments of both successes
and failures. Needed are appropriate surrogate markers, as well as
exploration of how the surrogate markers will be perceived by
stakeholders and the public. Using the war on cancer as an example, it
would be helpful to know how much of discovery has addressed the goal,
given the decades-long investment in this effort.

*Multiple-Phase Model*. To design a theoretical model and incorporate
specific cases into that model, one must begin with the input of all
variables---that is, the indicators of public health.

> *\"Things should be as simple as possible, but not simpler.\"* *Albert
> Einstein*

Because of the timeframe that can be involved in discovery, which can
range from 10 to 30 years from the time a research finding is made to
the development and implementation of public health treatments and cures
from that finding, a truly useful model will incorporate the appropriate
phases. Such a model also should enable researchers to perform
experiments that might lead to an increased public health impact, and it
should lead to definable improvements in public health, such as
innovations that may be associated with improvement of cardiovascular
health.

*Strategies.* Measure the state of public health---both morbidity and
mortality. Gather input from the public to help determine goals, and
identify which aspects of these goals require NIH funding. Assess what
should be considered long-term efforts that may involve research that
could cross over a number of diseases and look at what funding would be
required to fund this research, recognizing that it these may represent
high-risk efforts that may not ultimately result in the finding answers
that are needed. Work toward flexible investment that will provide the
ability to pursue a direction of research that might fall under the
high-risk category, with the recognition that different levels
risk-aversion will influence choices.

*Model Implementation*. The theoretical model needs to be applied
differently to basic versus clinical/translational research, because
qualitatively they are different and serendipity plays a variable role
in each. A flexible and more sensitive model would better address the
balance in the NIH research portfolio, recognizing, for example, that
some research is observational. The model should recognize variability
across different fields of research, because not all research takes
three decades to transition to clinical application. The model should
take into account that it must function within a particular social and
political environment, and such factors should be taken into account.

*Key Players: Mediators and Modulators.* Identify the impediments to
translating research into practice that are not within NIH's purview;
these include various social, political, and economic forces and
factors, and work to accommodate these factors (modulators) into models,
some of which may include new legislation, political changes, changes in
immigration patterns, economic changes, and changes in global public
health status. Recognize the importance of mediators and the many
variables that are involve in the healthcare system, such as the varying
levels of quality among physicians, patients' access to healthcare
services, and changes that may occur involving new discoveries and
disease outbreaks.

*Constructs*. These include decreasing mortality; reductions in
hospitalization; extended lifespan of people with AIDS; reduction in
dialysis; and reduction in the costs associated with managing diseases.

**Descending Objectives for Proposed Model(s)**

-   Create a model of the impact of NIH research on public health.

-   Find qualitative ways to assess the impact of research on public
    health.

-   Create methods to measure the impact of NIH research on public
    health.

-   Improve the tools that enable decision-makers to make public health
    assessments.

-   Use the SF-36 as a way to measure quality, cost, and health outcomes
    across all diseases AND track funding outcomes.

-   Design strategies that serve the public\'s interests.

-   Decide what NIH should be focusing on in the future, then assess
    available resources and determine what is needed in order to reach
    research goals.

-   Create a taxonomy regarding the decision-making pathways at NIH.

**Conceptual Schema: Relationships among the Constructs Depicting the
Components of the Priority Question**

![](media/image5.png){width="6.928472222222222in"
height="5.429166666666666in"}