# Presentation: 017588

## Patient-Reported Outcomes Measurement Information System 

## Why the need for improved PRO measures? 

-    Can we easily use our outcomes data for:
- clinical trial outcome evaluation?
- clinical decision-making?
- administrative and management purposes?
- health policy decision making?
- regulatory and market purposes?

**Notes:**
NOTE:  SLIDES #8 - #11 ARE FOR PRESENTING PROMIS TO OTHER AUDIENCES. PLEASE MODIFY ACCORDINGLY.

## Why the need for improved PRO measures in Clinical Trials? 

- Planned benefits of the PROMIS measures
- Responsiveness to differences should they exist across treatment groups.
- Improved performance where floor and ceiling effects are expected.
- Potential to reduce respondent burden
- Potential to reduce research cost

**Notes:**
NOTE:  SLIDES #5 - #7 ARE FOR PRESENTING PROMIS TO CLINICAL TRIALISTS.

## What's wrong with today's measurements ?

**0**

**1**

**2**

**3**

**- 1**

**- 2**

**- 3**

**Questionnaire****with a wide****range -****but low precision**

**1**

**2**

**Questionnaire****with a high****precision -****but small range**

**Notes:**
Static questionnaire are either not precise enough or their measurement range is to narrow ... ceiling and floor effects ...

## Computer Adaptive Tests

**0**

**1**

**2**

**3**

**- 1**

**- 2**

**- 3**

**high ****depression**

**low ****depression**

**Questionnaire****with a high****precision -****AND**** a****wide range**

**Notes:**
Purpose of this slide is to walk the audience through the CAT process using an example ...and thereby transmit the message of narrower measurement range and higher precision with each question ...
I typically ese e.g. an example for depression, or any other topic I want to focus on ... like first question “do you feel depressed”, answer “some of the time” ... than we would like to know how depressed and the CAT might choose a question asking about self worth, and if the patient indicates that this is diminished also, a third question asking about thoughts of suicide might be appropriate ... in this case the answer would be no, never ... 
And maybe compare it with another scenario like the participant would have answered the first question, no I am not at all depressed, asking about suicide would not make sense, instead the CAT might have chosen a question asking about happiness ...  bla bla
The conclusion is, we can make precise test over the entire range ...

## Themes of the NIH Roadmap

**New Pathways to Discovery**
**Research Teams of the Future**
**Re-engineering the Clinical Research Enterprise**

## Re-engineering the Clinical Research Enterprise

- To continue NIH’s mission of successful medical research, it will need to recast its entire clinical research system
- Requires the development of new partnerships of research with organized patient communities, community-based health care providers, industry, and academic researchers.
- Need new paradigms in how clinical research information is collected, used, and reported.
- Includes the advances in information technology, psychometrics, and qualitative, cognitive, and health survey research.

## The PROMIS of a better future...

- A publicly available, adaptable and sustainable Internet-based system that will:
- Administer individually “tailored” questionnaires (using Computer Adaptive Testing (CAT) technology) to measure health status outcomes
- Collect and analyze responses
- Provide instant health status reports to users to:
- Enhance research
- Improve clinical decision-making
- Facilitate policy-making by health plan and systems and public programs

## PROMIS: Long-term Objectives

- Create a publicly available, adaptable and sustainable Internet-based system, the Patient-Reported Outcomes Measurement Information System (PROMIS) -- that will:
- Administer “tailored” questionnaires (using CAT technology) that measure a patient’s health status.
- Collect the patients’ responses for research and for upgrading the system.
- Provide instant health status reports to patients and health care providers to improve treatment decision making.
- Lay groundwork for public-private partnership to extend the PROMIS beyond its five-year development stage.

## PROMIS integrates the fields of...

**Notes:**
This project will utilize the advances in information technologies, the strengths of modern measurement theory, and the contributions from the fields of psychometrics, cognitive psychology, and health survey methods to develop a new and dynamic PRO measurement and reporting.

## PROMIS Network Structure  

- PROMIS Network Structure

**Notes:**
$25+ Million, 5-Year cooperative agreement between NIH and extramural investigators (using a U01 Grant Mechanism) including six Primary Research and Data Collection Sites (PRS) and a Statistical Coordinating Center (SCC).  The PROMIS Steering Committee (SC) holds conversations among PIs on a monthly basis and meets face-to-face about every 3-4 months.  The Scientific Advisory Board (SAB) advises the NIH.  The SAB consists of Drs. Mick Couper (U. of Michigan), Jacqueline Dunbar-Jacob (U. of Pittsburgh), Ronald Kessler (Harvard), Colleen McHorney (Merck), Dick Moxley (Strong Memorial Hospital), Donald Patrick (U. of Washington), Ruth Stein (Albert Einstein College of Medicine), Laurie Burke (FDA), Lisa Hines (CMS), Don Detmer (U. of Virginia), and Margaret Rothman (Johnson and Johnson).

## The PROMIS Network

**Notes:**
Note:  This map identifies the Statistical Coordinating Center (Evanston Northwestern – David Cella, PI) and 6 Primary Research and Data Collection Sites (Duke, Kevin Weinfurt, PI; Stanford, James Fries, PI; SUNY, Arthur Stone, PI; UNC, Harry Guess, PI; Pittsburgh, Paul Pilkonis, PI; Washington, Dagmar Amtmann, PI).  Non-labeled dots represent sub-contractors.  Deborah Ader is the NIH Program Director.

## PROMIS Domain Hierarchy

**Mental Health**

**Physical Health**

**  ****Symptoms**

**Pain**

**Other**

**Social Health**

**Role Participation**

**Social Support**

**Fatigue**

**Positive Psychological Functioning**

**Cognitive Function**

** ****Emotional Distress**

**Subjective Well-Being (positive effect)**

**Positive Impacts of Illness**

**Meaning and Coherence (spirituality) **

** ****Mastery and Control (self-efficacy)**

**Performance**

**Satisfaction**

**Health****PRO**

**Satisfaction**

**Satisfaction**

**Satisfaction**

**Notes:**
One possible way to focus on one sub domain ... (with animation)

Note:  Yellow Highlighted Domains are the PRO domains that will be covered in PROMIS first round of testing to begin in April 2006.  The “General Health” Domain is an overall evaluation by patient of the condition of their health.  Each measured domain will also include a global item assessing their level on the construct.

## Advantages of adding IRT to Classical Test Theory

- Item Response Theory focuses on the mathematical relationship of *items, not scales,* to the latent trait
- Advantages:
- Scale reduction – potentially more precision with less items
- Scale flexibility – different items to measure the same trait
- Equate scores of different scales (crosswalking)
- Test item equivalence across groups (DIF)
- Tailored administration (CAT)

## Leveraging Advances in Computer and Internet Technology

- Continuous access to PRO measurement
- Automated administration, validation checks, and data recording, storing, and scoring
- Enhanced graphic interface to improve format and presentation for patients (e.g. increased accessibility)
- Immediate feedback of a patient’s health status both to the patient and provider

## Item Bank

**(Validated & IRT-Calibrated Emotional Distress Items)**

## Item Bank

**(Validated & IRT-Calibrated Emotional Distress Items)**

- -3

- -2

- -1

- 0

- 1

- 2

- 3

- Severe

- high

- moderate

- low

- very low

**Emotional Distress**

**How often did you feel nervous?**

**Some of the time **

## Item Bank

**(Validated & IRT-Calibrated Emotional Distress Items)**

- -3

- -2

- -1

- 0

- 1

- 2

- 3

- Severe

- high

- moderate

- low

- very low

**Emotional Distress**

**How often did you feel nervous?**

**Some of the time **

## Item Bank

**(Validated & IRT-Calibrated Emotional Distress Items)**

- -3

- -2

- -1

- 0

- 1

- 2

- 3

- Severe

- high

- moderate

- low

- very low

**Emotional Distress**

## Item Bank

**(Validated & IRT-Calibrated Emotional Distress Items)**

- -3

- -2

- -1

- 0

- 1

- 2

- 3

- Severe

- high

- moderate

- low

- very low

**Emotional Distress**

**How often did you feel hopeless?**

**Some of the time **

## Item Bank

**(Validated & IRT-Calibrated Emotional Distress Items)**

- -3

- -2

- -1

- 0

- 1

- 2

- 3

- Severe

- high

- moderate

- low

- very low

**Emotional Distress**

## Item Bank

**(Validated & IRT-Calibrated Emotional Distress Items)**

- -3

- -2

- -1

- 0

- 1

- 2

- 3

- Severe

- high

- moderate

- low

- very low

**Emotional Distress**

**How often did you feel worthless?**

**Little of the time **

## Item Bank

**(Validated & IRT-Calibrated Emotional Distress Items)**

- -3

- -2

- -1

- 0

- 1

- 2

- 3

- Severe

- high

- moderate

- low

- very low

**How often did you feel worthless?**

**Little of the time **

**Target in on**
**emotional **
**distress score**

## PROMIS Item Bank Development

- Comprehensive collection and review of existing items (legacy items)
- Development of new and modified items (approx. 8000 total items)
- Binning and winnowing of items (1064 items)
- Readability analysis and revisions
- Focus groups
- Cognitive interviews (784 items)

## How Clinical Researchers would use PROMIS

- Set up assessment protocol, including selecting domains and modes of administration
- Patients complete at office or anywhere
- Researchers provided with tracking on completion and results
- Downloadable dataset
- Documentation of the PROMIS system for use in publications, FDA submissions, etc.

## What PROMIS Is Not

- PROMIS measures patient reports –
- It is not a laboratory or performance measure
- PROMIS currently measures a limited set of clinically relevant domains
- It does not measure all clinically relevant patient reported outcomes (at least not yet)
- PROMIS measures outcomes –
- It is not adequate alone to use for most screening or diagnostic needs

## PROMIS: More than the Network Project

**Independent Projects**
**Patient Reported Sleep Domains (Pittsburgh)**
**Pain and Fatigue in Children with Disabilities (Washington)**
**PROMIS for Pediatric Samples (UNC)**
**Outcomes in Arthritis and Aging Populations (Stanford)**
**IRT in Multi-Site Clinical Trials (Duke)**
**Ecological Validity in Patient Reported Outcomes (Stony Brook)**
**Spin-Off Projects**
**Patient Reported Outcomes in Cancer Treatment Trials**
**Quality of Life Outcomes in Neurological Disorders**

## Information

**PROMIS:     **_[www.nihPROMIS.org](http://www.nihpromis.org/)_
**Roadmap:    **_[www.nihroadmap.nih.gov](http://www.nihroadmap.nih.gov/)_
**NIH Science Officers:**
**William Riley, NIMH**
**Bryce Reeve, NCI**
**Larry Fine, NHLBI**
**Lou Quatrano, NICHD**
**Susan Czajkowski, NHLBI**
**Suzana Serrate-Sztein, NIAMS**
**NIH representatives from numerous other institutes**

## What is Computerized Adaptive Testing (CAT)?

- A technique for administering a PRO instrument that
- selects questions on the basis of a person’s response to previously administered questions
- each question, stored in an “item bank” has been psychometrically and qualitatively reviewed as informative for measuring the health construct
- determines a person’s score with the minimal number of questions and no loss of measurement precision

## Advantages of CAT Assessments

-   Provide an accurate estimate of a  person’s score with the minimal number of questions
- Questions are selected to match the health status of the respondent
-   Minimize floor and ceiling effects
- People near the top or bottom of a scale will receive items that are designed to assess their health status

## Item Bank

**(IRT-calibrated items reviewed for**
**reliability, validity, and sensitivity) **

## Item Response Theory (IRT) Modeling

- 	A family of models that describe, in probabilistic terms, the relationship between people’s responses to  questions and their position on the continuum of what is being measured (e.g., pain)

## Item Response Theory (IRT) Modeling

- IRT assigns properties to each question that provide information on which people a given question is best suited for

## What is the PROMIS Potential?

- Enhance national capacity to
- evaluate effectiveness of all health interventions, prevention, diagnosis, treatment, rehabilitation, palliation
- monitor progress against burdens of disease
- support a wide range of studies on the determinants of health care utilization and outcomes

## What is the PROMIS Time Line?

**2004-05:** 	Choose specific domains
-                   	Identify, review instruments and items
**2006-07:**  	Build item pools in 5 domains
- 			Collect response data
- 		 	Create alpha version of CAT
- 			Build collaborative alliances
**2008-09:**       	Conduct final calibration process
- 	             	Put CAT into final form
- 			Conduct second major network project
- 			Feasibility tests and User Group meetings                		Build sustaining partnerships

## PROMIS Website http://www.nihPROMIS.org/

**Contact Information:**
- Shani Rolle, M.S.
- NIH Coordinator
- RolleS@mail.nih.gov