User’s Guide to Implementing
Patient-Reported Outcomes
Assessment in Clinical PracticeVersion 2: January 2015

Produced on behalf of theInternational Society for Quality of Life Research by(in alphabetical order):Neil Aaronson, PhD
Thomas Elliott, MD
Joanne Greenhalgh, PhD
Michele Halyard, MD
Rachel Hess, MD
Deborah Miller, PhD
Bryce Reeve, PhD
Maria Santana, PhD
Claire Snyder, PhD

1

TABLE OF CONTENTS
Introduction ......................................................................................................................................................... 3
What are your goals for collecting PROs in your clinical practice and what resources are available? Whichkey barriers require attention? ........................................................................................................................... 4
Which groups of patients will you assess? .......................................................................................................... 7
How do you select which questionnaire to use? ............................................................................................... 11
How often should patients complete questionnaires? Should it be tied to visits or a way to follow patients
between visits? .................................................................................................................................................. 14
How will the PROs be administered and scored? .............................................................................................. 17
What tools are available to aid in score interpretation and how will scores requiring follow-up be
determined? ...................................................................................................................................................... 21
When will results be presented? ...................................................................................................................... 27
Where will results be presented? .....................................................................................................................29
How will results be presented?.........................................................................................................................31
Who will receive score reports? ...................................................................................................................... 33
What will be done to respond to issues identified through the PROs? ........................................................... 35
How will the value of using PROs be evaluated? ............................................................................................ 37
Bibliography .................................................................................................................................................... 40
DISCLAIMER: This User’s Guide was reviewed and approved by the ISOQOL Board of Directors as an ISOQOL
publication and does not reflect an endorsement of the ISOQOL membership.
SUGGESTED CITATION: International Society for Quality of Life Research (prepared by Aaronson N, Elliott T,
Greenhalgh J, Halyard M, Hess R, Miller D, Reeve B, Santana M, Snyder C). User’s Guide to Implementing
Patient-Reported Outcomes Assessment in Clinical Practice, Version: January 2015.
ACKNOWLEDGEMENT: We appreciate the contributions of Ali Choucair, MD, on the first version.

2

INTRODUCTION

This User’s Guide to Implementing Patient-Reported Outcomes Assessment in Clinical Practice was developedby a team of volunteers from the International Society for Quality of Life Research (ISOQOL). The purpose ofthis User’s Guide is to help clinicians who are interested in using patient-reported outcome (PRO) measures intheir clinical practice as a tool in patient management. For the purposes of this Guide, PROs refer to patientreports on any of various outcomes, including symptom burden, functioning, health status, and health-related
quality of life. Notably, while this Guide addresses patient-reported outcomes specifically, many of the same issues
apply to other types of patient-reported information such as health behavior (e.g., exercise levels, alcohol and tobaccouse) and health history (e.g., family history, medical history). While PROs can be considered a subset of all patientreported information, not all patient-reported information is necessarily a PRO.

The questions addressed by this User’s Guide include the following:
1. What are your goals for collecting PROs in your clinical practice and what resources are available? Whichkey barriers require attention?
2. Which groups of patients will you assess?
3. How do you select which questionnaire to use?
4. How often should patients complete questionnaires? Should it be tied to visits or a way to follow patients
between visits?
5. How will the PROs be administered and scored?
6. What tools are available to aid in interpretation and how will scores requiring follow-up be determined?
7. When, where, how, and to whom will results be presented?
8. What will be done to respond to issues identified through the PROs?
9. How will the value of using PROs be evaluated?
As you will see, this User’s Guide does not aim to provide the “right” answers to these questions. Rather, theUser’s Guide presents different options for responding to each of these questions, so that each practice candetermine which approach is “right” for its own goals and needs. To that end, for each of the questions, theUser’s Guide provides the following information:
 the different options for answering each of the questions posed above, the resources needed to implement the various options,
 the advantages and disadvantages of the various options, and useful references for more information.
We hope that a wide range of practice types will find this User’s Guide helpful. The field of assessing PROs inclinical practice is continuing to develop and evolve, and the web version of this User’s Guide will be updated
periodically. We welcome input regarding what you have found works (and doesn’t work) in your practice.
Please send feedback and questions to info@isoqol.org.

3

What are your goals for collecting PROs in your clinical practice and what
resources are available? Which key barriers require attention?
Before implementing any intervention involving the use of PROs in your clinical practice, it is critical to clarify
the goals of the intervention and to assess the resources available for implementing the intervention. Itshould be noted that the options presented below are not necessarily mutually exclusive, and that practices
may be able to implement PROs to accomplish multiple objectives. For example, PROs can initially becollected for individual patient management, aggregated for quality evaluation, and then analyzed to inform
quality improvement.
The resources involved include manpower, information systems and technical support, space, and financial
investment. The level of resources required depends on how the intervention is implemented and is discussed
in more detail in later sections.
In many cases the advantages and disadvantages of implementing PROs in clinical practice are similar,
regardless of the particular goals. Advantages include encouraging clinicians to treat the “whole” patient
rather than just the disease, facilitating communication without increasing consultation time, and engaging
patients in their own care. Barriers are at the level of clinicians (lack of familiarity with the instruments, doubt
about the ability of PROs to modify outcomes, time and resource constraints, disagreements over impact onpatient-clinician relationship), patients (literacy, being too sick to complete questionnaires, concern about
impact on relationship with clinician), and health system (reimbursement, fit within the clinical workflow).
A taxonomy of applications of PROs in clinical practice includes the following:
A. SCREENING TOOLS
Resources Needed: One-time PRO assessment with feedback to clinician.
Advantages:
 Can help identify problems that may have otherwise gone unnoticed.
Disadvantages:
 Provides no information on how the patient’s outcomes may be changing over time.

B. MONITORING TOOLS
Resources Needed: PRO data collection over time with feedback of results to clinician.
Advantages:
 Can track patient’s outcomes over time.
 Helps evaluate whether interventions are effective.
4

May facilitate modifying interventions as needed.

Disadvantages:
 More resource intensive.
C. PATIENT-CENTERED CARE
Resources Needed: PRO data collection with feedback of PRO results to patients, in addition to clinicians.
Advantages:
 Facilitates discussion between patients and clinicians regarding patients’ issues, including
their concerns and their priorities for care.
 Can lead to patients becoming more involved in their care and improved self-efficacy.
 May also produce better patient compliance, improved health outcomes, and greater
patient satisfaction.
Disadvantages:
 May be more resource intensive to feedback results to patients in addition to clinicians.
D. DECISION AIDS
Resources Needed: While PROs are not decision aids themselves, PRO data can be used in decision aids toprovide information on the impact of the different treatment options on PROs. Decision
aids which include PRO data require information about treatment options, their impact onPROs, and the probability of various PRO outcomes.
 Decision aid that presents PRO information to patients in an understandable manner andthat allows them to clarify their values.
Advantages:
 Helps patients understand treatment options’ impact on PROs.
 Allows for an explicit weighing of risks and benefits.
Disadvantages:
 Aids are not available for all decisions.
E. FACILITATING MULTIDISCIPLINARY TEAM COMMUNICATIONSResources Needed: Method for multidisciplinary teams to share patient PRO data.
Advantages:
 Provides a common data source for clinicians from diverse backgrounds to discuss patients’progress.
 Provides the patient perspective to clinicians’ discussions of treatment planning andevaluation.
5May assist clinicians in agreeing on and implementing care plan.

Disadvantages:
 Requires training of clinicians from multiple disciplines in interpreting PRO results.
F. EVALUATING QUALITY OF CARE
Resources Needed: Pooled data from patients from within the practice, preferably with normative data forcomparison.
Advantages:
 Can help identify strengths and weaknesses in care provided.
 May allow clinicians to compare their practice outcomes to benchmarks or other normativedata.
 Provides information on effectiveness, rather than efficacy.
 If data are publicly available, enables patients and purchasers to compare providers on
PROs.
Disadvantages:
 Does not necessarily involve using individuals’ PRO results to aid in their management.
 Link between quality of care and PROs has been questioned, due to confounders andlimitations in case mix adjustment.
KEY REFERENCES
1. Detmar SB, Aaronson NK. Quality of life assessment in daily clinical oncology practice: a feasibility study.
Eur J Cancer 1998; 34: 1181—6.
2. Donaldson M. Using patient-reported outcomes in clinical oncology practice: Benefits, challenges, andnext steps. Expert Review of Pharmacoeconomics and Outcomes Research 2006; 6: 87—95.
3. Greenhalgh J. The applications of PROs in clinical practice: What are they, do they work, and why? Qual
Life Res 2009; 18: 115—123.
4. Higginson IJ, Carr A. Measuring quality of life: Using quality of life measures in the clinical setting. BMJ
2001; 322: 1297—1300.
5. McHorney CA. Health status assessment methods for adults: Past accomplishments and futurechallenges. Annual Rev of Public Health 1999; 20: 309—35.
6. Santana MJ, Feeny D. Framework to assess the effects of using patient-reported outcome measures inchronic care management. Qual Life Res 2013 Dec 7 [Epub ahead of print].
7. Velikova G, Booth L, Smith AB, et al. Measuring quality of life in routine oncology practice improvescommunication and patient well-being: A randomized clinical trial. J Clin Oncol 2004; 22: 714—24.
8. Wasson JH, Stukel TA, Weiss JE, et al. A randomized trial of the use of patient self-assessment data to
improve community practices. Eff Clin Pract 1999; 2(1): 1—10.
9. Wu AW, Jensen RE, Salzburg C, Snyder C. Advances in the Use of Patient Reported Outcome Measures inElectronic Health Records: Including Case Studies. Landscape Review Prepared for the PCORI National
Workshop to Advance the Use of PRO measures in Electronic Health Records. Atlanta, GA. November 1920, 2013, pp 10-14. Available at: http://www.pcori.org/assets/2013/11/PCORI-PRO-Workshop-EHRLandscape-Review-111913.pdf.
6Which groups of patients will you assess?
The key considerations in determining which patients in your practice you want to complete PROs include
patients’ ability to self-report and the setting of care. Options include the following:
A. ONLY PATIENTS WHO CAN SELF REPORTResources Needed: Ability to identify patients capable of self-report.
 Versions of questionnaires appropriate for the languages/cultures of the patient population(translations and cultural adaptations are frequently available from instrument developers).
Advantages:
 Individuals who self-report are providing direct assessment.
Disadvantages:
 Some individuals may appear to be competent but may be too impaired to validly andreliably represent themselves.

B. PATIENTS REQUIRING ASSISTANCE (E.G., YOUNG CHILDREN, MENTALLY OR COGNITIVELY LIMITED,ETC.)
Resources Needed: Proxies need to be identified in terms of relationship to patients and it should be explicitly
noted when proxy reporter varies from administration to administration.
Advantages:
 In cases where the index patient is unable to provide responses because of incapacity, theinput of significant others provides some perspective on the affected person.
 Proxies (e.g., caregivers, physicians) can provide useful information particularly on more
concrete, observable PROs.
Disadvantages:
 Proxies may have a difficult time distinguishing between how their family member would
respond and what they perceive that person’s status to be.
 Responses from proxies may be influenced by their own feelings about and experiences ofcaring for the patient.
 It has been demonstrated that even when using instruments developed using modern test
theory (MTT), there is limited agreement between children or adolescents and parent
proxies.

7

C. ALL AMBULATORY PATIENTS
Resources Needed: Ambulatory patients are more likely to be more independent and have more discretionary
time to complete measures.
 It is recommended that a generic PRO is administered once a year.
Advantages:
 Provides an opportunity to identify unknown problems, disabilities, and limitations.
Disadvantages:
 Ambulatory patients may have had significant illness episodes affecting PROs between theirambulatory visits. These illness episodes may have resolved and thus not be captured in thecurrent assessment.
D. AMBULATORY PATIENTS WITH SPECIFIC CONDITIONSResources Needed: Individuals with an identified chronic illness, whether followed in a general or specialty
clinic, should complete a related PRO at least every 6 months, with weekly assessments
considered for patients undergoing outpatient treatment for certain conditions (e.g.,
cancer).
 Ambulatory patients with specific conditions that affect vision or hand function should haveaccess to adapted means of completing questionnaires either remotely or on site.
Advantages:
 Focusing on patients with specific conditions allows for a more targeted measurement
strategy.
Disadvantages:
 Additional staff may be required to assist ambulatory patients with specific conditions that
affect vision or hand function.
E. INPATIENT ACUTE CARE PATIENTS
Resources Needed: Hospitalized patients in acute medical settings will likely require help completing PROs,
regardless of method of administration.
 Time to complete PROs will need to be built into the schedule.
 State of alertness will determine whether acute care patients can respond.
Advantages:
 Acute care patients are readily available to complete measures depending on availability ofstaff to help.
 Hospitalized patients will experience more rapid variation in PROs and assessments should
occur more often and with appropriate recall.
8

Disadvantages:
 Acute care patients may find that completing the questionnaires is not useful during theirshort-term stay.
 For hospitalized patients, there will be no information to be gained about the long-term
benefit of a hospitalization, if there is no post-discharge assessment.
 The environment of a hospitalized patient may influence responses.

F. INPATIENT REHABILITATION PATIENTS
Resources Needed: Patients hospitalized in rehabilitation settings have structured schedules and areundergoing constant evaluation for progress.
Advantages:
 Patients hospitalized in rehabilitation settings can have these assessments built into theirschedules.
Disadvantages:
 If lack of progress is reflected in these measures, insurers may use the data to shorten
length of stay.
KEY REFERENCES
1. Ackerley SJ, Gordon HJ, Elston AF, Crawford LM, McPherson KM. Assessment of quality of life andparticipation within an outpatient rehabilitation setting. [Erratum appears in Disabil Rehabil. 2009; 31(13):1107]. Disability & Rehabilitation 2009; 31: 906—913.
2. Addington-Hall J, Kalra L. Who should measure quality of life? BMJ 2001; 322: 1417—1420.
3. Coons SJ, Gwaltney CJ, Hays RD, et al. Recommendations on evidence needed to support measurement
equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO
Good Research Practices Task Force Report. Value Health 2009; 12(4): 419—429. Available at:
http://www.ispor.org/taskforces/eprotf.asp.
4. Eaton ML. Surrogate decision making for genetic testing for Alzheimer disease. Genetic Testing 1999; 3:
93—97.
5. Foley JF, Brandes DW. Redefining functionality and treatment efficacy in multiple sclerosis. Neurology
2009; 72: S1—11.
6. Giesinger JM, Golser M, Erharter A, et al. Do neurooncological patients and their significant others agree
on quality of life ratings? Health & Quality of Life Outcomes 2009; 7: 87.
7. Hays RD, Kim S, Spritzer KL, et al. Effects of mode and order of administration on generic health-related
quality of life scores. Value in Health 2009; 12: 1035—1039.
8. Kocks JWH, van den Berg JWK, Kerstjens HAM, et al. Day-to-day measurement of patient-reported
outcomes in exacerbations of chronic obstructive pulmonary disease. International Journal of COPD 2013;
8: 273—286.
9. Masskulpan P, Riewthong K, Dajpratham P, Kuptniratsaikul V. Anxiety and depressive symptoms after
stroke in 9 rehabilitation centers. Journal of the Medical Association of Thailand 2008; 91: 1595—1602.
10. Naglie G, Tomlinson G, Tansey C, et al. Utility-based quality of life measures in Alzheimer's disease. Qualityof Life Research 2006; 15: 631—643.
911. Novella JL, Jochum C, Jolly D, et al. Agreement between patients' and proxies' reports of quality of life in
Alzheimer's disease. Quality of Life Research 2001; 10: 443—452.
12. Pickard AS, Knight SJ. Proxy evaluation of health-related quality of life: a conceptual framework forunderstanding multiple proxy perspectives. Medical Care 2005; 43: 493—499.
13. Pickard AS, Lin H-W, Knight SJ, et al. Proxy assessment of health-related quality of life in African Americanand White respondents with prostate cancer: perspective matters. [Erratum appears in Med Care 2009
Apr; 47 (4): 491 Note: Knight, Sara L [corrected to Knight, Sara J]]. Medical Care 2009; 47: 176—183.
14. Ready RE, Ott BR, Grace J. Patient versus informant perspectives of quality of life in mild cognitive
impairment and Alzheimer's disease. International Journal of Geriatric Psychiatry 2004; 19: 256—265.
15. Seid M, Limbers CA, Driscoll KA, Opipari-Arrigan LA, Gelhard LR, Varni JW. Reliability, validity, andresponsiveness of the pediatric quality of life inventory (PedsQL) generic core scales and asthma symptoms
scale in vulnerable children with asthma. Journal of Asthma; 47: 170—177.
16. Varni JW, Thissen D, Stucky BD, et al. PROMIS parent proxy report scales: an item response theory analysisof the parent proxy report item banks. Quality of Life Research 2012; 21:1223—40.
17. Veenstra M, Moum T, Garratt AM. Patient experiences with information in a hospital setting: associations
with coping and self-rated health in chronic illness. Quality of Life Research 2006; 15: 967—978.
18. Yazicioglu K, Duyan V, Karatas K, et al. Effects of sociodemographic characteristics, illness process, andsocial support on the levels of perceived quality of life in veterans. Military Medicine 2006; 171: 1083—
1088.

10

How do you select which questionnaire to use?
There are several considerations when determining which questionnaire to use. These include whether to use
generic or disease-specific questionnaires, profile or preference-based measures, single or multi-item scales,and static or dynamic questionnaires. For all of the options, various attributes of the questionnaires should be
evaluated, including response formats (verbal descriptor scale or numeric rating scale), focus of assessment(severity, frequency, interference, bother), time burden, and the level psychometric evidence (validity,
floor/ceiling effect, etc.). One should also consider the reference period. More recent recall periods moreaccurately capture patients’ actual experiences, but short reference periods require either more frequentassessments (meaning more burden) or may miss important symptoms between assessments.
Careful attention should be given to determining what type of patient-reported outcome to collect, such as
symptoms (e.g., pain, fatigue, nausea, depressive mood), functioning (e.g., activities of daily living, cognitive
functioning), or quality of life (e.g., mental, physical, and social well-being). Some questionnaires include a mixof these types of data. Selection of the content of the questionnaire should consider clinician or patient
preferences. Physicians may only want to assess symptoms they know how to treat. For patients, there may
be specific symptoms and quality-of-life issues they want to talk to the doctor about.
The different types of PRO questionnaires are described below.
A. GENERIC OR DISEASE-SPECIFIC QUESTIONNAIRES
Resources Needed: Permission to use questionnaire, if required.
 User fee, if required by instrument developers.
Advantages:
 Generic questionnaires may capture more common health-related quality of life domains
and allow comparisons to normative populations.
 Disease-specific questionnaires may be more sensitive to specific symptoms experienced bypatients.
Disadvantages:
 Generic questionnaires may not be sensitive to changes over time.
 Disease-specific questionnaires may miss domains affecting the patient but unrelated to thedisease under study.
B. NON-PREFERENCE (PROFILE) OR PREFERENCE-BASED MEASURES
Resources Needed: Permission to use questionnaire, if required.
 User fee, if required by instrument developers.

11Advantages:
 Profile measures provide multiple scores (and sometimes summary measures) across a
broad range of PRO domains. This helps the clinician determine what aspects of a person’shealth is impacted.
 Preference measures provide a single score aggregated across multiple PRO domains for an
estimate of burden of disease. Often, the score is weighted based on feedback from a key
stakeholder group (e.g., general population, patients, clinicians).
Disadvantages:
 Profile measures are often longer to complete.
 Preference measures may not provide clinically relevant information on the specific PRO
domains affecting the patient.
C. SINGLE OR MULTI-ITEM SCALES
Resources Needed: Permission to use questionnaire, if required.
 User fee, if required by instrument developers.
Advantages:
 Able to measure more domains if only use a single item for each.
 Multi-item scales provide more reliable/sensitive/content valid measurement.
Disadvantages:
 Single items are less reliable for tracking change.
 Multi-item scales are more burdensome (time consuming) for patients and clinicians.
D. STATIC OR DYNAMIC QUESTIONNAIRES
Resources Needed: Permission to use questionnaire, if required.
 User fee, if required by instrument developers.
 Dynamic questionnaires require computer-based assessment and access to validated item
banks and computer-adaptive test (CAT) software.
Advantages:
 Static forms can work on paper and on computer.
 CAT measures are more efficient and allow more domains to be assessed.
Disadvantages:
 Unless very brief with simple scoring, static forms are a burden to administer and score.
 CAT requires computer assessment.

12

KEY REFERENCES
1. Cella D, Gershon R, Lai JS, Choi S. The future of outcomes measurement: item banking, tailored shortforms, and computerized adaptive assessment. Qual Life Res 2007; 16 Suppl 1: 133—141.
2. Chang CH. Patient-reported outcomes measurement and management with innovative methodologies andtechnologies. Qual Life Res 2007; 16 Suppl 1: 157—166.
3. Chen TH, Li L, Kochen MM. A systematic review: how to choose appropriate health-related quality of life
(HRQOL) measures in routine general practice? J Zhejiang Univ Sci B 2005 Sep; 6(9): 936—940.
4. Lohr KN, Zebrack BJ. Using patient-reported outcomes in clinical practice: challenges and opportunities.
Qual Life Res 2009 Feb; 18(1): 99—107.
5. Nowels D, McGloin J, Westfall JM, Holcomb S. Validation of the EQ-5D quality of life
instrument in patients after myocardial infarction. Qual Life Res 2005 Feb; 14(1): 95—105.
6. Osoba D. Translating the science of patient-reported outcomes assessment into clinical practice. J NatlCancer Inst Monogr 2007; 37: 5—11.
7. Snyder CF, Dy SM, Hendricks DE, Brahmer JR, Carducci MA, Wolff AC, Wu AW. Asking the right questions:investigating needs assessments and health-related quality-of-life questionnaires for use in oncology
clinical practice. Support Care Cancer 2007 Sep; 15(9): 1075—85.
8. Snyder CF, Herman JM, White SM, et al. When using patient-reported outcomes in clinical practice, themeasure matters: a randomized controlled trial. Journal of Oncology Practice 2014; 10: e299—306.
9. Talley NJ, Wiklund I. Patient reported outcomes in gastroesophageal reflux disease: An overview ofavailable measures. Qual Life Res 2005 Feb; 14(1): 21—33.
10. Walter OB, Becker J, Bjorner JB, Fliege H, Klapp BF, Rose M. Development and evaluation of a computer
adaptive test for ‘Anxiety’ (Anxiety-CAT). Qual Life Res 2007; 16 Suppl 1: 143—155.
11. Zimmerman M, Ruggero CJ, Chelminski I, et al. Developing brief scales for use in clinical practice: thereliability and validity of single-item self-report measures of depression symptom severity, psychosocial
impairment due to depression, and quality of life. J Clin Psychiatry 2006 Oct; 67(10): 1536—41.

13

How often should patients complete questionnaires? Should it be tied tovisits or a way to follow patients between visits?
It is also important to consider how frequently patients will complete PROs. Options range from one-time only
to frequent completion, with assessments tied to visits or a way to monitor patients between visits. These
options are described in more detail below.
A. ADMINISTRATION AT VISIT: ONE-TIME OCCURRENCE
Resources Needed: Time for patients to get familiar with measures and means for completion.
 Time for clinicians to train on the PRO’s use and interpretation.
 Resources for data management and use, timely review of data, and responses to patientneeds identified through PRO measurement.
Advantages:
 Can screen for problems and unexpected conditions.
 Provides information about what is important to the patient with regards to his/her
condition and treatment.
Disadvantages:
 Does not enable clinicians to monitor changes.
 Assumes that decision-making about patient’s treatment can occur during a single
consultation.
B. ADMINISTRATION AT VISIT: MULTIPLE VISITS
Resources Needed: Generally requires same resources as one-time administration, but incrementally more ofthem.
Advantages:
 Allows seeing the trajectory of the disease and its treatment.
 Helps patients understand and monitor changes in their PROs, promoting their involvement
and confidence and patient-centered care.
 Helps clinicians understand the PROs.
Disadvantages:
 Incrementally higher costs to assess PROs at multiple visits compared to single visits.
 Requires additional workforce capacity.
 Involves increased administrative complexity.

14

C. ADMINISTRATION BETWEEN VISITS
Resources Needed: Patients need access to a website or other method to complete PROs from home.
 System requires built-in alerts so that clinicians are sent an email and/or page for issues
requiring immediate attention.
 Alerts should direct clinicians to resources and personnel (e.g., nurses) available to help
address problems.
Advantages:
 Improved accessibility to health care.
 Potential of improving patient care between visits.
 Allows seeing the trajectory of the disease and its treatment.
 Helps patients understand and monitor changes in their PROs, promoting their involvement
and confidence and patient-centered care.
 Helps clinicians understand the PROs.
Disadvantages:
 Increased burden on clinicians and healthcare infrastructure.
 Extra resources needed to ensure that critical problems reported by patients are addressedon time.
 Alerts may be burdensome.
D. DETERMINING FREQUENCY OF ADMINISTRATION
Resources Needed: Generally requires more resources for more frequent administration.
Advantages:
 More frequent assessment can provide a more complete picture for patients who are very
symptomatic and/or in active treatment.
 Less frequent assessment is less burdensome and may be appropriate for generally healthy
patients.
Disadvantages:
 More frequent assessments can be burdensome.
 Less frequent assessment may miss key changes in patients’ outcomes.

KEY REFERENCES
1. A Guide to PROMs Methodology. Available at: http://www.hscic.gov.uk/media/1537/A-Guide-to-PROMsMethodology/pdf/PROMS_Guide_v5.pdf
2. Ayers DC, Zheng H, Franklin PD. Integrating patient-reported outcomes into orthopaedic clinical practice:
proof of concept from FORCE-TJR. Clin Orthop Relat Res 2013; 471(11): 3419—25.

15

3. Bush N, Donaldson G, Moinpour C, et al. Development, feasibility and compliance of a web-based systemfor very frequent QOL and symptom home self-assessment after hematopoietic stem cell transplantation.
Qual Life Res 2005; 14(1): 77—93.
4. Donaldson MS. Taking PROs and patient-centered care seriously: incremental and disruptive ideas forincorporating PROs in oncology practice. Qual Life Res 2008; 17: 1323—1330.
5. Donaldson MS. Use of patient-reported outcomes in clinical oncology practice. A nonvisit approach topatient care based on the IOM report. J Ambulatory Care Manage 2007; 30 (4): 302—307.
6. Engelen V, Haverman L, Koopman H, et al. Development and implementation of a patient reported
outcome intervention (QLIC-ON PROfile) in clinical paediatric oncology practice. Patient Educ Couns 2010;
81(2): 235—244.
7. Frost MH, Bonomi AE, Cappelleri JC, et al. Applying quality-of-life data formally and systematically into
clinical practice. Mayo Clin Proc 2007; 82 (10): 1214—1228.
8. Fung C, Hays RD. Prospects and challenges in using patient-reported outcomes in clinical practice. Qual LifeRes 2008; 17: 1297—1302.
9. Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measures in routine clinical care: lackof impact or lack of theory? Social Science & Medicine 2005; 60: 833—843.
10. Haverman, L, Engelen, V, Van Rossum, MA, Heymans, HS, & Grootenhuis, MA. Monitoring health-related
quality of life in paediatric practice: development of an innovative web-based application. BMC Pediatrics,
2011; 11: 3—10.
11. Haverman, L, van Rossum, MA, van, VM, van den Berg, JM, Dolman, KM, Swart, J et al. Effectiveness of a
web-based application to monitor health-related quality of life. Pediatrics, 2013; 131(2), e533-e543.
12. Lohr KN, Zebrack BJ. Using patient-reported outcomes in clinical practice: challenges and opportunities.
Qual of Life Res 2009; 18: 99—107.
13. Rose M, Bezjak A. Logistics of collecting patient-reported outcomes (PRO) in clinical practice: an overview
and practical examples. Qual Life Res 2009; 18: 125—136.
14. Snyder CF, Jensen R, Courtin O, Wu AW. PatientViewpoint: a website for patient-reported outcomes
assessment. Qual Life Res 2009; 18; 793—800.

16

How will the PROs be administered and scored?
Various options for mode of administration and mode of data capture are described below.
A. SELF-ADMINISTERED: IN CLINIC
Resources Needed: Personnel to supervise and assist, where necessary.
 Space.
 Administrative personnel for data entry.
Advantages:
 Low-technology requirements.
 Can be implemented in any clinical setting.
 Relatively low cost.
Disadvantages:
 Problem with low literacy patients and those with visual handicap.
 Potentially difficult with other special populations (e.g., very young, very old, severely
symptomatic).
 Potentially higher rate of missing data compared to interviewer administeredquestionnaires.
B. INTERVIEW ADMINISTERED: IN-CLINIC
Resources Needed: Skilled interviewer.
 Space.
 Administrative personnel for data entry.
Advantages:
 More personal.
 Facilitates more in-depth questioning.
 Largely circumvents literacy problem and/or visual handicap.
Disadvantages:
 Relatively expensive.
 May create problems with social desirability.
 Increased costs to dedicate staff time to administer PRO measure.
C. COMPUTER-ASSISTED: IN-CLINIC (including portable devices)
Resources Needed: Personnel to supervise and assist, where necessary.
 Software to collect and report the PRO data.
17

Advantages:
 Efficient data capture with simultaneous data entry.
Disadvantages:
 Potential problems finding space/providing privacy.
 Upfront costs to develop (or purchase) the PRO system and ongoing costs to maintain it.
 Potential software problems.
D. SELF-ADMINISTERED VIA MAIL
Resources Needed: Personnel to manage mailing.
 Administrative personnel for data entry.
Advantages:
 Low-technology requirements.
 Potentially simpler logistics than in-clinic administration.
 Relatively low cost.
Disadvantages:
 Potentially high non-response rate.
 Cannot ensure patient completes questionnaire alone.
 More difficult to respond immediately if the patient reports severe symptoms and moredifficult to time the assessment close to the clinical visit.
 Other limitations similar to Self-Administered In-Clinic.
E. TELEPHONE ADMINISTRATION: LIVE INTERVIEW
Resources Needed: Skilled interviewer.
 Administrative personnel for data entry.
Advantages:
 More personal.
 More convenient for patient.
 Largely circumvents literacy problem and/or visual handicap.
Disadvantages:
 Lack of visual cues as compared to face-to-face.
 Relatively expensive.
 Potential problem with social desirability.
 Some topics may be more difficult to address.

18

F. TELEPHONE INTERVIEW: VOICE ACTIVATED
Resources Needed: Administrative personnel to oversee data collection.
 A validated and efficient interactive voice response (IVR) system.
Advantages:
 Relatively low cost due to automation.
Disadvantages:
 Automated system may be off-putting to patients.
 Upfront costs to develop (or purchase) the IVR system and ongoing costs to maintain it.
 Requires process to track and respond to any urgent problem reported by patients.
 Other disadvantages similar to Live Telephone Interview, plus impersonal nature.
G. WEB-BASED (including portable devices)
Resources Needed:
 Systems management personnel.
 Software to collect and report the PRO data.
 Training patients.
Advantages:
 Efficient data capture with simultaneous data entry.
 Convenient for patient.
 Flexible timing for data collection.
Disadvantages:
 Difficult to ensure privacy.
 Upfront costs to develop (or purchase) the PRO system and ongoing costs to maintain it.
 Potential software problems.
KEY REFERENCES
1. Bjorner JB, Rose M, Gandek B, Stone AA, Junghaenel DU, Ware JE Jr. Method of administration of PROMIS
scales did not significantly impact score level, reliability, or validity. J Clin Epidemiol 2014; 67(1): 108—113.
2. Chambers LW, Haight M, Norman G, MacDonald L. Sensitivity to change and the effect of mode ofadministration on health status measurement. Med Care 1987; 25(6): 470—80.
3. Cheung YB, Goh C, Thumboo J, Khoo KS, Wee J. Quality of life scores differed according to mode ofadministration in a review of three major oncology questionnaires. J Clin Epidemiol 2006; 59(2): 185—91.
4. Coons SJ, Gwaltney CJ, Hays RD, et al. Recommendations on evidence needed to support measurement
equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO
Good Research Practices Task Force report. Value Health 2009; 12(4): 419—429.
5. Gundy CM, Aaronson NK. Effects of mode of administration (MOA) on the measurement properties of the
EORTC QLQ-C30: a randomized study. Health Qual Life Outcomes 2010; 8: 35.
6. Hanmer J, Hays RD, Fryback DG. Mode of administration is important in US national estimates of healthrelated quality of life. Med Care 2007; 45(12): 1171—1179.
19

7. Hepner KA, Brown JA, Hays RD. Comparison of mail and telephone in assessing patient experiences inreceiving care from medical group practices. Eval Health Prof 2005; 28(4): 377—389.
8. Hess R, Santucci A, McTigue K, Fischer G, Kapoor W. Patient difficulty using tablet computers to screen in
primary care. J Gen Intern Med 2008; 23(4): 476—480.
9. Hood K, Robling M, Ingledew D, et al. Mode of data elicitation, acquisition and response to surveys: asystematic review. Health Technol Assess 2012; 16(27): 1—162.
10. Jörngården A, Wettergen L, von Essen L. Measuring health-related quality of life in adolescents and young
adults: Swedish normative data for the SF-36 and the HADS, and the influence of age, gender, and methodof administration. Health Qual Life Outcomes 2006; 4: 91.
11. Norman R, King MT, Clarke D, Viney R, Cronin P, Street D. Does mode of administration matter?
Comparison of online and face-to-face administration of a time trade-off task. Qual Life Res 2010; 19(4):499—508.
12. Puhan MA, Ahuja A, van Natta ML, Ackatz LE, Meinert C. Interviewer versus self-administered healthrelated quality of life questionnaires - Does it matter? Health Qual Life Outcomes 2011; 9: 30.
13. Raat H, Mangunkusumo RT, Landgraf JM, Kloek G, Brug J. Feasibility, reliability, and validity of adolescenthealth status measurement by the Child Health Questionnaire Child Form (CHQ-CF): internet
administration compared with the standard paper version. Qual Life Res 2007; 16(4): 675—685.
14. Ring AE, Cheong KA, Watkins CL, Meddis D, Cella D, Harper PG. A randomized study of electronic diaryversus paper and pencil collection of patient-reported outcomes in patients with non-small cell lungcancer. Patient 2008; 1(2): 105—113.
15. Swartz RJ, de Moor C, Cook KF, et al. Mode effects in the Center for Epidemiologic Studies Depression(CES-D) scale: personal digital assistant vs. paper and pencil administration. Qual Life Res 2007; 16(5):803—813.
16. Weinberger M, Oddone EZ, Samsa GP, Landsman PB. Are health-related quality-of-life measures affectedby the mode of administration? J Clin Epidemiol 1996; 49(2): 135—140.

20

What tools are available to aid in score interpretation and how will scores
requiring follow-up be determined?
Tools to aid the interpretation of PROs vary depending on whether the patient’s current score only has been
fed back to the clinicians, or whether the clinician is presented with the change in the patient’s score.
Although different options are discussed separately below for clarity, studies in this area have tended to utilize
a combination of different tools to facilitate interpretation. Further, it is important to note that PROs are justone piece of information to help clinicians identify problems and enhance the care of the patients.
Considering the PRO data in the context of the other clinical information about the patient, as well as theclinician’s assessment, is helpful.
A. GENERAL WRITTEN GUIDELINES
Resources Needed: General written guidelines of score meaning (e.g., “higher scores mean better functioning”).
Advantages:
 Simple to read.
 Provides general indication of the meaning of the scores.
Disadvantages:
 Provides no information about the clinical importance or importance to the patient.
B. CUT-OFF SCORE FOR “CASENESS” OR FOR LEVELS OF SEVERITY
Resources Needed: Information on what to use as the cut-off for “caseness” or for previously validated
categories (e.g., no disability, moderate disability, severe disability).
Advantages:
 Simple and easy to apply (i.e., is the patient’s score above or below the threshold; or withina certain category).
Disadvantages:
 Assumes these cut-offs and/or categories have been established for the measure inquestion.
 More likely to have established cut-offs and/or categories for anxiety and depression,
versus quality of life measures.
 Usefulness depends on the sensitivity and specificity of cut-offs; likely to be
misclassification; predictive value depends on prevalence of the condition in the population
being screened.

21

C. REFERENCE SCORES FROM RESEARCH STUDIES WITH SIMILAR PATIENTS
Resources Needed: Information on mean endpoint scores from clinical trials/systematic reviews of patientswith the same condition undergoing the same treatment.
Advantages:
 Enables clinicians to compare their patients with patients in trials undergoing same
treatment.
Disadvantages:
 May not be available for all instruments.
 Patients in trials different from patients that clinicians may see in clinic.
 Just because it is experienced by other patients does not mean it is not a problem for thisparticular patient – “to be expected” is not the same as “not problematic.”
 Significantly larger error of measurement in individual, compared with group, PRO scorescan make comparison with benchmarks from group data problematic.
D. REFERENCE SCORES FROM THE GENERAL POPULATION WITH SAME CONDITION
Resources Needed: Information on mean endpoint scores from the populations in the community with thesame condition.
Advantages:
 Enables the clinician to compare patient’s current score with the average of the wider
population with the same condition.
 Approach can be used with both generic and disease-specific questionnaires.
Disadvantages:
 May not be available for all instruments; needs to have been administered to a large
number of patients to generate valid norms.
 Patient in front of the clinician may not be similar to the population (e.g., comorbid
conditions and preferences).
 Does not provide any information about whether the patients themselves see that area asproblematic – only that their score is higher/lower or the same as the reference population.
 Significantly larger error of measurement in individual, compared with group, PRO scorescan make comparison with benchmarks from group data problematic.
E. REFERENCE SCORES FROM THE HEALTHY POPULATION
Resources Needed: Information on mean endpoint scores from healthy populations.
Advantages:
 Enables the clinician to compare patient’s current score with the average of the healthy
population.
22Can be used to determine distance from “full health.”

Disadvantages:
 Only useful for generic instruments.
 Probably only useful for conditions where there is an expectation that the patient may
return to full health.
 Significantly larger error of measurement in individual, compared with group, PRO scorescan make comparison with benchmarks from group data problematic.
F. STRUCTURED INTERVIEWS TO CLARIFY SCORES
Resources Needed: Personnel to review patient’s scores with the patient to clarify and elaborate on problems
indicated by the PROs.
Advantages:
 Can provide further information on what the patient’s problems are and why.
Disadvantages:
 Resource intensive.
G. LINKING SCORES TO MANAGEMENT GUIDELINES
Resources Needed: Recommendations for how clinicians could respond to issues identified by the PROs.
 Access to published and accepted clinical guidelines/recommendations to tie the PRO data
to effective care.
Advantages:
 May increase the ability of PRO results to affect patient care and outcomes.
Disadvantages:
 May be perceived as challenging the clinician’s expertise and autonomy.
H. SIMPLE COMPARISON WITH PATIENT’S PREVIOUS SCORES
Resources Needed: Patient’s current and previous scores.
Advantages:
 Easy for the clinician to assess.
Disadvantages:
 Provides no information about the importance of the change either clinically or to thepatient.

23

I. MINIMALLY CLINICALLY IMPORTANT DIFFERENCE: DISTRIBUTION-BASED METHODS
Resources Needed: Scores over time presented as effect sizes or standard error of measurement, along withwritten interpretation of what constitutes small, medium, and large changes.
Advantages:
 Enables the clinician to compare patient’s current score with the average of the wider
population with the same condition.
 Standard error of measurement remains relatively constant across ability ranges within the
population.
Disadvantages:
 May not be intuitively meaningful to clinicians.
 Criteria for what constitutes small, medium, and large changes criticized as arbitrary.
 Meaning is improved if they are combined with anchor-based measures of change.
 Minimally important differences based on the standard error of measurement is highly
dependent on scale reliability, which is required to be at least 0.9 – few instruments may
meet this criterion.
 Applying minimally important differences based on groups of patients to individual patients
may be problematic due to the larger measurement error in individual measurements.
J. MINIMALLY CLINICALLY IMPORTANT DIFFERENCE: ANCHOR-BASED METHODS
Resources Needed: Data on changes in scores related to external anchors (e.g., patient global ratings of change,clinician rating of change, clinical measures).
Advantages:
 Simple for the clinician to interpret, as it involves simply comparing the change in score to
the point difference determined to be clinically important based on the anchor.
 Can be improved by triangulating patient and clinician ratings of important change.
 Applying minimally important differences based on groups of patients to individual patients
may be problematic due to the larger measurement error in individual measurements.
Disadvantages:
 Global transition questions have been criticized, as patient reports of their previous healthstatus may be influenced by their current health status.
KEY REFERENCES
1. Beaton DE, van Eerd D, Smith P, et al. Minimal change is sensitive, less specific to recovery: a diagnostic
testing approach to interpretability. J Clin Epidemiol 2011; 64(5): 487—496.
2. Bowling A, Bond M, Jenkinson C, Lamping DL. Short Form 36 (SF-36) Health Survey questionnaire: which
normative data should be used? Comparisons between the norms provided by the Omnibus Survey in
Britain, the Health Survey for England and the Oxford Healthy Life Survey. J Public Health Med 1999; 21(3):
255—270.
24

3. de Wit M, Delemarre-van de Waal HA, Bokma JA, et al. Monitoring and discussing health related quality oflife in adolescents with type 1 diabetes improves psychosocial well being. Diabetes Care 2008; 31(8):1521—1526.
4. Detmar SB, Muller MJ, Schornagel JH, Wever LD, Aaronson NK. Health related quality of life assessments
and patient-physician communication. JAMA 2002; 288(23): 3027—3034.
5. Donaldson G. Patient reported outcomes and the mandate for measurement. Qual Life Res 2008; 17:1303– 1313.
6. Dowrick C. Does testing for depression influence diagnosis or management by general practitioners? FamPract 1995; 12(4): 461—465.
7. Fayers PM, Hays RD. Don’t middle your MIDs: regression to the mean shrinks estimates of minimally
important differences. Qual Life Res 2014: 23 (1): 1—4.
8. Gilbody S, Sheldon T, Wessely S. Should we screen for depression? BMJ 2006; 332(7548): 1027—1030.
9. Greenhalgh J, Flynn R, Long AF, Tyson S. Tacit and encoded knowledge in the use of standardised outcome
measures in multidisciplinary team decision making: a case study of in-patient neurorehabilitation. Soc Sci
Med 2008; 67: 183—194.
10. Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of health in routine
practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract 1999;
5(4): 401—416.
11. Gutteling JJ, Darlington AS, Janssen HL, Duivenvoorden HJ, Busschbach JJ, deMan RA. Effectiveness ofhealth related quality of life measurement in clinical practice: a prospective, randomised controlled trial inpatients with chronic liver disease and their physicians. Qual Life Res 2008; 17: 195—205.
12. Hays RD, Brodsky M, Johnston MF, Spritzer KL, Hui K-K. Evaluating the statistical significance of healthrelated quality-of-life change in individual patients. Eval Health Prof 2005; 28(2): 160—171.
13. Hilarius DL, Kloeg PH, Gundy CM, Aaronson NK. Use of health related quality of life assessments in dailyclinical oncology nursing practice: A community hospital based intervention study. Cancer 2008; 113(3):628—637.
14. Hildon Z, Allwood D, Black N. Impact of format and content of visual display of data on comprehension,
choice and preference: a systematic review. Int J Qual Health Care 2012; 24: 55—64.
15. Kazis LE, Callahan LF, Meenan RF, Pincus T. Health status reports in the care of patients with rheumatoid
arthritis. J Clin Epidemiol 1990; 43(11): 1243—1253.
16. Kemmler G, Zabernigg A, Gattringer K, et al. A new approach to combining clinical relevance and statisticalsignificance for evaluation of quality of life changes in the individual patient. J Clin Epidemiol 2010; 63:
171—179.
17. Magruder-Habib K, Zung WWK, Feussner JR. Improving physicians recognition and treatment of depression
in general medical care. Results from a randomized clinical trial. Med Care 1990; 28: 239—250.
18. Mathias SD, Fifer SK, Mazonson PD, Lubeck DP, Buesching DP, Patrick DL. Necessary but not sufficient: theeffect of screening and feedback on outcomes of primary care patients with untreated anxiety. J Gen
Intern Med 1994; 9(11): 606—615.
19. Meyer KB, Espindle DM, DeGiacomo JM, Jenuleson CS, Kurtin PS, Davies AR. Monitoring dialysis patients'
health status. Am J Kidney Dis 1994; 24(2): 267—279.
20. Reifler DR, Kessler HS, Bernhard EJ, Leon AC, Martin GJ. Impact of screening for mental health concerns on
health service utilization and functional status in primary care patients. Arch Intern Med 1996; 156(22):2593—2599.
21. Rose MS, Koshman ML, Spreng S, Sheldon R. Statistical issues encountered in the comparison of healthrelated quality of life in diseased patients to published general population norms: problems and solutions.
J Clin Epidemiol 1999; 52(5): 405—412.

25

22. Rosenbloom SK, Victorson DE, Hahn EA, Peterman AH, Cella D. Assessment is not enough: A randomized
controlled trial of the effects of HRQoL assessment on quality of life and satisfaction in oncology clinicalpractice. Psychooncology 2007; 16: 1069—1079.
23. Rubenstein LV, McCoy JM, Cope DW, et al. Improving patient quality of life with feedback to physicians
about functional status. J Gen Intern Med 1995; 10(11): 607—614.
24. Santana MJ, Feeny D, Johnson JA, et al. Assessing the use of health related quality of life measures in theroutine clinical care of lung transplant patients. Qual Life Res 2010; 19: 371—379.
25. Snyder CF, Blackford AL, Brahmer JR, et al. Needs assessments can identify scores on HRQoL
questionnaires that represent problems for patients: an illustration with the Supportive Care Needs Survey
and the QLQ-C30. Qual Life Res 2010; 19(6): 847—845.
26. Snyder CF, Blackford AL, Okuyama T, et al. Using the EORTC QLQ-C30 in clinical practice for patientmanagement: identifying scores requiring a clinician's attention. Qual Life Res 2013; 22: 2685—2691.
27. Street RL, Gold WR, McDowell T. Using health status surveys in medical consultations. Med Care 1994;
32(7): 732—744.
28. Velikova G, Booth L, Smith AB, et al. Measuring quality of life in routine oncology practice improvescommunication and patient well being- a randomized controlled trial. J Clin Oncol 2004; 22(4): 714—724.
29. Wilson TR, Birks Y, Alexander DJ. Pitfalls in the interpretation of standardised quality of life instruments forindividual patients? A qualitative study in colorectal cancer. Qual Life Res 2013; 22: 1879–1888.
30. Wyrwich KW, Norquist JM, Lenderking WR, Acaster S, Industry Advisory Committee of International
Society for Quality of Life Research (ISOQOL). Methods for interpreting change over time in patientreported outcome measures. Qual Life Res 2013; 22: 475–483.

26

When will results be presented?
Another important consideration is when and how to present the results and discuss them with patients.
Options include before, during, or after a patient visit, as described below.
A. AT TIME OF VISIT
Resources Needed: Method to collect data at time of or prior to visit.
 Method to score PROs prior to or at time of visit.
 Method to display data.
 Clinicians willing to discuss pertinent issues identified by patients.
Advantages:
 Information available at time of clinical encounter.
 Alerts clinicians to area of patient concern.
 Enhances patient-clinician communication.
 Helps to clarify priorities for care.
Disadvantages:
 Possibility of omission of discussion of certain issues by clinicians due to lack of time,
expertise, patient/clinician unwillingness to discuss, etc.
 Resources for scoring PROs in real time may be lacking without using e-PROs.
 Start-up costs of e-PROs may be prohibitive.
B. PRIOR TO VISIT
Resources Needed: Method to gather PROs from patient outside of visit.
 Method of communicating results from patient to clinician.
 Method to ensure results are also available for clinical encounter.
Advantages:
 Time to score instruments if not e-PRO.
 Clinicians have time to prepare for discussion.
Disadvantages:
 Clinician must respond to results prior to clinical visit.
 Need for patient to provide information outside of clinical visit.
C. AFTER VISIT
Resources Needed: Method to gather PROs at the time of visit.
27

Advantages:
 PROs can be gathered at time of visit but scored and presented later.
 May work better within office work flow.
Disadvantages:
 Clinician must react to results after visit decreasing usefulness.
 If patients are to receive PRO results, ensuring that they get them with appropriate
interpretation may be challenging.
KEY REFERENCES
1. Brundage M, Leis A, Bezjak A, et al. Cancer patients' preferences for communicating clinical trial quality oflife information: a qualitative study. Qual Life Res 2003; 12(4): 395—404.
2. Detmar SB, Muller MJ, Schornagel JH, Wever LD, Aaronson NK. Health related quality of life assessments
and patient-physician communication. JAMA 2002; 288(23): 3027—3034.
3. Frost MH, Bonomi AE, Cappelleri JC, et al. Applying quality-of-life data formally and systematically into
clinical practice. Mayo Clin Proc 2007; 82(10): 1214—1228.
4. Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why? Qual LifeRes 2009; 18(1): 115—123.
5. Higginson IJ, Carr AJ. Measuring quality of life: Using quality of life measures in the clinical setting. BMJ
2001; 322(7297): 1297—1300.
6. Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring patient-reported outcomes in clinicalpractice: a systematic review of the literature. Qual Life Res 2008; 17(2): 179—193.
7. Velikova G, Booth L, Smith AB, et al. Measuring quality of life in routine oncology practice improvescommunication and patient well being- a randomized controlled trial. J Clin Oncol 2004; 22(4): 714—724.
8. Wasson JH, Stukel TA, Weiss JE, Hays RD, Jette AM, Nelson EC. A randomized trial of the use of patient selfassessment data to improve community practices. Eff Clin Pract 1999; 2(1): 1—10.
9. Wright EP, Selby PJ, Crawford M, et al. Feasibility and compliance of automated measurement of quality oflife in oncology practice. J Clin Oncol 2003; 21(2): 374—382.

28

Where will results be presented?
It should also be determined whether results will be presented within or outside of the clinical workflow.
A. WITHIN CLINICAL WORKFLOW
Resources Needed: If clinical workflow is paper-based: paper-based report that is presented to clinical staff
within usual workflow.
 If clinical workflow is electronic: resources to integrate electronic PROs into electronic
chart, or for non-electronic PROs, resources to add PROs to the electronic chart.
Advantages:
 Places PROs where clinician is prepared to receive them.
 Allows integration of PRO data with other clinical data.
Disadvantages:
 Additional information for clinicians to review.
 Electronic integration can be costly and may not be supported by electronic medical record
vendor, depending on the system.
B. OUTSIDE OF USUAL CLINICAL WORKFLOW
Resources Needed: Clinical staff to track change in PRO status and address ongoing PRO issues which may
require intervention beyond visit (e.g., phone follow-up, or self-directed supplemental
information via web links).
 Electronic or paper-based system to present PROs to clinician that is reliable outside ofusual workflow.
Advantages:
 PRO presentation can be customized to clinician or group needs.
 More frequent assessment beyond episodic office visit which may enhance resolution ofproblems.
Disadvantages:
 Data are outside of usual workflow.
 Documentation and retrieval may not be complete.
 Potentially time and effort consuming (e.g., nursing resources).
KEY REFERENCES
1. Ahles TA, Wasson JH, Seville JL, et al. A controlled trial of methods for managing pain in primary carepatients with or without co-occurring psychosocial problems. Ann Fam Med 2006; 4(4): 341—350.
29

2. Donaldson M. Taking PROs and patient-centered care seriously: incremental and disruptive ideas forincorporating PROs in oncology practice. Qual Life Res 2008; 17(10): 1323—1330.

30

How will results be presented?
There are various ways to present the PRO score results, as described below.
A. NUMERIC PRESENTATION
Resources Needed: None beyond resources to collect and score the PROs.
Advantages:
 Does not require data manipulation.
 Easy to integrate into standard workflow.
 Can be presented in the context of normative data ranges.
Disadvantages:
 Difficult to interpret as there may be no context for an individual without sophisticatedknowledge of the instrument.
B. GRAPHICAL PRESENTATION
Resources Needed: Requires manipulation of the data to place them in a graphical context.
Advantages:
 Likely easier to interpret than strict numerical presentation.
 Allows visual presentation in the context of normative data.
 May improve usability by patients and clinicians.
Disadvantages:
 More complex presentation requires data manipulation.
 Graphical representation may be more difficult to integrate into standard workflow (e.g.,
electronic medical record) than strict numeric.
C. PRESENTATION OF TRENDS OVER TIME
Resources Needed: Requires ability to recall prior results and place them in the context of change.
 Consider presenting text with graphical information.
Advantages:
 Provides context for individuals over time.
 Sophisticated presentation could allow receiver of information to customize presentation
format.
31

Line graphs of means without extraneous information (e.g., error bars) may be clearer forpatients.

Disadvantages:
 Systems that recall and integrate prior data are needed.
 More complex presentation requires data manipulation for change in the context of prior
score and normative data.
 As with graphical representation, it may be more difficult to integrate into standard
workflow (e.g., electronic medical record) than strict numeric.
KEY REFERENCES
1. Brundage M, Feldman-Stewart D, Leis A, et al. Communicating quality of life information to cancer
patients: a study of six presentation formats. J Clin Oncol 2005; 23(28): 6949—6956.
2. Bunge M, Muhlhauser I, Steckelberg A. What constitutes evidence-based patient information? Overview of
discussed criteria. Patient Educ Couns 2010; 78: 316—328.
3. Fagerlin A, Wang C, Ubel PA. Reducing the influence of anecdotal reasoning on people's health caredecisions: is a picture worth a thousand statistics? Med Decis Making 2005; 25(4): 398—405.
4. Gaissmaier W, Wegwarth O, Skopec D, et al. Numbers can be worth a thousand pictures: individual
differences in understanding graphical and numerical representations of health-related information.
Health Psychol 2012; 31: 286—296.
5. Rothman RL, Montori VM, Cherrington A, Pignone MP. Perspective: The role of numeracy in health care. J
Health Commun 2008; 13(6): 583—595.
6. Schwartz LM, Woloshin S, Black WC, Welch HG. The role of numeracy in understanding the benefit ofscreening mammography. Ann Intern Med 1997; 127(11): 966–972.
7. Wu AW, Kharrazi H, Boulware LE, Snyder CF. Measure once, cut twice--adding patient-reported outcome
measures to the electronic health record for comparative effectiveness research. J Clin Epidemiol 2013; 66:
S12—20.

32

Who will receive score reports?
It is also important to consider who should receive the score reports, including determining which clinicians
and whether the patient should see the results, as well.
A. MEMBER OF THE HEALTH CARE TEAM (physician, nurse, other ancillary personnel)
Resources Needed: Individual health care team member responsible for reacting to PROs.
Advantages:
 Customizable to practice patterns.
 Personnel other than physician may be better able to respond to PRO.
 PRO data can be treated like other clinical data such as laboratory tests or radiology reports.
Disadvantages:
 Needs to be customized to practice.
 Necessitates systems based practice redesign.
B. FEEDBACK TO PATIENTResources Needed: Method to get patients information by the time of clinician discussion, either paper at timeof visit or electronic communication prior to appointment.
 Follow up from the clinician after the appointment.
Advantages:
 Patient is an active member of the care team and can participate in decision-making.
 Consistent with current movement towards care transparency.
Disadvantages:
 If results are presented without context or appropriate interpretation, may result in
confused patient.
 Increases need for practices to ensure that data are collected and disseminated to patients
through a variety of formats.
KEY REFERENCES
1. Brundage M, Leis A, Bezjak A, et al. Cancer patients' preferences for communicating clinical trial quality oflife information: a qualitative study. Qual Life Res 2003; 12(4): 395—404.
2. Delbanco T, Walker J, Darer JD, et al. Open notes: doctors and patients signing on. Ann Intern Med 2010;
153: 121—125.
33

3. Detmar SB, Muller MJ, Schornagel JH, Wever LD, Aaronson NK. Health related quality of life assessments
and patient-physician communication. JAMA 2002; 288(23): 3027—3034.
4. Espallargues M, Valderas JM, Alonso J. Provision of feedback on perceived health status to health care
professionals: a systematic review of its impact. Med Care 2000; 38(2): 175—186.
5. Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of health in routine
practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract 1999;
5(4): 401—416.
6. Guyatt GH, Ferrans CE, Halyard MY, et al. Exploration of the value of health-related quality-of-life
information from clinical research and into clinical practice. Mayo Clin Proc 2007; 82(10): 1229—1239.
7. Hess R, Tindle H, Conroy MB, et al. A randomized controlled pilot trial of the functional assessment
screening tablet to engage patients at the point of care. J Gen Intern Med 2014 Aug 8 [Epub ahead of print].
8. Hughes EF, Wu AW, Carducci MA, Snyder C. What can I do? Recommendations for responding to issues
identified by patient-reported outcomes assessments used in clinical practice. J Support Oncol 2012; 10:
143—148.
9. MacArthur C, Winter HR, Bick DE, et al. Effects of redesigned community postnatal care on womens' health4 months after birth: a cluster randomised controlled trial. Lancet 2002; 359(9304): 378—385.
10. Marshall S, Haywood K, Fitzpatrick R. Impact of patient-reported outcome measures on routine practice: astructured review. J Eval Clin Pract 2006; 12(5): 559—568.
11. Wagner EH. Chronic disease management: what will it take to improve care for chronic illness? Eff Clin
Pract 1998; 1(1): 2—4.

34

What will be done to respond to issues identified through the PROs?
Another important question is how to respond to issues identified through the PRO assessments. There are avariety of approaches that can be taken to address this issue, as described below.
A. UTILIZATION OF DISEASE MANAGEMENT PATHWAYS
Resources Needed: Useful disease management pathways applicable to the PRO.
Advantages:
 Uniform method of addressing issue.
Disadvantages:
 Possibility of lack of utilization of intervention as recommended.
 Pathways may not exist for all applications, resulting in burden of creating and validating
pathway.
B. FURTHER EXPLORATION OF PRO ISSUES IDENTIFIED WITH THE PATIENT TO GAIN FULLUNDERSTANDING
Resources Needed: Time for clinicians to explore issues in greater depth.
Advantages:
 Does not rely on PRO data as full extent of the issues.
 Allows delving deeper into issues.
Disadvantages:
 Time consuming.
C. UTILIZATION OF MULTIDISCIPLINARY TEAM MEMBER EXPERTISE TO ADDRESS ISSUES
Resources Needed: Health professionals from different disciplines.
Advantages:
 Utilization of varied skill sets.
Disadvantages:
 Skills may not be readily available for referrals in all settings.

35

KEY REFERENCES
1. Ahmed S, Berzon RA, Revicki DA, et al. The use of patient-reported outcomes (PRO) within comparative
effectiveness research: implications for clinical practice and health care policy. Med Care 2012; 50(12):1060—1070.
2. Bainbridge D, Seow H, Sussman J, et al. Multidisciplinary health care professionals' perceptions of the use
and utility of a symptom assessment system for oncology patients. J Oncol Pract 2011; 7(1): 19—23.
3. Basch E, Abernethy AP. Supporting clinical practice decisions with real-time patient-reported outcomes. J
Clin Oncol 2011; 29: 954—956.
4. Basch E, Abernethy AP. Commentary: encouraging clinicians to incorporate longitudinal patient-reported
symptoms in routine clinical practice. J Oncol Pract 2011; 7(1): 23—25.
5. Bennett AV, Jensen RE, Basch E. Electronic patient-reported outcome systems in oncology clinical practice.
CA Cancer J Clin 2012; 62(5): 337—347.
6. Berry DL, Blumenstein BA, Halpenny B, et al. Enhancing patient-provider communication with theelectronic self-report assessment for cancer: a randomized trial. J Clin Oncol 2011; 29(8): 1029—1035.
7. Berry DL, Hong F, Halpenny B, et al. Electronic self-report assessment for cancer and self-care support:
results of a multicenter randomized trial. J Clin Oncol 2014; 32(3): 199—205.
8. Callahan MB. Using quality of life measurement to enhance interdisciplinary collaboration. Adv Ren
Replace Ther 2001; 8(2): 148—151.
9. Cleeland CS, Wang XS, Shi Q, et al. Automated symptom alerts reduce postoperative symptom severity
after cancer surgery: a randomized controlled clinical trial. J Clin Oncol 2011; 29(8): 994—1000.
10. Eton, DT, Beebe TJ, Hagen PT, et al. Harmonizing and consolidating the measurement of patient reported
information at health care institutions; a position statement of Mayo Clinic. Patient Relat Outcome Meas
2014; 5: 7—15.
11. Lohr KN, Zebrack BJ. Using patient-reported outcomes in clinical practice: challenges and opportunities.
Qual Life Res 2009; 18(1): 99—107.
12. Mooney KH, Beck SL, Friedman RH, Farzanfar R, Wong B. Automated monitoring of symptoms during
ambulatory chemotherapy and oncology providers' use of the information: a randomized controlled
clinical trial. Support Care Cancer 2014 Apr 1 [Epub ahead of print].
13. Verhoef J, Toussaint PJ, Putter H, Zwetsloot-Schonk JH, Vliet Vlieland TP. The impact of introducing an ICFbased rehabilitation tool on staff satisfaction with multidisciplinary team care in rheumatology: an
exploratory study. Clin Rehabil 2008; 22(2): 23—37.
14. Verhoef J, Toussaint PJ, Zwetsloot-Schonk JH, Breedveld FC, Putter H, Vliet Vlieland TP. Effectiveness of theintroduction of an International Classification of Functioning, Disability and Health-based rehabilitation
tool in multidisciplinary team care in patients with rheumatoid arthritis. Arthritis Rheum 2007; 57(2): 240—
248.

36

How will the value of using PROs be evaluated?

Finally, practices which implement the routine collection and use of PROs may want to assess the impact thisintervention has on the quality of their care. Below, approaches to evaluate the value of using PROs in clinicalpractice are described, using either experimental designs and methods or quasi-experimental/qualityimprovement designs and methods. For the purposes of this section, value is defined as the sum of clinicalquality, service quality, and safety divided by the sum of monetary cost and time.
A. EXPERIMENTAL DESIGNS AND METHODS (e.g., randomized controlled trials, cluster-randomized
trials)Resources Needed: Monetary resources ranging from several hundred thousand dollars to several million
dollars, depending on work scope, duration, and complexity of the trial.
 Skilled investigators.
 Robust research staff, including data managers, collectors, editors, analysts, project
coordinators/managers, research clinical assistants, engaged clinicians, biostatisticians.
 Institutional support and approval, including ethics review.
 May be facilitated by health information system with a robust electronic health record,
information systems analysts and programmers, and health informaticians.
 May require health economists.
 Generally requires grant support.
Advantages:
 Minimize bias.
 Increase rigor.
 Strong internal validity – tests efficacy, not effectiveness.
Disadvantages:
 Very complex.
 Resource intensive, in terms of monetary and personnel costs.
 Long process.
 Weak generalizability.
 Persons randomized to control condition may not benefit.
 May have insufficient follow-up time to detect impact.
 Subject to bias, including selection, differential history, differential maturation,
contamination.
 May have insufficient power to detect differences.
 Designs traditionally suited to explore the efficacy of conceptually neat components ofclinical practice may be inadequate to study PROs in clinical practice, or conduct
improvement or implementation research.

37

B. QUASI-EXPERIMENTAL, OBSERVATIONAL, SURVEY, OR QUALITY IMPROVEMENT DESIGNS AND
METHODS (e.g., improvement research, realistic evaluations, plan-do-study-act cycles, time series,
cross-over, case-control, etc.)
Resources Needed: Relatively lower monetary costs.
 Quality improvement personnel.
 Health information technology frequently required, but not generally other scientists oranalysts.
 May require institutional support and approval, including ethics review.
Advantages:
 Most health systems and community settings have well developed quality improvement
programs in place.
 Improvement research is usually fast, cheap, and may be more generalizable.
 External validity may be good – tests effectiveness, not efficacy.
 Sensitive to details of implementation, organizational history, leadership, and context.
 Relies heavily on simple pre-post, uncontrolled designs with the goal of identifying how to
implement effective changes.
 In the case of plan-do-study-act (PDSA), requires minimal training and involves making smallchanges incrementally and learning from experience while doing so.
 All subjects assessed and may benefit.
Disadvantages:
 Significant risk for bias and lack of validity.
 Lack of optimal experimental control.
 Difficult to determine if an intervention resulted in an improvement.
 Generalizability from one site to another is limited.
KEY REFERENCES
1. Baker GR. Strengthening the contributions of quality improvement research to evidence based healthcare.
Qual Saf Health Care 2006; 15: 150—151.
2. Batalden PB, Davidoff F. What is “quality improvement” and how can it transform healthcare? Qual Saf
Health Care 2007; 16: 2—3.
3. Berwick DM. Developing and testing changes in delivery of care. Ann Intern Med 1998; 128: 651—656.
4. Berwick DM. The science of improvement. JAMA 2008; 299: 1182—1184.
5. Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. 1963. Chicago: Rand
McNally College Publishing Company.
6. Campbell M, Fitzpatrick R, Haines A, et al. Framework for design and evaluation of complex interventions
to improve health. BMJ 2000; 321: 694—696.
7. Chang C-H. Patient-reported outcomes measurement and management with innovative methodologies
and techniques. Qual Life Res 2007; 16: 157—166.
8. Cook TD, Campbell DT. Quasi-Experimentation: Design and Analysis Issues for Field Settings. 1979. Chicago:
Rand McNally College Publishing Company.
9. Davidoff F, Batalden P. Toward stronger evidence on quality improvement. Draft publication guidelines:
the beginning of a consensus project. Qual Saf Health Care 2005; 14: 319—325.
38

10. Donner A, Klar N. Design and Analysis of Cluster-randomized Trials in Health Research. 2000. London UK:
Arnold.
11. Donner A, Klar N. Statistical considerations in the design and analysis of community intervention trials. J
Clin Epidemiol 1996; 49: 435—439.
12. Fayers PM. Evaluating the effectiveness of using PROs in clinical practice: a role for cluster-randomized
trials. Qual Life Res 2008; 17: 1315—1321.
13. Kish L. Statistical Design for Research. 1987. New York: John Wiley and Sons.
14. Meinert C. Clinical Trials: Design, Conduct, and Analysis. 1986. New York: Oxford University Press, Inc.
15. Murray DM. Design and Analysis of Group-Randomized Trials. 1998. New York: Oxford University Press,Inc.
16. Nelson EC, Splaine ME, Plume SK, et al. Good measurement for good improvement work. Qual Manage
Health Care 2004; 13: 1—16.
17. Pawson R, Tilley N. Realistic Evaluation. 1997. London, England: Sage Publications, Ltd.
18. Priebe S, McCabe R, Bullenkamp J, et al. Structured patient-clinician communication and 1-year outcome incommunity mental healthcare: cluster-randomized controlled trial. British J Psychiat 2007; 191: 420—426.

39

BIBLIOGRAPHY
1.
2.

3.
4.
5.

6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.

19.

20.

A Guide to PROMs Methodology. Available at: http://www.hscic.gov.uk/media/1537/A-Guide-to-PROMsMethodology/pdf/PROMS_Guide_v5.pdf
Ackerley SJ, Gordon HJ, Elston AF, Crawford LM, McPherson KM. Assessment of quality of life andparticipation within an outpatient rehabilitation setting. [Erratum appears in Disabil Rehabil. 2009;31(13): 1107]. Disability & Rehabilitation 2009; 31: 906—913.
Addington-Hall J, Kalra L. Who should measure quality of life? BMJ 2001; 322: 1417—1420.
Ahles TA, Wasson JH, Seville JL, et al. A controlled trial of methods for managing pain in primary carepatients with or without co-occurring psychosocial problems. Ann Fam Med 2006; 4(4): 341—350.
Ahmed S, Berzon RA, Revicki DA, et al. The use of patient-reported outcomes (PRO) within comparative
effectiveness research: implications for clinical practice and health care policy. Med Care 2012; 50(12):1060—1070.
Ayers DC, Zheng H, Franklin PD. Integrating patient-reported outcomes into orthopaedic clinical practice:
proof of concept from FORCE-TJR. Clin Orthop Relat Res 2013; 471(11): 3419—25.
Bainbridge D, Seow H, Sussman J, et al. Multidisciplinary health care professionals' perceptions of theuse and utility of a symptom assessment system for oncology patients. J Oncol Pract 2011; 7(1): 19—23.
Baker GR. Strengthening the contributions of quality improvement research to evidence based
healthcare. Qual Saf Health Care 2006; 15: 150—151.
Basch E, Abernethy AP. Supporting clinical practice decisions with real-time patient-reported outcomes. J
Clin Oncol 2011; 29: 954—956.
Basch E, Abernethy AP. Commentary: encouraging clinicians to incorporate longitudinal patient-reported
symptoms in routine clinical practice. J Oncol Pract 2011; 7(1): 23—25.
Batalden PB, Davidoff F. What is “quality improvement” and how can it transform healthcare? Qual Saf
Health Care 2007; 16: 2—3.
Beaton DE, van Eerd D, Smith P, et al. Minimal change is sensitive, less specific to recovery: a diagnostic
testing approach to interpretability. J Clin Epidemiol 2011; 64(5): 487—496.
Bennett AV, Jensen RE, Basch E. Electronic patient-reported outcome systems in oncology clinicalpractice. CA Cancer J Clin 2012; 62(5): 337—347.
Berry DL, Blumenstein BA, Halpenny B, et al. Enhancing patient-provider communication with theelectronic self-report assessment for cancer: a randomized trial. J Clin Oncol 2011; 29(8): 1029—1035.
Berry DL, Hong F, Halpenny B, et al. Electronic self-report assessment for cancer and self-care support:
results of a multicenter randomized trial. J Clin Oncol 2014; 32(3): 199—205.
Berwick DM. Developing and testing changes in delivery of care. Ann Intern Med 1998; 128: 651—656.
Berwick DM. The science of improvement. JAMA 2008; 299: 1182—1184.
Bjorner JB, Rose M, Gandek B, Stone AA, Junghaenel DU, Ware JE Jr. Method of administration of
PROMIS scales did not significantly impact score level, reliability, or validity. J Clin Epidemiol 2014; 67(1):108—113.
Bowling A, Bond M, Jenkinson C, Lamping DL. Short Form 36 (SF-36) Health Survey questionnaire: which
normative data should be used? Comparisons between the norms provided by the Omnibus Survey in
Britain, the Health Survey for England and the Oxford Healthy Life Survey. J Public Health Med 1999;
21(3): 255—270.
Brundage M, Feldman-Stewart D, Leis A, et al. Communicating quality of life information to cancer
patients: a study of six presentation formats. J Clin Oncol 2005; 23(28): 6949—6956.
4021. Brundage M, Leis A, Bezjak A, et al. Cancer patients' preferences for communicating clinical trial qualityof life information: a qualitative study. Qual Life Res 2003; 12(4): 395—404.
22. Bunge M, Muhlhauser I, Steckelberg A. What constitutes evidence-based patient information? Overview
of discussed criteria. Patient Educ Couns 2010; 78: 316—328.
23. Bush N, Donaldson G, Moinpour C, et al. Development, feasibility and compliance of a web-based
system for very frequent QOL and symptom home self-assessment after hematopoietic stem celltransplantation. Qual Life Res 2005; 14(1): 77—93.
24. Callahan MB. Using quality of life measurement to enhance interdisciplinary collaboration. Adv Ren
Replace Ther 2001; 8(2): 148—151.
25. Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. 1963. Chicago:
Rand McNally College Publishing Company.
26. Campbell M, Fitzpatrick R, Haines A, et al. Framework for design and evaluation of complex interventions
to improve health. BMJ 2000; 321: 694—696.
27. Cella D, Gershon R, Lai JS, Choi S. The future of outcomes measurement: item banking, tailored shortforms, and computerized adaptive assessment. Qual Life Res 2007; 16 Suppl 1: 133—141.
28. Chambers LW, Haight M, Norman G, MacDonald L. Sensitivity to change and the effect of mode ofadministration on health status measurement. Med Care 1987; 25(6): 470—80.
29. Chang CH. Patient-reported outcomes measurement and management with innovative methodologies
and technologies. Qual Life Res 2007; 16 Suppl 1: 157—166.
30. Chen TH, Li L, Kochen MM. A systematic review: how to choose appropriate health-related quality of life
(HRQOL) measures in routine general practice? J Zhejiang Univ Sci B 2005 Sep; 6(9): 936—940.
31. Cheung YB, Goh C, Thumboo J, Khoo KS, Wee J. Quality of life scores differed according to mode ofadministration in a review of three major oncology questionnaires. J Clin Epidemiol 2006; 59(2): 185—91.
32. Cleeland CS, Wang XS, Shi Q, et al. Automated symptom alerts reduce postoperative symptom severity
after cancer surgery: a randomized controlled clinical trial. J Clin Oncol 2011; 29(8): 994—1000.
33. Cook TD, Campbell DT. Quasi-Experimentation: Design and Analysis Issues for Field Settings. 1979.
Chicago: Rand McNally College Publishing Company.
34. Coons SJ, Gwaltney CJ, Hays RD, et al. Recommendations on evidence needed to support measurement
equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR
ePRO Good Research Practices Task Force Report. Value Health 2009; 12(4): 419—429. Available at:
http://www.ispor.org/taskforces/eprotf.asp.
35. Davidoff F, Batalden P. Toward stronger evidence on quality improvement. Draft publication guidelines:
the beginning of a consensus project. Qual Saf Health Care 2005; 14: 319—325.
36. de Wit M, Delemarre-van de Waal HA, Bokma JA, et al. Monitoring and discussing health related qualityof life in adolescents with type 1 diabetes improves psychosocial well being. Diabetes Care 2008; 31(8):1521—1526.
37. Delbanco T, Walker J, Darer JD, et al. Open notes: doctors and patients signing on. Ann Intern Med 2010;
153: 121—125.
38. Detmar SB, Aaronson NK. Quality of life assessment in daily clinical oncology practice: a feasibility study.
Eur J Cancer 1998; 34: 1181—6.
39. Detmar SB, Muller MJ, Schornagel JH, Wever LD, Aaronson NK. Health related quality of life assessments
and patient-physician communication. JAMA 2002; 288(23): 3027—3034.
40. Donaldson G. Patient reported outcomes and the mandate for measurement. Qual Life Res 2008;
17:1303 – 1313.
41. Donaldson M. Using patient-reported outcomes in clinical oncology practice: Benefits, challenges, andnext steps. Expert Review of Pharmacoeconomics and Outcomes Research 2006; 6: 87—95.
42. Donaldson MS. Taking PROs and patient-centered care seriously: incremental and disruptive ideas forincorporating PROs in oncology practice. Qual Life Res 2008; 17: 1323—1330.
41

43. Donaldson MS. Use of patient-reported outcomes in clinical oncology practice. A nonvisit approach topatient care based on the IOM report. J Ambulatory Care Manage 2007; 30 (4): 302—307.
44. Donner A, Klar N. Design and Analysis of Cluster-randomized Trials in Health Research. 2000. London UK:
Arnold.
45. Donner A, Klar N. Statistical considerations in the design and analysis of community intervention trials. J
Clin Epidemiol 1996; 49: 435—439.
46. Dowrick C. Does testing for depression influence diagnosis or management by general practitioners?
Fam Pract 1995; 12(4): 461—465.
47. Eaton ML. Surrogate decision making for genetic testing for Alzheimer disease. Genetic Testing 1999; 3:
93—97.
48. Engelen V, Haverman L, Koopman H, et al. Development and implementation of a patient reported
outcome intervention (QLIC-ON PROfile) in clinical paediatric oncology practice. Patient Educ Couns
2010; 81(2): 235—244.
49. Espallargues M, Valderas JM, Alonso J. Provision of feedback on perceived health status to health care
professionals: a systematic review of its impact. Med Care 2000; 38(2): 175—186.
50. Eton, DT, Beebe TJ, Hagen PT, et al. Harmonizing and consolidating the measurement of patient reported
information at health care institutions; a position statement of Mayo Clinic. Patient Relat Outcome Meas
2014; 5: 7—15.
51. Fagerlin A, Wang C, Ubel PA. Reducing the influence of anecdotal reasoning on people's health caredecisions: is a picture worth a thousand statistics? Med Decis Making 2005; 25(4): 398—405.
52. Fayers PM. Evaluating the effectiveness of using PROs in clinical practice: a role for cluster-randomized
trials. Qual Life Res 2008; 17: 1315—1321.
53. Fayers PM, Hays RD. Don’t middle your MIDs: regression to the mean shrinks estimates of minimally
important differences. Qual Life Res 2014: 23 (1): 1—4.
54. Foley JF, Brandes DW. Redefining functionality and treatment efficacy in multiple sclerosis. Neurology
2009; 72: S1—11.
55. Frost MH, Bonomi AE, Cappelleri JC, et al. Applying quality-of-life data formally and systematically into
clinical practice. Mayo Clin Proc 2007; 82 (10): 1214—1228.
56. Fung C, Hays RD. Prospects and challenges in using patient-reported outcomes in clinical practice. Qual
Life Res 2008; 17: 1297—1302.
57. Gaissmaier W, Wegwarth O, Skopec D, et al. Numbers can be worth a thousand pictures: individual
differences in understanding graphical and numerical representations of health-related information.
Health Psychol 2012; 31: 286—296.
58. Giesinger JM, Golser M, Erharter A, et al. Do neurooncological patients and their significant others agree
on quality of life ratings? Health & Quality of Life Outcomes 2009; 7: 87.
59. Gilbody S, Sheldon T, Wessely S. Should we screen for depression? BMJ 2006; 332(7548): 1027—1030.
60. Greenhalgh J. The applications of PROs in clinical practice: What are they, do they work, and why? Qual
Life Res 2009; 18: 115—123.
61. Greenhalgh J, Flynn R, Long AF, Tyson S. Tacit and encoded knowledge in the use of standardised
outcome measures in multidisciplinary team decision making: a case study of in-patientneurorehabilitation. Soc Sci Med 2008; 67: 183—194.
62. Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measures in routine clinical care:lack of impact or lack of theory? Social Science & Medicine 2005; 60: 833—843.
63. Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of health in routine
practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract
1999; 5(4): 401—416.
64. Gundy CM, Aaronson NK. Effects of mode of administration (MOA) on the measurement properties of
the EORTC QLQ-C30: a randomized study. Health Qual Life Outcomes 2010; 8: 35.
42

65. Gutteling JJ, Darlington AS, Janssen HL, Duivenvoorden HJ, Busschbach JJ, deMan RA. Effectiveness ofhealth related quality of life measurement in clinical practice: a prospective, randomised controlled trial
in patients with chronic liver disease and their physicians. Qual Life Res 2008; 17: 195—205.
66. Guyatt GH, Ferrans CE, Halyard MY, et al. Exploration of the value of health-related quality-of-life
information from clinical research and into clinical practice. Mayo Clin Proc 2007; 82(10): 1229—1239.
67. Hanmer J, Hays RD, Fryback DG. Mode of administration is important in US national estimates of healthrelated quality of life. Med Care 2007; 45(12): 1171—1179.
68. Haverman, L, Engelen, V, Van Rossum, MA, Heymans, HS, & Grootenhuis, MA. Monitoring health-related
quality of life in paediatric practice: development of an innovative web-based application. BMC
Pediatrics, 2011; 11: 3—10.
69. Haverman, L, van Rossum, MA, van, VM, van den Berg, JM, Dolman, KM, Swart, J et al. Effectiveness of a
web-based application to monitor health-related quality of life. Pediatrics, 2013; 131(2), e533-e543.
70. Hays RD, Brodsky M, Johnston MF, Spritzer KL, Hui K-K. Evaluating the statistical significance of healthrelated quality-of-life change in individual patients. Eval Health Prof 2005; 28(2): 160—171.
71. Hays RD, Kim S, Spritzer KL, et al. Effects of mode and order of administration on generic health-related
quality of life scores. Value in Health 2009; 12: 1035—1039.
72. Hepner KA, Brown JA, Hays RD. Comparison of mail and telephone in assessing patient experiences inreceiving care from medical group practices. Eval Health Prof 2005; 28(4): 377—389.
73. Hess R, Santucci A, McTigue K, Fischer G, Kapoor W. Patient difficulty using tablet computers to screen in
primary care. J Gen Intern Med 2008; 23(4): 476—480.
74. Hess R, Tindle H, Conroy MB, et al. A randomized controlled pilot trial of the functional assessment
screening tablet to engage patients at the point of care. J Gen Intern Med 2014 Aug 8 [Epub ahead of print].
75. Higginson IJ, Carr A. Measuring quality of life: Using quality of life measures in the clinical setting. BMJ
2001; 322: 1297—1300.
76. Hilarius DL, Kloeg PH, Gundy CM, Aaronson NK. Use of health related quality of life assessments in dailyclinical oncology nursing practice: A community hospital based intervention study. Cancer 2008; 113(3):628—637.
77. Hildon Z, Allwood D, Black N. Impact of format and content of visual display of data on comprehension,
choice and preference: a systematic review. Int J Qual Health Care 2012; 24: 55—64.
78. Hood K, Robling M, Ingledew D, et al. Mode of data elicitation, acquisition and response to surveys: asystematic review. Health Technol Assess 2012; 16(27): 1—162.
79. Hughes EF, Wu AW, Carducci MA, Snyder C. What can I do? Recommendations for responding to issues
identified by patient-reported outcomes assessments used in clinical practice. J Support Oncol 2012; 10:
143—148.
80. Jörngården A, Wettergen L, von Essen L. Measuring health-related quality of life in adolescents andyoung adults: Swedish normative data for the SF-36 and the HADS, and the influence of age, gender, andmethod of administration. Health Qual Life Outcomes 2006; 4: 91.
81. Kazis LE, Callahan LF, Meenan RF, Pincus T. Health status reports in the care of patients with rheumatoid
arthritis. J Clin Epidemiol 1990; 43(11): 1243—1253.
82. Kemmler G, Zabernigg A, Gattringer K, et al. A new approach to combining clinical relevance andstatistical significance for evaluation of quality of life changes in the individual patient. J Clin Epidemiol
2010; 63: 171—179.
83. Kish L. Statistical Design for Research. 1987. New York: John Wiley and Sons.
84. Kocks JWH, van den Berg JWK, Kerstjens HAM, et al. Day-to-day measurement of patient-reported
outcomes in exacerbations of chronic obstructive pulmonary disease. International Journal of COPD
2013; 8: 273—286.
85. Lohr KN, Zebrack BJ. Using patient-reported outcomes in clinical practice: challenges and opportunities.
Qual Life Res 2009 Feb; 18(1): 99—107.
43

86. MacArthur C, Winter HR, Bick DE, et al. Effects of redesigned community postnatal care on womens'
health 4 months after birth: a cluster randomised controlled trial. Lancet 2002; 359(9304): 378—385.
87. Magruder-Habib K, Zung WWK, Feussner JR. Improving physicians recognition and treatment ofdepression in general medical care. Results from a randomized clinical trial. Med Care 1990; 28: 239—250.
88. Marshall S, Haywood K, Fitzpatrick R. Impact of patient-reported outcome measures on routine practice:a structured review. J Eval Clin Pract 2006; 12(5): 559—568.
89. Masskulpan P, Riewthong K, Dajpratham P, Kuptniratsaikul V. Anxiety and depressive symptoms after
stroke in 9 rehabilitation centers. Journal of the Medical Association of Thailand 2008; 91: 1595—1602.
90. Mathias SD, Fifer SK, Mazonson PD, Lubeck DP, Buesching DP, Patrick DL. Necessary but not sufficient:
the effect of screening and feedback on outcomes of primary care patients with untreated anxiety. J Gen
Intern Med 1994; 9(11): 606—615.
91. McHorney CA. Health status assessment methods for adults: Past accomplishments and futurechallenges. Annual Rev of Public Health 1999; 20: 309—35.
92. Meinert C. Clinical Trials: Design, Conduct, and Analysis. 1986. New York: Oxford University Press, Inc.
93. Meyer KB, Espindle DM, DeGiacomo JM, Jenuleson CS, Kurtin PS, Davies AR. Monitoring dialysis patients'
health status. Am J Kidney Dis 1994; 24(2): 267—279.
94. Mooney KH, Beck SL, Friedman RH, Farzanfar R, Wong B. Automated monitoring of symptoms during
ambulatory chemotherapy and oncology providers' use of the information: a randomized controlled
clinical trial. Support Care Cancer 2014 Apr 1 [Epub ahead of print].
95. Murray DM. Design and Analysis of Group-Randomized Trials. 1998. New York: Oxford University Press,
Inc.
96. Naglie G, Tomlinson G, Tansey C, et al. Utility-based quality of life measures in Alzheimer's disease.
Quality of Life Research 2006; 15: 631—643.
97. Nelson EC, Splaine ME, Plume SK, et al. Good measurement for good improvement work. Qual Manage
Health Care 2004; 13: 1—16.
98. Norman R, King MT, Clarke D, Viney R, Cronin P, Street D. Does mode of administration matter?
Comparison of online and face-to-face administration of a time trade-off task. Qual Life Res 2010; 19(4):499—508.
99. Novella JL, Jochum C, Jolly D, et al. Agreement between patients' and proxies' reports of quality of life in
Alzheimer's disease. Quality of Life Research 2001; 10: 443—452.
100. Nowels D, McGloin J, Westfall JM, Holcomb S. Validation of the EQ-5D quality of life
instrument in patients after myocardial infarction. Qual Life Res 2005 Feb; 14(1): 95—105.
101. Osoba D. Translating the science of patient-reported outcomes assessment into clinical practice. J NatlCancer Inst Monogr 2007; 37: 5—11.
102. Pawson R, Tilley N. Realistic Evaluation. 1997. London, England: Sage Publications, Ltd.
103. Pickard AS, Knight SJ. Proxy evaluation of health-related quality of life: a conceptual framework forunderstanding multiple proxy perspectives. Medical Care 2005; 43: 493—499.
104. Pickard AS, Lin H-W, Knight SJ, et al. Proxy assessment of health-related quality of life in African
American and White respondents with prostate cancer: perspective matters. [Erratum appears in Med
Care 2009 Apr; 47 (4): 491 Note: Knight, Sara L [corrected to Knight, Sara J]]. Medical Care 2009; 47:
176—183.
105. Priebe S, McCabe R, Bullenkamp J, et al. Structured patient-clinician communication and 1-year outcome
in community mental healthcare: cluster-randomized controlled trial. British J Psychiat 2007; 191: 420—426.
106. Puhan MA, Ahuja A, van Natta ML, Ackatz LE, Meinert C. Interviewer versus self-administered healthrelated quality of life questionnaires - Does it matter? Health Qual Life Outcomes 2011; 9: 30.
44

107. Raat H, Mangunkusumo RT, Landgraf JM, Kloek G, Brug J. Feasibility, reliability, and validity of adolescenthealth status measurement by the Child Health Questionnaire Child Form (CHQ-CF): internet
administration compared with the standard paper version. Qual Life Res 2007; 16(4): 675—685.
108. Ready RE, Ott BR, Grace J. Patient versus informant perspectives of quality of life in mild cognitive
impairment and Alzheimer's disease. International Journal of Geriatric Psychiatry 2004; 19: 256—265.
109. Reifler DR, Kessler HS, Bernhard EJ, Leon AC, Martin GJ. Impact of screening for mental health concerns
on health service utilization and functional status in primary care patients. Arch Intern Med 1996;
156(22): 2593—2599.
110. Ring AE, Cheong KA, Watkins CL, Meddis D, Cella D, Harper PG. A randomized study of electronic diaryversus paper and pencil collection of patient-reported outcomes in patients with non-small cell lungcancer. Patient 2008; 1(2): 105—113.
111. Rose M, Bezjak A. Logistics of collecting patient-reported outcomes (PRO) in clinical practice: an
overview and practical examples. Qual Life Res 2009; 18: 125—136.
112. Rose MS, Koshman ML, Spreng S, Sheldon R. Statistical issues encountered in the comparison of healthrelated quality of life in diseased patients to published general population norms: problems andsolutions. J Clin Epidemiol 1999; 52(5): 405—412.
113. Rosenbloom SK, Victorson DE, Hahn EA, Peterman AH, Cella D. Assessment is not enough: A randomized
controlled trial of the effects of HRQoL assessment on quality of life and satisfaction in oncology clinicalpractice. Psychooncology 2007; 16: 1069—1079.
114. Rothman RL, Montori VM, Cherrington A, Pignone MP. Perspective: The role of numeracy in health care.
J Health Commun 2008; 13(6): 583—595.
115. Rubenstein LV, McCoy JM, Cope DW, et al. Improving patient quality of life with feedback to physicians
about functional status. J Gen Intern Med 1995; 10(11): 607—614.
116. Santana MJ, Feeny D. Framework to assess the effects of using patient-reported outcome measures inchronic care management. Qual Life Res 2013 Dec 7 [Epub ahead of print].
117. Santana MJ, Feeny D, Johnson JA, et al. Assessing the use of health related quality of life measures in theroutine clinical care of lung transplant patients. Qual Life Res 2010; 19: 371—379.
118. Schwartz LM, Woloshin S, Black WC, Welch HG. The role of numeracy in understanding the benefit ofscreening mammography. Ann Intern Med 1997; 127(11): 966–972.
119. Seid M, Limbers CA, Driscoll KA, Opipari-Arrigan LA, Gelhard LR, Varni JW. Reliability, validity, andresponsiveness of the pediatric quality of life inventory (PedsQL) generic core scales and asthma
symptoms scale in vulnerable children with asthma. Journal of Asthma; 47: 170—177.
120. Snyder CF, Blackford AL, Brahmer JR, et al. Needs assessments can identify scores on HRQoL
questionnaires that represent problems for patients: an illustration with the Supportive Care Needs
Survey and the QLQ-C30. Qual Life Res 2010; 19(6): 847—845.
121. Snyder CF, Blackford AL, Okuyama T, et al. Using the EORTC QLQ-C30 in clinical practice for patientmanagement: identifying scores requiring a clinician's attention. Qual Life Res 2013; 22: 2685—2691.
122. Snyder CF, Dy SM, Hendricks DE, Brahmer JR, Carducci MA, Wolff AC, Wu AW. Asking the right questions:investigating needs assessments and health-related quality-of-life questionnaires for use in oncology
clinical practice. Support Care Cancer 2007 Sep; 15(9): 1075—85.
123. Snyder CF, Herman JM, White SM, et al. When using patient-reported outcomes in clinical practice, themeasure matters: a randomized controlled trial. Journal of Oncology Practice 2014; 10: e299—306.
124. Snyder CF, Jensen R, Courtin O, Wu AW. PatientViewpoint: a website for patient-reported outcomes
assessment. Qual Life Res 2009; 18; 793—800.
125. Street RL, Gold WR, McDowell T. Using health status surveys in medical consultations. Med Care 1994;
32(7): 732—744.

45

126. Swartz RJ, de Moor C, Cook KF, et al. Mode effects in the Center for Epidemiologic Studies Depression(CES-D) scale: personal digital assistant vs. paper and pencil administration. Qual Life Res 2007; 16(5):803—813.
127. Talley NJ, Wiklund I. Patient reported outcomes in gastroesophageal reflux disease: An overview ofavailable measures. Qual Life Res 2005 Feb; 14(1): 21—33.
128. Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring patient-reported outcomes inclinical practice: a systematic review of the literature. Qual Life Res 2008; 17(2): 179—193.
129. Varni JW, Thissen D, Stucky BD, et al. PROMIS parent proxy report scales: an item response theory
analysis of the parent proxy report item banks. Quality of Life Research 2012; 21:1223—40.
130. Veenstra M, Moum T, Garratt AM. Patient experiences with information in a hospital setting:associations with coping and self-rated health in chronic illness. Quality of Life Research 2006; 15: 967—
978.
131. Velikova G, Booth L, Smith AB, et al. Measuring quality of life in routine oncology practice improvescommunication and patient well-being: A randomized clinical trial. J Clin Oncol 2004; 22: 714—24.
132. Verhoef J, Toussaint PJ, Putter H, Zwetsloot-Schonk JH, Vliet Vlieland TP. The impact of introducing an
ICF-based rehabilitation tool on staff satisfaction with multidisciplinary team care in rheumatology: an
exploratory study. Clin Rehabil 2008; 22(2): 23—37.
133. Verhoef J, Toussaint PJ, Zwetsloot-Schonk JH, Breedveld FC, Putter H, Vliet Vlieland TP. Effectiveness of
the introduction of an International Classification of Functioning, Disability and Health-based
rehabilitation tool in multidisciplinary team care in patients with rheumatoid arthritis. Arthritis Rheum
2007; 57(2): 240—248.
134. Wagner EH. Chronic disease management: what will it take to improve care for chronic illness? Eff Clin
Pract 1998; 1(1): 2—4.
135. Walter OB, Becker J, Bjorner JB, Fliege H, Klapp BF, Rose M. Development and evaluation of a computer
adaptive test for ‘Anxiety’ (Anxiety-CAT). Qual Life Res 2007; 16 Suppl 1: 143—155.
136. Wasson JH, Stukel TA, Weiss JE, Hays RD, Jette AM, Nelson EC. A randomized trial of the use of patient
self-assessment data to improve community practices. Eff Clin Pract 1999; 2(1): 1—10.
137. Weinberger M, Oddone EZ, Samsa GP, Landsman PB. Are health-related quality-of-life measures affectedby the mode of administration? J Clin Epidemiol 1996; 49(2): 135—140.
138. Wilson TR, Birks Y, Alexander DJ. Pitfalls in the interpretation of standardised quality of life instruments
for individual patients? A qualitative study in colorectal cancer. Qual Life Res 2013; 22: 1879–1888.
139. Wright EP, Selby PJ, Crawford M, et al. Feasibility and compliance of automated measurement of qualityof life in oncology practice. J Clin Oncol 2003; 21(2): 374—382.
140. Wu AW, Jensen RE, Salzburg C, Snyder C. Advances in the Use of Patient Reported Outcome Measures inElectronic Health Records: Including Case Studies. Landscape Review Prepared for the PCORI National
Workshop to Advance the Use of PRO measures in Electronic Health Records. Atlanta, GA. November 1920, 2013, pp 10-14. Available at: http://www.pcori.org/assets/2013/11/PCORI-PRO-Workshop-EHRLandscape-Review-111913.pdf.
141. Wu AW, Kharrazi H, Boulware LE, Snyder CF. Measure once, cut twice--adding patient-reported outcome
measures to the electronic health record for comparative effectiveness research. J Clin Epidemiol 2013;
66: S12—20.
142. Wyrwich KW, Norquist JM, Lenderking WR, Acaster S, Industry Advisory Committee of International
Society for Quality of Life Research (ISOQOL). Methods for interpreting change over time in patientreported outcome measures. Qual Life Res 2013; 22: 475–483.
143. Yazicioglu K, Duyan V, Karatas K, et al. Effects of sociodemographic characteristics, illness process, andsocial support on the levels of perceived quality of life in veterans. Military Medicine 2006; 171: 1083—
1088.
46

144. Zimmerman M, Ruggero CJ, Chelminski I, et al. Developing brief scales for use in clinical practice: thereliability and validity of single-item self-report measures of depression symptom severity, psychosocial
impairment due to depression, and quality of life. J Clin Psychiatry 2006 Oct; 67(10): 1536—41.

47