COSMIN manual for systematic reviews of PROMs

COSMIN methodology for systematic reviews ofPatient‐Reported Outcome Measures (PROMs)
user manual
Version 1.0 dated February 2018
Lidwine B Mokkink
Cecilia AC Prinsen
Donald L PatrickJordi Alonso
Lex M Bouter
Henrica CW de Vet
Caroline B Terwee

Contact
LB Mokkink, PhD
VU University Medical Center
Department of Epidemiology and Biostatistics
Amsterdam Public Health research institute1081 BT Amsterdam
The Netherlands
Website: www.cosmin.nl
E‐mail: w.mokkink@vumc.nl

1

COSMIN manual for systematic reviews of PROMs

The development of the COSMIN guideline for systematic reviews of Patient‐Reported
Outcome Measures (PROMs) and the COSMIN Risk of Bias checklist for systematic
reviews of PROMs was financially supported by the Amsterdam Public Health research
institute, VU University Medical Center, Amsterdam, the Netherlands.

2

COSMIN manual for systematic reviews of PROMs
Table of contents
Foreword
1. Background information
1.1 The COSMIN initiative

57
7

1.2 How to cite this manual

8

1.3 Development of a comprehensive COSMIN guideline for systematic reviews ofPatient‐Reported Outcome Measures (PROMs)

8

1.4 COSMIN taxonomy and definitions

9

1.5 The COSMIN Risk of Bias Checklist

13

1.6 What has changed in the COSMIN methodology?

15

1.7 Ten‐step procedure & outline of the manual17

2. Part A Steps 1‐4: Perform the literature search
2.1 Step 1: Formulate the aim of the review

20
20

2.2 Step 2: Formulate eligibility criteria

20

2.3 Step 3: Perform a literature search

21

2.4 Step 4: Select abstracts and full‐text articles

24

3. Part B steps 5‐7: Evaluating the measurement properties of the included PROMs
3.1 General methodology

25
25

3.1.2 Applying criteria for good measurement properties

27

3.2. Step 5: Evaluating content validity

37

3.3 Step 6. Evaluation of the internal structure of PROMs: structural validity, internalconsistency, and cross‐cultural validity\measurement invariance38

3.4 Step 7. Evaluation of the remaining measurement properties: reliability,
measurement error, criterion validity, hypotheses testing for construct validity, andresponsiveness4. Part C steps 8‐10: Selecting a PROM
4.1 Step 8: Describe interpretability and feasibility

40
44
44

4.2 Step 9: formulate recommendations45

4.3 Step 10: report the systematic review

45

5. COSMIN Risk of Bias checklist
5.1 Assessing risk of bias in a study on structural validity (box 3)
5.2 Assessing risk of bias in a study on internal consistency (box 4)

47
47
50

5.3 Assessing risk of bias in a study on cross‐cultural validity\ measurement
invariance (box 5)51

5.4 Assessing risk of bias in a study on reliability (box 6)

53
3

COSMIN manual for systematic reviews of PROMs
5.5 Assessing risk of bias in a study on measurement error (box 7)

56

5.6 Assessing risk of bias in a study on criterion validity (box 8)

57

5.7 Assessing risk of bias in a study on hypotheses testing for construct validity (box9) 58
5.8 Assessing risk of bias in a study on responsiveness (box 10)
Appendix 1. Example of a search strategy (81)Appendix 2. Example of a flowchart
Appendix 4. Example of a table on characteristics of the included study populations
Appendix 5. Information to extract on interpretability of PROMs
Appendix 6. Information to extract on feasibility of PROMs
Appendix 7. Table on results of studies on measurement properties
Appendix 8. Summary of Findings Tables
6. References

60
64
65
67
68
69
70
72
74

4

COSMIN manual for systematic reviews of PROMs

Foreword
Research performed with outcome measurement instruments of poor or unknown
quality constitutes a waste of resources and is unethical (3). Unfortunately this practice
is widespread (4). Selecting the best outcome measurement instrument for the outcomeof interest in a methodologically sound way requires: (1) high quality studies that
document the evaluation of the measurement properties (in total nine different aspectsof reliability, validity, and responsiveness) of relevant outcome measurement
instruments in the target population; and (2) a high quality systematic review of studies
on measurement properties in which all information is gathered and evaluated in asystematic and transparent way, accompanied by clear recommendations for the mostsuitable available outcome measurement instrument. However, conducting such asystematic review is quite complex and time consuming, and it requires expertise within
the research team on the construct to be measured, on the patient population, and on
the methodology of studies of measurement properties.
High quality systematic reviews can provide a comprehensive overview of themeasurement properties of Patient‐Reported Outcome Measures (PROMs) and supports
evidence‐based recommendations in the selection of the most suitable PROM for a given
purpose. For example, for selecting the most suitable PROM for an outcome included in acore outcome set (COS)(5). These systematic reviews can also identify gaps inknowledge on the measurement properties of PROMs, which can be used to design newstudies on measurement properties.
The COSMIN (COnsensus‐based Standards for the selection of health Measurement
INstruments) initiative aims to improve the selection of outcome measurement
instruments in research and clinical practice by developing tools for selecting the mostsuitable instrument for the situation at issue.
Recently, a comprehensive methodological guideline for systematic reviews of PROMs
was developed by the COSMIN initiative(2). In addition, the COSMIN checklist wasadapted for assessing the risk of bias in studies on measurement properties insystematic reviews of PROMs(6). Also, a Delphi study was performed to develop
standards and criteria for assessing the content validity of PROMs(7).
The present manual is a supplement to the COSMIN guideline for systematic reviews of
measurement instruments (2) including the use of the COSMIN Risk of Bias checklist(6).
This manual contains additional detailed information on how to perform each of the
proposed ten steps to conduct a systematic review of PROMs, on how to use the COSMIN
Risk of Bias checklist, and on how to come to recommendations on the most suitable
instrument for a given.
In the COSMIN methodology we use the word patient. However, sometimes the target
population of the systematic review or the PROM is not patients, but e.g. healthy
individuals (e.g. for generic PROMs) or caregivers (when a PROM measures caregiverburden). In these cases, the word patient should be read as e.g. healthy person or
caregiver.

5

COSMIN manual for systematic reviews of PROMs
Throughout the manual we provide some examples to explain the COSMIN Risk of Bias
checklist. We would like to emphasize that these are arbitrarily chosen and used forillustrative purposes.
The COSMIN methodology focusses on PROMs used as outcome measurement
instruments (i.e. evaluative application). The methodology can also be used for othertypes of measurement instruments (like clinician‐reported outcome measures or
performance‐based outcome measures), or other applications (e.g. diagnostic or
predictive applications), but the methodology may need to be adapted for these otherpurposes. For example, structural validity may not be relevant for some types of
instruments and responsiveness is less relevant when an instrument is used as adiagnostic instrument. New standards for assessing the quality of a study on thereliability of a MRI scan should be developed.
This manual aims to facilitate the understanding and practice performance of asystematic review on PROMs. We give detailed instructions on each step to take,and we will provide many examples and suggestions on how to perform each step.
We aim to continue updating this manual when deemed necessary, based on
experience and suggestions by users of the COSMIN methodology. If you have anysuggestions or questions, please contact us (w.mokkink@vumc.nl).

6

COSMIN manual for systematic reviews of PROMs

1. Background information
1.1 The COSMIN initiative
The COSMIN initiative aims to improve the selection of health measurement instruments
both in research and clinical practice by developing tools for selecting the most suitable
instrument for a given situation. COSMIN is an international initiative consisting of amultidisciplinary team of researchers with expertise in epidemiology, psychometrics,
and qualitative research, and in the development and evaluation of outcome
measurement instruments in the field of health care, as well as in performing systematic
reviews of outcome measurement instruments.
COSMIN steering committee
Lidwine B Mokkink1
Cecilia AC Prinsen1
Donald L Patrick2
Jordi Alonso3
Lex M Bouter1
Henrica CW de Vet1
Caroline B Terwee1
1 Department of Epidemiology and Biostatistics, Amsterdam Public Health research

institute, VU University Medical Center, De Boelelaan 1089a, 1081 HV, Amsterdam, theNetherlands;
2 Department of Health Services, University of Washington, Thur Canal St Research

Office, 146 N Canal Suite 310, 98103, Seattle, USA;

3 IMIM (Hospital del Mar Medical Research Institute), CIBER Epidemiology and Public

Health (CIBERESP), Dept. Experimental and Health Sciences, Pompeu Fabra University(UPF), Doctor Aiguader 88, 08003, Barcelona, Spain.

7

COSMIN manual for systematic reviews of PROMs
1.2 How to cite this manualThis manual was based on three articles, published in peer‐reviewed scientific journals.
If you use the COSMIN methodology, please refer to these articles:
Prinsen, C. A.C., Mokkink, L. B., Bouter, L. M., Alonso, J., Patrick, D. L., De Vet, H. C., et al.
(2018). COSMIN guideline for systematic reviews of Patient‐Reported Outcome
Measures. Qual Life Res, accept.
Mokkink, L.B., de Vet, H.C.W., Prinsen, C.A.C., Patrick, D.L., Alonso, J., Bouter, L.M.,
Terwee, C.B. (2017). COSMIN Risk of Bias checklist for systematic reviews of Patient‐
Reported Outcome Measures. Qual Life Res. doi: 10.1007/s11136‐017‐1765‐4. [Epubahead of print].
Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, Bouter LM, deVet HCW, Mokkink LB. COSMIN methodology for evaluating the content validity ofPatient‐Reported Outcome Measures: a Delphi study, submitted.
1.3 Development of a comprehensive COSMIN guideline for systematic reviews ofPatient‐Reported Outcome Measures (PROMs)
In the absence of empirical evidence, the COSMIN guideline for systematic reviews of
PROMs (2) was based on our experience that we (that is: the COSMIN steeringcommittee) have gained over the past years in conducting systematic reviews of PROMs(8, 9), in supporting other systematic reviewers in their work (10, 11) and in thedevelopment of COSMIN methodology (12, 13). In addition, we have studied the qualityof systematic reviews of PROMs in two consecutive reviews (14, 15), and in reviews thathave used the COSMIN methodology we have specifically searched for the comments
made by review authors relating to the COSMIN methodology. Further, we have had
iterative discussions by the COSMIN steering committee, both at face‐to‐face meetings(CP, WM, HdV and CT) and by email. We gained experience from results of a recent
Delphi study on the content validity of PROMs (7), and from results of a previous Delphi
study on the selection of outcome measurement instruments for outcomes included incore outcome sets (COS) (5).
Further, the guideline was developed in concordance with existing guidelines forreviews, such as the Cochrane Handbook for systematic reviews of interventions (16),and for diagnostic test accuracy reviews (17), the PRISMA Statement(18), the Instituteof Medicine (IOM) standards for systematic reviews of comparative effectivenessresearch(19), and the Grading of Recommendations Assessment, Development and
Evaluation (GRADE) principles (20).
This guideline focuses on the methodology of systematic reviews of existing PROMs, forwhich at least a PROM development study or some information on its measurement
properties is available, and that are used for evaluative purposes (e.g. to measure theeffects of treatment or to monitor patients over time). Instruments that are being used
for discriminative or predictive purposes are not being considered in the guideline. The
methods can also be used for systematic reviews of other types of outcome
measurement instruments such as clinician reported outcome measurement

8

COSMIN manual for systematic reviews of PROMs
instruments (ClinROMs) or performance‐based outcome measurement instruments(PerBOMs), for reviews of one single outcome measurement instrument, or for apredefined set of outcome measurement instruments. However, some adaptations may
be needed in some of the steps or in the COSMIN Risk of Bias checklist.
1.4 COSMIN taxonomy and definitions
In the literature different terminology and definitions for measurement properties arecontinuously being used. The COSMIN initiative developed a taxonomy of measurement
properties relevant for evaluating PROMs. In the first COSMIN Delphi study, conducted
in 2006‐2007, consensus was reached on terminology and definitions of all included
measurement properties in the COSMIN checklist(21). This taxonomy (Figure 1) formed
the foundation on which the guideline was based.

Figure 1. The COSMIN taxonomy(21)9COSMIN manual for systematic reviews of PROMs

Taxonomy of measurement properties
The COSMIN taxonomy of measurement properties is presented in Figure 1. It was
decided that all measurement properties included in the taxonomy are relevant andshould be evaluated for PROMs used in an evaluative application.
In assessing the quality of a PROM we distinguish three domains, i.e. reliability, validity,
and responsiveness. Each domain contains one or more measurement properties, i.e.
quality aspects of measurement instruments. The domain reliability contains threemeasurement properties: internal consistency, reliability, and measurement error. The
domain validity also contains three measurement properties: content validity (including
face validity), structural validity, hypotheses testing for construct validity, cross‐culturalvalidity and criterion validity. The domain responsiveness contains only one
measurement property, which is also called responsiveness.
Definitions of measurement properties
Consensus‐based definitions of all included measurement properties in the COSMIN
checklist are presented in Table 1.

10

COSMIN manual for systematic reviews of PROMs
Table 1. COSMIN definitions of domains, measurement properties, and aspects ofmeasurement properties(21)TermDomainMeasurement
property

Definition
Aspect of ameasurement
property

Reliability
Reliability(extendeddefinition)

Internal
consistency
Reliability

Measurement
error
Validity
Content
validity
Face validity

The degree to which themeasurement is free frommeasurement error
The extent to which scores forpatients who have not changed
are the same for repeatedmeasurement under several
conditions: e.g. using different
sets of items from the same PROM
(internal consistency); over time
(test‐retest); by different persons
on the same occasion (inter‐
rater); or by the same persons(i.e. raters or responders) on
different occasions (intra‐rater)
The degree of theinterrelatedness among the items
The proportion of the total
variance in the measurements
which is due to ‘true’† differences
between patients
The systematic and random errorof a patient’s score that is not
attributed to true changes in theconstruct to be measured
The degree to which a PROM
measures the construct(s) it
purports to measure
The degree to which the content
of a PROM is an adequate
reflection of the construct to bemeasured
The degree to which (the itemsof) a PROM indeed looks as
though they are an adequate
reflection of the construct to bemeasured11

COSMIN manual for systematic reviews of PROMs
Constructvalidity

Structuralvalidity
Hypotheses
testing
Cross‐
culturalvalidity

The degree to which the scores ofa PROM are consistent withhypotheses (for instance withregard to internal relationships,
relationships to scores of other
instruments, or differences
between relevant groups) basedon the assumption that the PROM
validly measures the construct tobe measured
The degree to which the scores ofa PROM are an adequate
reflection of the dimensionality of
the construct to be measured
Idem construct validity

The degree to which theperformance of the items on atranslated or culturally adapted
PROM are an adequate reflectionof the performance of the items of
the original version of the PROM
Criterion
The degree to which the scores ofvaliditya PROM are an adequate
reflection of a ‘gold standard’ResponsivenessThe ability of a PROM to detectchange over time in the construct
to be measured
Responsiveness
Idem responsivenessInterpretability*
Interpretability is the degree towhich one can assign qualitative
meaning ‐ that is, clinical orcommonly understoodconnotations – to a PROM’s
quantitative scores or change inscores.
† The word ‘true’ must be seen in the context of the CTT, which states that any observationis composed of two components – a true score and error associated with the observation.
‘True’ is the average score that would be obtained if the scale were given an infinite
number of times. It refers only to the consistency of the score, and not to its accuracy (22)* Interpretability is not considered a measurement property, but an importantcharacteristic of a measurement instrument

12

COSMIN manual for systematic reviews of PROMs
1.5 The COSMIN Risk of Bias Checklist
It was decided to create three separate versions of the original COSMIN checklist (1) tobe used for different purposes when assessing the methodological quality of studies on
measurement properties, i.e. the COSMIN Design checklist, the COSMIN Risk of Bias
checklist, and the COSMIN Reporting checklist. The Risk of Bias checklist was exclusively
developed for assessing the methodological quality of single studies included insystematic reviews of PROMs. The purpose of assessing the methodological quality of astudy in a systematic review is to screen for risk of bias in the included studies. The term‘risk of bias’ is in compliance with the Cochrane methodology for systematic reviews oftrials and diagnostic studies(16). It refers to whether the results based on themethodological quality of the study are trustworthy.
The checklist contains standards referring to design requirements and preferredstatistical methods of studies on measurement properties. For each measurement
property a COSMIN box was developed containing all standards needed to assess thequality of a study on that specific measurement property. Both preferred statisticalmethods based on Classical test Theory (CTT) and Item Response Theory (IRT) or Raschanalyses are included in the standards.
COSMIN standards and criteria
It is important to note that COSMIN makes a distinction between“standards” and “criteria”: Standards refer to design requirements andpreferred statistical methods for evaluating the methodological quality ofstudies on measurement properties. Criteria refer to what constitutes
good measurement properties (quality of PROMs). In the first COSMIN
Delphi study (1), only standards were developed for evaluating the qualityof studies on measurement properties. Criteria for what constitutes goodmeasurement properties were not developed. However, such criteria areneeded in systematic reviews to provide evidence‐based
recommendations for which PROMs are good enough to be used inresearch or clinical practice. Therefore, criteria were developed for goodmeasurement properties (Table 4) (2).

Boxes from the checklist
The COSMIN Risk of Bias checklist contains ten boxes with standards for PROM
development (box 1) and for nine measurement properties: Content validity (box 2),
Structural validity (box 3), Internal consistency (box 4), Cross‐culturalvalidity\measurement invariance (box 5), Reliability (box 6), Measurement error (box7), Criterion validity (box 8), Hypotheses testing for construct validity (box 9), andResponsiveness (box 10).

13

COSMIN manual for systematic reviews of PROMs
Modular tool
In one article one or more studies (often on different measurement properties) can be
described. The methodological quality of each study should be assessed separately by
rating all standards included in the corresponding box. Therefore, the COSMIN checklist
should be used as a modular tool. This means that it may not be necessary to complete
the whole checklist when evaluating the quality of studies described in an article. In
accordance with the COSMIN taxonomy(21), the measurement properties evaluated in
an article determine which boxes need to be completed. Each assessment of ameasurement property is considered to be a separate study. For example, if in an article
the internal consistency and construct validity of an instrument were assessed, only twoboxes need to be completed, i.e. box 4 Internal consistency and box 9 Hypotheses testingfor construct validity. This modular system was developed because not all measurement
properties are assessed in all articles.
COSMIN considers each subscale of a (multi‐dimensional) PROM
separately
The measurement properties of a PROM should be rated separately foreach set of items that make up a score. This can be a single item if this item
is used as a standalone score, a set of items making up a subscale scorewithin a multi‐dimensional PROM, or a total PROM score if all items of a
PROM are being summarized into a total score. Each score is assumed torepresent a construct and is therefore considered a separate PROM. In theremaining of this manual when we refer to a PROM, we mean a PROM
score or subscore.
For example, if a multidimensional PROM consists of three subscales andone single item, each scored separately, the measurement properties of
the three subscales and the single item need to be rated separately. If thesubscale scores are also summarized into a total score, the measurement
properties of the total score should also be rated separately.

Four‐point rating system
For each study, an overall judgement is needed on the quality of the particular study. We
therefore developed a four‐point rating system where each standard within a COSMIN
box can be rated as ‘very good’, ‘adequate’, ‘doubtful’ or ‘inadequate’ (6). The overallrating of the quality of each study is determined by taking the lowest rating of any
standard in the box (i.e. “the worst score counts” principle) (12). This overall rating of
the quality of the studies can be used in grading the quality of the evidence (see Chapter3), taking into account that results of ‘inadequate’ quality studies will decrease the trust
one can put on the results and the overall conclusion about a measurement property of a
PROM (as used in a specific context).
Each box contains a standard asking if there were any other important methodologicalflaws that are not covered by the checklist, but that may lead to biased results or
conclusions. For example, in a reliability study, the administrations should be
independent. Independent administrations imply that the first administration has not
influenced the second administration, i.e. at the second administration the patient
should not have been aware of the scores on the first administration (recall bias).
14

COSMIN manual for systematic reviews of PROMs
1.6 What has changed in the COSMIN methodology?
Inadequate studies are no longer ignored
In our previous protocol for performing systematic reviews of PROMs, we recommended
to ignore the results of inadequate (previously called ‘poor’) quality studies because theresults of these studies might be biased. In the current methodology, we recommendthat results from inadequate studies may be included when pooling or summarizing theresults from all studies is possible, or if the results of inadequate studies are consistent
with the results from studies of better quality. The reasoning behind this is that if theresults from inadequate studies are included, it should be considered to downgrade thequality of the evidence because of risk of bias (see Chapter 3). Herewith, the current
methodology is more in line with recommendations from the Cochrane Collaboration forsystematic reviews of intervention studies (16) and of diagnostic test accuracy studies(17).
Removing standards about a reasonable gold standard for criterion validity andresponsivenessWe decided to delete the standards about deciding whether a gold standard used can beconsidered a reasonable gold standard. We now recommend the review team todetermine before assessing the methodological quality of studies, which outcome
measurement instrument can be considered a reasonable gold standard. When an
included study uses this particular gold standard instrument to asses validity, the study
can be considered as a study on criterion validity. The COSMIN panel reached consensus
that no gold standard exist for PROMs (13). The only exception is when a shortened
instrument is compared to the original long version. In that case, the original long
version can be considered the gold standard. Often, authors consider their comparator
instrument incorrectly a gold standard, for example when they compare the scores of a
new instrument to a widely used and well‐known instrument. In this situation, the study
is considered a study on construct validity and box 9 Hypotheses testing for constructvalidity should be completed.
Removing standards on formulating hypotheses for hypotheses testing for constructvalidity and responsivenessWe decided to delete all standards about formulating hypotheses a priori from the boxes
Hypotheses testing for construct validity and Responsiveness. Although we consider it
majorly important to define hypotheses in advance when assessing construct validity or
responsiveness of a PROM, results of studies without these hypotheses can –in manycases– still be used in a systematic review on PROMs because the presented correlations
or mean differences between (sub)groups are not necessarily biased and thus can beevaluated. The conclusions of the authors though may be biased when a priori
hypotheses are lacking. We recommend that the review team formulates a set of
hypotheses about the expected direction and magnitude of correlations between the
PROM of interest and other PROMs and of mean differences in scores between groups(23). This way, all results are compared to the same relevant hypothesis. If constructvalidity studies do include hypotheses, the review team can adopt these hypotheses ifthey consider them adequate. This way, the results from many studies can still be usedin the systematic review as studies without hypotheses will no longer receive a‘inadequate’ (previously called ‘poor’) quality rating. A detailed explanation forcompleting these boxes can be found in Chapter 5.
15

COSMIN manual for systematic reviews of PROMs

Removal of standards on sample size
We decided to remove the standard about adequate sample size for single studies fromthose boxes where it is possible to pool the results (i.e. the boxes Internal consistency,
Reliability, Measurement error, Criterion validity, Hypotheses testing for constructvalidity, and Responsiveness) to a later phase of the review, i.e. when drawing
conclusions across studies on the measurement properties of the PROM (Chapter 3).
This was decided because several small high‐quality studies can together provide
sufficient evidence for the measurement property. Therefore, we recommend to take theaggregated sample size of the available studies into account when assessing the overall
quality of evidence for a measurement property in a systematic review. This is incompliance with Cochrane Handbooks (16, 17). However, the standard about adequate
sample size for single studies was maintained in the boxes Content validity, Structuralvalidity, and Cross‐cultural validity\measurement invariance, because the results ofthese studies cannot be pooled. In these boxes factor analyses, IRT or Rasch analyses areincluded as preferred statistical methods and these methods require sample sizes thatare sufficiently large to obtain reliable results. The suggested sample size requirements
should be considered as general guidance; in some situations, dependent on type ofmodel, number of factors or items, more nuanced criteria might be applied.
Removal of standards on missing data and handling missing data
Each box of the original COSMIN checklist, except for box on content validity, contains
standards about whether the percentage missing items was reported, and how thesemissing items were handled. Although we consider information on missing items very
important to report, we decided to remove these standards from all boxes in the
COSMIN Risk of Bias checklist, as it was agreed upon within the COSMIN steeringcommittee that lack of reporting on number of missing items and on how missing items
were handled would not necessarily lead to biased results of the study. Furthermore, atthe moment there is little evidence and no consensus yet about what the best way is tohandle missing items in studies on measurement properties.
New order of evaluating the measurement properties
A new order of evaluating the measurement properties is proposed, as shown in Table 2.
Content validity is considered to be the most important measurement property because
first of all it should be clear that the items of the PROM are relevant, comprehensive, andcomprehensible with respect to the construct of interest and target population (2).
Therefore, we recommend to first evaluate the development and content validity studiesof the PROMs. PROMs with high quality evidence of inadequate content validity can be
excluded from further assessment in the systematic review.
Next, we recommend to evaluate the internal structure of PROMs, including themeasurement properties structural validity, internal consistency, and cross‐culturalvalidity\measurement invariance. Internal structure refers to how the different items in
a PROM are related, which is important to know for deciding how items might be
combined into a scale or subscale. Evaluating the internal structure of the instrument isrelevant for PROMs that are based on a reflective model. In a reflective model theconstruct manifests itself in the items, i.e. the items are a reflection of the construct to bemeasured (24). Finally, the remaining measurement properties are considered, i.e.
reliability, measurement error, criterion validity, hypotheses testing for constructvalidity and responsiveness.

16

COSMIN manual for systematic reviews of PROMs

Table 2. Order in which the measurement properties are assessed
Content validity
1. PROM development*
2. Content validity
Internal structure
3. Structural validity
4. Internal consistency
5. Cross‐cultural validity\measurement invariance
Remaining measurement properties
6. Reliability
7. Measurement error
8. Criterion validity
9. Hypotheses testing for construct validity
10. Responsiveness* not a measurement property, but taken into account when evaluating content validity
The COSMIN methodology is specifically developed for use in reviews of outcome
measurement instruments. The proposed order of evaluating the measurement
properties is therefore based on use/application as an outcome measure, i.e. in an
evaluative purpose. This order is also reflected in the ordering of boxes in the COSMIN
Risk of Bias checklist. We think that some of the included measurement properties areless relevant for other purposes. For example, responsiveness is less relevant when an
instrument is used as a diagnostic tool.
More information on removal of other standards and adaptations to individual
standards and how these changes were decided upon, can be found elsewhere (6).
1.7 Ten‐step procedure & outline of the manual
In a systematic review of PROMs an overview is given on available evidence of eachmeasurement property of each included PROM to come to overall conclusions per
measurement property and to give recommendation for the most suitable PROM for agiven purpose. A guideline, consisting of a sequential ten‐step procedure was developedby the COSMIN steering committee, subdivided into three parts: A, B and C (Figure 2)(2).
Part A consists of steps 1-4 and generally, these steps are standard procedures when
performing systematic reviews, and are in agreement with existing guidelines for reviews (16,17): preparing and performing the literature search, and selecting relevant publications.
The methodology of part A is presented in Chapter 2.

17

COSMIN manual for systematic reviews of PROMs

Part B consists of steps 5-7 and concerns the evaluation of the measurement properties of the
included PROMs. These steps were particularly developed for systematic reviews of PROMs.
Part B is described in Chapter 3. We first describe the general methodology, and next, we
explain step 5 (evaluation of content validity), step 6 (evaluation of internal structure),
and step 7 (evaluation of the remaining measurement properties) in more detail.
Part C consists of steps 8‐10 and concerns the evaluation of the interpretability andfeasibility of the PROMs (step 8), formulating (step 9) and the reporting of thesystematic review (step 10). Part C is described in Chapter 4.
In Chapter 5 we elaborate on how to rate each standard included in the COSMIN Risk ofBias checklist. We close the manual with providing examples of a search strategy, a
flowchart, and all tables that need to be presented in a review (appendices 1‐8).
We aim to continue updating this manual when deemed necessary, based on
experience and suggestions by users of the COSMIN methodology. If you have anysuggestions or questions, please contact us.

18

COSMIN manual for systematic reviews of PROMs

Figure 2. Ten steps for conducting a systematic review of PROMs (2)19

COSMIN manual for systematic reviews of PROMs

2. Part A Steps 1‐4: Perform the literature search
Steps 1‐4 are standard procedures when performing a systematic reviews (2). It
concerns formulating the research aim (step 1) and eligibility criteria (step 2), andperforming the literature search (step 3) and the selection of articles (step 4). These arein agreement with existing guidelines for reviews (16, 17).
2.1 Step 1: Formulate the aim of the review
The aim of a systematic review of PROMs focuses on the quality of the PROMs. It should
include the following four key elements: 1) the construct; 2) the population(s); 3) thetype of instrument(s); and 4) the measurement properties of interest. An example of a
research aim could be: “our aim is to critically appraise, compare and summarize thequality of the measurement properties of all self‐report fatigue questionnaires forpatients with multiple sclerosis (MS), Parkinson’s disease (PD) or stroke” (25). In theaim of this review the construct of interest is ‘fatigue’, the population of interest is‘patients with MS, PD or stroke’, the type of instrument of interest is ‘self‐reportquestionnaire’, and ‘all’ measurement properties are explored in the review.
We also recommend that these four key elements are included in the title of the review.
For example: “Self‐report fatigue questionnaires in multiple sclerosis, Parkinson's
disease and stroke: a systematic review of measurement properties.” The four key
elements will also inform both the eligibility criteria in Step 2 and the search strategy to
be conducted in Step 3.
2.2 Step 2: Formulate eligibility criteria
The eligibility criteria should be in agreement with the four key elements of the review
aim: 1) the PROM(s) should aim to measure the construct of interest; 2) the study
sample (or an arbitrary majority, e.g. ≥50%) should represent the population of interest;3) the study should concern PROMs; and 4) the aim of the study should be the evaluationof one or more measurement properties, the development of a PROM (to rate thecontent validity), or the evaluation of the interpretability of the PROMs of interest (e.g.
evaluating the distribution of scores in the study population, percentage of missing
items, floor and ceiling effects, the availability of scores and change scores for relevant(sub)groups, and the minimal important change or minimal important difference(26)).
We recommend to exclude studies that only use the PROM as an outcome measurement
instrument, for instance, studies in which the PROM is used to measure the outcome(e.g. in randomized controlled trials), or studies in which the PROM is used in avalidation study of another instrument. Including all studies that use a specific PROM
would require a much more extended search strategy because the search filter forstudies on measurement properties could not be used (see step 3). It would also be
extremely time‐consuming to find all studies using a specific PROM because the PROMs
used in a study may not be reported in the abstract. This basically means that all studies
performed in the population of interest should be screened full‐text. As it might be

20

COSMIN manual for systematic reviews of PROMs
unlikely that this can be done in a standardized way, we therefore consider thisapproach not feasible.
Further, we recommend to include only full text articles because often very limited
information on the design of a study is found in abstracts. Information found in abstracts
will hamper the quality assessment of the study and the results of the measurement
properties in steps 5 through 7.
2.3 Step 3: Perform a literature search
In agreement with the Cochrane methodology (16, 17), and based on consensus (5),MEDLINE and EMBASE are considered to be the minimum databases to be searched. In
addition, it is recommended to search in other (content‐specific) databases, dependingon the construct and population of interest, for example Web of Science, Scopus,
CINAHL, or PsycINFO.
Search strategy
In the guideline and in this manual we focus on a systematic review including all PROMs
measuring a specific construct which have ‐ at least some extent ‐ been validated. An
adequate search strategy therefore consists of a comprehensive collection of searchterms (i.e. index terms and free text words) for the four key elements of the review aim:
1) construct; 2) population(s); 3) type of instrument(s), and 4) measurement properties.
It is recommended to consult a clinical librarian as well as experts on the construct andstudy population of interest.
A comprehensive PROM filter has been developed for PubMed by the Patient Reported
Outcomes Measurement Group, University of Oxford, that can be used as a search blockfor ‘type of measurement instrument(s)’, and is available through the COSMIN website.
Regarding search terms for measurement properties we recommend to use a highly
sensitive validated search filter for finding studies on measurement properties, which isavailable for PubMed and EMBASE and can be found on the COSMIN website (27, 28). An
example of a PubMed search strategy is included in Appendix 1 (25).
Additional sources for search blocks can be found on the website https://blocks.bmi‐
online.nl/. Here a group of Dutch medical information specialists have compiled anumber of ‘Search Blocks’. These building blocks are intended for use when performing
omplex search strategies in medical and health bibliographic databases, like PubMed,
Embase, PsycInfo, Web of Science and others. These blocks can be used by experts in thefield of searching for literature, e.g. information specialists, clinical librarians, healthlibrarians and professionals in closely related areas.
As, in principle, the aim is finding all PROMs, in agreement with the Cochranemethodology, it is recommended to search databases from the date of inception tillpresent (16, 17). The use of language restrictions depends on the inclusion criteria
defined in Step 2. In general, it is recommend not to use language restrictions in the
search strategy, even if there are no resources to translate the articles for the review. In
this way, review authors are at least able to report their existence.
Next, it is recommended to use software such as Endnote or Reference Manager tomanage references. Covidence (ww.covidence.org) could be of use when managing the21

COSMIN manual for systematic reviews of PROMs
review screening performance. Covidence is an online tool that supports systematic
reviewers in uploading search results, screening abstracts and full texts, resolve
disagreements, and export data into RevMan or Excel. It is recommended by the
Cochrane Collaboration.
Searches must be as up to date as possible when the review is published, so it may benecessary to update the search before submitting (or before acceptation) of themanuscript.
In addition to searching for all PROMs, Figure 3 shows the search strategy for othertypes of reviews. For example, if the aim of a systematic review is to identify all PROMs
that have been used, search terms for measurement properties should not be used (firstcolumn of Figure 3). If the aim of a systematic review is to report on the measurement
properties of one particular PROM (or a selection of predefined PROMs), search terms
for the construct of interest should not be used and search terms for the type of
instrument can be replaced by search terms for the names of the instruments of interest(third column of Figure 3).
As in all systematic reviews the literature search should be carefully documented in the
study protocol. We recommend to document the following: names of the databases thatwere searched, including the interface used to search the databases (for example,
PubMed was used for searching MEDLINE); all search terms used; any restrictions thatwere used (for example, human studies only, not animals); the number of records
retrieved from each database; and the number of unique records.
In accordance with the PRISMA statement, it is recommend to add the documentation of
the selection process to a flow diagram. An example of the PRISMA flow diagram can befound in Appendix 2 (29). This flow diagram includes information on the total number of
abstracts selected, the total number of full‐text articles that were selected, and the main
reasons for excluding full‐text articles. Note that the included articles can described one
or more studies (on one or more different measurement properties). Therefore, we
recommend to describe in the flow diagram the total number of articles included, thetotal number of studies described in those articles, and the total number of different(versions of) PROMs found.

22

COSMIN manual for systematic reviews of PROMs* Search filter described by Terwee et al (27)Figure 3. Search strategy for different types of systematic reviews of measurement
instruments
23

COSMIN manual for systematic reviews of PROMs
2.4 Step 4: Select abstracts and full‐text articles
It is generally recommended to perform the selection of abstracts and full‐text articles
by two reviewers independently (16, 17). If a study seems relevant by at least one
reviewer based on the abstract, or in case of doubt, the full‐text article needs to be
retrieved and screened. Differences should be discussed and if consensus between thetwo reviewers cannot be reached, it is recommended to consult a third reviewer. It isalso recommended to check all references of the included articles to search for
additional potentially relevant studies. If many new articles are found, the initial searchstrategy might have been insufficiently comprehensive and may need to be improved
and redone.
Note that Cochrane reviews use various additional sources in finding relevant studies,such as dissertations, editorials, conference proceedings, and reports. The probability offinding additional relevant articles for systematic reviews of PROMs in these type of
additional sources, however, appears to be small.

24

COSMIN manual for systematic reviews of PROMs

3. Part B steps 5‐7: Evaluating the measurement properties of the included PROMs
Part B consists of steps 5‐7 of the guideline on conducting a systematic review of PROMs
and concerns the evaluation of the measurement properties of the included PROMs. In
Chapter 3.1 we will start with explaining the general methodology of Part B. In Chapter
3.2 we discuss step 5 in which the content validity is assessed. Next, in Chapter 3.3 step
6 is explained, which concerns evaluating the internal structure of a PROM (i.e.
structural validity, internal consistency, and cross‐cultural validity\measurement
invariance). Lastly, in Chapter 3.4 we describe step 7 of the guideline that concerns theevaluation of the remaining measurement properties (i.e. reliability, measurement error,
criterion validity, hypotheses testing for construct validity, and responsiveness).
3.1 General methodology
The evaluation of each measurement property includes three sub steps (see Figure 4):

Figure 4. Practical outline for performing a systematic review
 First, the methodological quality of each single study on a measurement property
is assessed using the COSMIN Risk of Bias checklist. Detailed instructions for howeach standard included in the COSMIN Risk of Bias checklist should be rated aredescribed in Chapter 5.
 Second, the result of each single study on a measurement property is rated
against the updated criteria for good measurement properties (+ / ‐ / ?) (Table
4);
 Third, the evidence is summarized per measurement property per PROM, theoverall result is rated against the criteria for good measurement properties,

25

COSMIN manual for systematic reviews of PROMs(Table 4), and the quality of the evidence will be graded by using the GRADE
approach (2).
3.1.1 Evaluating the methodological quality of studies
To evaluate the methodological quality of the included studies using the COSMIN Risk ofBias checklist, it should first be determined which measurement properties are assessedin each article.
Determine which measurement properties are assessed
Often multiple studies on different measurement properties are described in one article(i.e. one study for each measurement property, e.g. a study on internal consistency,
construct validity, and reliability, each with its own specific design requirements). The
quality of each study is separately evaluated using the corresponding COSMIN box(Table 3). The COSMIN Risk of Bias checklist should be used as a modular tool; only
those boxes should be completed for the measurement properties that are evaluated in
the article.
Table 3. Boxes of the COSMIN Risk of Bias checklist
Mark the measurement properties that have been evaluated in the article*.
Content validity
Box 1. PROM development
Box 2. Content validity
Internal structure
Box 3. Structural validity
Box 4. Internal consistency
Box 5. Cross‐cultural validity\measurement invariance
Remaining measurement properties
Box 6. Reliability
Box 7. Measurement error
Box 8. Criterion validity
Box 9. Hypotheses testing for construct validity
Box 10. Responsiveness* If a box needs to be completed more than once two or more marks can be placed.
Sometimes the same measurement property is reported for multiple (sub)groups in one
article. For example, when an instrument is validated in two different countries and themeasurement properties are reported for both countries separately. In that case, thesame box may need to be completed multiple times if the design of the study wasdifferent among countries. The review team should decide which boxes should becompleted (and how many times).
In general, we recommend reviewers to consider how the designs and analyses
presented in the article relate to the COSMIN taxonomy (see Figure 1) (21), and26

COSMIN manual for systematic reviews of PROMs
subsequently complete the corresponding COSMIN box. This facilitates a consistentevaluation of the measurement properties across the included studies, regardless of theterminology used by the authors of the different articles.
Determining which box need to be completed sometimes requires a subjective
judgement because the terms and definitions for measurement properties used in anarticle may not be similar to the terms used in the COSMIN taxonomy. For example,
authors may use the term reliability when they present limits of agreement. Whileaccording to the COSMIN taxonomy, this would be considered measurement error. In
that case, we recommend to complete box 7 (Measurement error). Also, authors often
use the term criterion validity for studies in which a PROM is compared to other PROMs
that measure a similar construct. In most cases, this would be considered evidence forconstruct validity, rather than criterion validity according to the COSMIN terms anddefinitions. In that case, we recommend to complete box 9 (Hypotheses testing forconstruct validity).
Sometimes results presented in studies on measurement properties actually do not (orhardly) provide any evidence on the measurement properties of a PROM, even though
they are presented as such. For example, a comparison of different modes ofadministration (e.g. paper versus computer) does not provide information on thereliability or validity of the PROM. Also, sometimes correlations of a PROM with othervariables (e.g. correlations with demographic variables) are reported and considered as
evidence for construct validity however, these correlations provide very limited
evidence for construct validity. Reviewers may decide to ignore such results in theirreview.
Assess the methodological quality of the studies
The quality of each study on a measurement property should be assessed separately,using the corresponding COSMIN box. Each study is rated as very good, adequate,
doubtful or inadequate quality. To determine the overall rating of the quality of eachsingle study on a measurement property, the lowest rating of any standard in the box istaken (i.e. “the worst score counts” principle)(12). For example, if the lowest rating of alleight items of the reliability box is ‘inadequate’, the overall methodological quality ofthat specific reliability study is rated as ‘inadequate’.
In Chapter 5 we will provide more details and examples for how each standard in the
COSMIN Risk of Bias checklist should be rated. We also explain in more detail how tocome to the overall conclusion about the methodological quality of a study, i.e. how toapply the worst score counts principle. The COSMIN Risk of Bias checklist can be foundon our website, along with a working document (created in Excel) that can be used to
document the COSMIN ratings for each box.
3.1.2 Applying criteria for good measurement properties
Data extraction
We recommend to subsequently extract the data on the characteristics of the PROM(s),on characteristics of the included sample(s), on results on the measurement properties,
and on information about interpretability and feasibility of the score(s) of the PROM(s).
This information is needed to decide whether the results of different studies are
sufficiently similar to be pooled or qualitatively summarized. This information can be27

COSMIN manual for systematic reviews of PROMs
presented in overview tables. We recommend to extract the required information fromthe articles and directly paste it into the overview tables. Examples of these tables are
given in Appendices 3‐7.
In the Cochrane Handbook for systematic reviews of interventions it is recommended
that the data extraction is done by two reviewers independently to avoid missing
relevant information (16).
Evaluate each result against criteria of good measurement properties
Next, the result of each study on a measurement property should be rated against the
updated criteria for good measurement properties (Table 4) (2). Each result is rated aseither sufficient (+), insufficient (–), or indeterminate (?). The result of eachmeasurement property and its quality rating can directly be added to the applicable
table (Appendix 7).
Table 4. Updated criteria for good measurement properties
Measurement
property

Rating1

Criteria
CTT:
CFA: CFI or TLI or comparable measure >0.95 OR RMSEA<0.06 OR SRMR <0.082+
Structural validity?

CTT: Not all information for ‘+’ reported
IRT/Rasch: Model fit not reported–

Criteria for ‘+’ not met
At least low evidence4 for sufficient structural validity5 AND
Cronbach's alpha(s) ≥ 0.70 for each unidimensional scale orsubscale6+
Internal consistency

IRT/Rasch:
No violation of unidimensionality3: CFI or TLI or comparable
measure >0.95 OR RMSEA <0.06 OR SRMR <0.08AND
no violation of local independence: residual correlations
among the items after controlling for the dominant factor <0.20 OR Q3's < 0.37
AND
no violation of monotonicity: adequate looking graphs OR item
scalability >0.30
ANDadequate model fit:
IRT: χ2 >0.01
Rasch: infit and outfit mean squares ≥ 0.5 and ≤ 1.5 OR Z‐
standardized values > ‐2 and <2?
–

Criteria for “At least low evidence4 for sufficient structuralvalidity5” not met
At least low evidence4 for sufficient structural validity5 AND
Cronbach’s alpha(s) < 0.70 for each unidimensional scale orsubscale628

COSMIN manual for systematic reviews of PROMs

Reliability

Measurement error

Hypotheses testing forconstruct validity

Cross‐culturalvalidity\measurement
invariance+

ICC or weighted Kappa ≥ 0.70?

ICC or weighted Kappa not reported–+

ICC or weighted Kappa < 0.70
SDC or LoA < MIC5?

MIC not defined–+

SDC or LoA > MIC5
The result is in accordance with the hypothesis7?
–

No hypothesis defined (by the review team)The result is not in accordance with the hypothesis7
No important differences found between group factors (suchas age, gender, language) in multiple group factor analysis OR
no important DIF for group factors (McFadden's R2 < 0.02)+?
+

No multiple group factor analysis OR DIF analysis performed
Important differences between group factors OR DIF was
found
Correlation with gold standard ≥ 0.70 OR AUC ≥ 0.70?

Not all information for ‘+’ reported–+

Correlation with gold standard < 0.70 OR AUC < 0.70–

Criterion validity

The result is in accordance with the hypothesis7 OR AUC ≥ 0.70?

No hypothesis defined (by the review team)The result is not in accordance with the hypothesis7 OR AUC <–0.70
The criteria are based on e.g. Terwee et al.(30) and Prinsen et al.(5)Responsiveness

AUC = area under the curve, CFA = confirmatory factor analysis, CFI = comparative fit index, CTT= classical test theory, DIF = differential item functioning, ICC = intraclass correlation coefficient,
IRT = item response theory, LoA = limits of agreement, MIC = minimal important change,
RMSEA: Root Mean Square Error of Approximation, SEM = Standard Error of Measurement, SDC= smallest detectable change, SRMR: Standardized Root Mean Residuals, TLI = Tucker‐Lewis
index
1 “+” = sufficient, ” –“ = insufficient, “?” = indeterminate
2 To rate the quality of the summary score, the factor structures should be equal across studies
3 unidimensionality refers to a factor analysis per subscale, while structural validity refers to afactor analysis of a (multidimensional) patient‐reported outcome measure
4 As defined by grading the evidence according to the GRADE approach
5 This evidence may come from different studies
6 The criteria ‘Cronbach alpha < 0.95’ was deleted, as this is relevant in the development phase ofa PROM and not when evaluating an existing PROM.
7 The results of all studies should be taken together and it should then be decided if 75% of theresults are in accordance with the hypotheses

The criteria provided in Table 4 are the preferred criteria for each measurement
property. However, for some measurement properties alternative criteria might also beappropriate. For example, additional criteria could be used for assessing the results ofstudies using exploratory factor analysis (EFA) or testing bi‐factor confirmatory factor
29

COSMIN manual for systematic reviews of PROMs
analysis (CFA) models. If the review team has the expertise to assess the results of thesetypes of studies, additional criteria could be used.
From this table, there is one aspect on which we would like to elaborate in more detail.
This concerns the ‘indeterminate’ rating for the result of a study on internal consistency,
i.e. when the criterion ‘at least low evidence for sufficient structural validity’ is not met.
This means that either (1) there is only very low evidence for sufficient structuralvalidity (e.g. because there was only one study on structural validity with a very low
sample size), (2) there was (any) evidence for insufficient structural validity, (3) there
are inconsistent results for structural validity which cannot be explained, or (4) there isno information on the structural validity available.
3.1.3 Summarize the evidence and grade the quality of the evidence
Whereas in Chapter 3.1.1 and 3.1.2 we focused on the quality of single studies on
measurement properties of a PROM, in the Chapters 3.1.3 and 3.1.4 we will focus on thequality of the PROM as a whole.
To come to an overall conclusion of the quality of the PROM, one should first decide
whether the results of all available studies per measurement property are consistent. If
the results are consistent, the results of studies can be quantitatively pooled or
qualitatively summarized, and compared against the criteria for good measurement
properties to determine whether overall, the measurement property of the PROM issufficient (+), insufficient (–), inconsistent (±), or indeterminate (?). Finally, the qualityof the evidence is graded (high, moderate, low, very low evidence), using a modified
GRADE approach, as explained below (page 32‐36).
If the results are inconsistent, there are several strategies that can be used: (a) findexplanations and summarize per subgroup; (b) do not summarize the results and do notgrade the evidence; or (c) base the conclusion on the majority of consistent results, anddowngrade for inconsistency. It is up to the review team to decide on which strategy ismost appropriate to use in their specific situation. Below each strategy is explained in
more detail.
Handling inconsistent results
Based on the quality ratings of single results (as discussed in Chapter 3.1.2.), one decides
whether or not all results on one measurement property of a PROM are consistent. If all
results are consistent, the overall rating will be sufficient (+) or insufficient (‐). If theresults are inconsistent (e.g. both sufficient and insufficient results for reliability or
appropriate model fit found but for different factor structures across studies),
explanations for inconsistency should be explored. For example, inconsistency could be
due to different populations or methods used. If an explanation is found, the results canbe summarized e.g. per subgroup of consistent results. For example, if different results
are found in studies performed in acute patients versus chronic patients, results per
subgroup should be separately summarized. The overall rating for the specificmeasurement property (e.g. reliability) may be sufficient (+) in acute patients, but
insufficient (‐) in chronic patients.

30

COSMIN manual for systematic reviews of PROMs
High quality studies could be considered as providing more evidence than low quality
studies and can be considered decisive in determining the overall rating when ratings
are inconsistent. If inconsistent results can be explained by the quality of the studies,one may decide to summarize the results of very good or adequate studies only, and toignore the results of doubtful an inadequate quality studies. This should then be
explained in the manuscript.
In some cases, more recent evidence can be considered more important than older
evidence. This can also be taken into account when determining the overall rating.
If no explanation for inconsistent results is found, there are two possibilities: (1) the
overall rating will be inconsistent (±); or (2) one could decide to base the overall ratingon the majority of the results (e.g. if the majority of the (consistent) results of studies are
sufficient, an overall rating of sufficient could be considered) and downgrade the qualityof the evidence for inconsistency (see Chapter 3.5).
Summarize the evidence
The results can be quantitatively pooled or qualitatively summarized. We recommend toreport these pooled or summarized results per measurement property per PROM in
Summary of Finding (SoF) Tables, accompanied by a rating of the pooled or summarized
results (+ / ─ / + / ?), and the grading of the quality of evidence (high, moderate, low,
very low). These SoF tables (i.e. one per measurement property) will ultimately be usedin providing recommendations for the selection of the most appropriate PROM for agiven purpose or a particular situation. See Appendix 8 for an example.
Quantitatively pooling the results
If possible, the results from different studies on one measurement property should bestatistically pooled in a meta‐analysis. Pooled estimates of measurement properties canbe obtained by calculating weighted means (based on the number of participants
included per study) and 95% confidence intervals (e.g. (31)). We recommend to consult
a statistician for performing meta‐analyses.
For test‐retest reliability, for example, weighted mean intraclass correlation coefficients(ICCs) and 95% confidence intervals can be calculated using a standard generic inverse
variance random effects model (32). ICC values can be combined based on estimatesderived from a Fisher transformation, z = 0.5 x ln((1+ICC)/(1‐ICC)), which has an
approximate variance, (Var(z) = 1/(N‐3)), where N is the sample size. This method wasapplied in a study by Collins et al. (8).
For construct validity, for example, it can be considered to pool all correlations of a
PROM with other PROMs that measure a similar construct. For example, whenevaluating the construct validity of a physical function subscale one could pool all
extracted correlations of this subscale with other comparison instruments measuringphysical function.
Qualitatively summarizing into a summarized result
If it is not possible to statistically pool the results, the results of studies should be
qualitatively summarized to come to a summarized result, for example by providing therange (lowest and highest) of MIC values found for interpretability, the percentage ofconfirmed hypotheses for construct validity, or the range of each model fit parameter on
a consistently found factor structure in structural validity studies.

31

COSMIN manual for systematic reviews of PROMs
Applying criteria for good measurement properties to the pooled or summarized
result
The pooled or summarized result per measurement property per PROM should again be
rated against the same quality criteria for good measurement properties (Table 4). The
overall rating for the pooled or summarized result can be sufficient (+), insufficient (–),
inconsistent (±), or indeterminate (?). This rating can be added to the pooled orsummarized result per PROM for each measurement property in the Summary of
Findings Tables (Appendix 8).
If the results per study are all sufficient (or all insufficient), the overall rating will also be
sufficient (or insufficient). To rate the qualitatively summarized results as sufficient (orinsufficient), in principle 75% of the results should met the criteria. For example, forstructural validity the criteria is that ‘at least 75% of the CFA studies found the same
factor structure’. The criteria for hypotheses testing and responsiveness (constructapproach) for summary results is that ‘at least 75% of the results should be inaccordance with the hypotheses’ to rate the overall results as ‘sufficient’ and ‘at least
75% of the results are not in accordance with the hypotheses’ to rate the overall results
as ‘insufficient’’.
If the results of single studies which can be pooled are inconsistent and theinconsistency is unexplained, the results could be pooled anyway, and this pooled result
could be rated as either sufficient or insufficient, and subsequently be downgraded due
to inconsistency (see also Chapter 3.1.3 and the next section). If the results of single
studies which cannot be pooled (e.g. results of CFAs) are inconsistent and theinconsistency is unexplained, the overall result will be rated as ‘inconsistent’. In thiscase, the results are actually not summarized, and the evidence will not be graded. If theresults per study are all indeterminate (?), the overall rating will also be indeterminate(?).
Grading the quality of evidence
After pooling or summarizing all evidence per measurement property per PROM, andrating the pooled or summarized result against the criteria for good measurement
properties, the next step is to grade the quality of this evidence. The quality of theevidence refers to the confidence that the pooled or summarized result is trustworthy.
The grading of the quality is based on the Grading of Recommendations Assessment,
Development, and Evaluation (GRADE) approach for systematic reviews of clinical trials(20). Using a modified GRADE approach, the quality of the evidence is graded as high,moderate, low, or very low evidence (for definitions, see Table 5).
The GRADE approach uses five factors to determine the quality of the evidence: risk ofbias (quality of the studies), inconsistency (of the results of the studies), indirectness(evidence comes from different populations, interventions or outcomes than the ones ofinterest in the review), imprecision (wide confidence intervals), and publication bias
(negative results are less often published). For evaluating measurement properties insystematic reviews of PROMs, the following four factors should be taken into account:(1) risk of bias (i.e. the methodological quality of the studies), (2) inconsistency (i.e.
unexplained inconsistency of results across studies), (3) imprecision (i.e. total sample size ofthe available studies), and (4) indirectness (i.e. evidence from different populations than the
population of interest in the review).

32

COSMIN manual for systematic reviews of PROMs
The fifth factor, i.e. publication bias, is difficult to assess in studies on measurement
properties, because of a lack of registries for these type of studies. Therefore, we do nottake this factor into account in this methodology.
Table 5. Definitions of quality levels
Quality level
DefinitionHigh
Moderate

Low
Very low

We are very confident that the true measurement property lies
close to that of the estimate* of the measurement property
We are moderately confident in the measurement property
estimate: the true measurement property is likely to be close to
the estimate of the measurement property, but there is apossibility that it is substantially different
Our confidence in the measurement property estimate islimited: the true measurement property may be substantiallydifferent from the estimate of the measurement property
We have very little confidence in the measurement property
estimate: the true measurement property is likely to be
substantially different from the estimate of the measurement
property* Estimate of the measurement property refers to the pooled or summarized result of themeasurement property of a PROM.
These definitions were adapted from the GRADE approach (20)The GRADE approach is used to downgrade evidence when there are concerns about thequality of the evidence. The starting point is always the assumption that the pooled or
overall result is of high quality. The quality of evidence is subsequently downgraded by
one or two levels per factor to moderate, low, or very low evidence when there is risk ofbias, (unexplained) inconsistency, imprecision (low sample size), or indirect results. The
quality of evidence can even be downgraded by three levels when the evidence is basedon only one inadequate study (i.e. extremely serious risk of bias). The grading of thequality of each measurement property can directly be added to the applicable table(Appendix 8).
Below we explain in more detail how the four GRADE factors can be interpreted andapplied in evaluating the measurement properties of PROMs.
How to apply GRADE
For each pooled result or for the summarized result for each measurement property per
PROM, the quality of the evidence will be determined by using Table 6.
If in summarizing the evidence and determining the overall rating of the pooled orsummarized result for a measurement property, the results of some studies are ignored,
these studies should also be ignored in determining the quality of the evidence. For
example, if only the results of high quality studies are considered in determining theoverall rating, then only the high quality studies determine the grading of the evidence(in this case we would not downgrade for risk of bias).

33

COSMIN manual for systematic reviews of PROMs
Table 6. Modified GRADE approach for grading the quality of evidence
Quality of evidence Lower if
High
Risk of bias
-1 Serious
Moderate-2 Very serious
Low-3 Extremely serious
Very low
Inconsistency-1 Serious-2 Very serious
Imprecision-1 total n=50-100-2 total n<50
Indirectness-1 Serious-2 Very serious
n=sample size
Below we explain in more detail how the four GRADE factors can be interpreted andapplied in evaluating the measurement properties of PROMs.
(1) Risk of bias can occur if the quality of the study is doubtful or inadequate, asassessed with the COSMIN Risk of Bias checklist, or if only one study of adequate
quality is available. The quality of evidence should be downgraded with one level(e.g. from high to moderate evidence) if there is serious risk of bias, with twolevels (e.g. from high to low) if there is very serious risk of bias, or with threelevels (i.e. from high to very low) of there is extremely risk of bias. In Table 7 we
explain what we consider as serious, very serious or extremely serious risk ofbias.
Table 7. Instructions on downgrading for Risk of Bias
Downgrading for Risk of Bias
Risk of bias
No
There are multiple studies of at least adequate quality, or
there is one study of very good quality available
Serious
There are multiple studies of doubtful quality available,or there is only one study of adequate quality
Very serious
There are multiple studies of inadequate quality, or there
is only one study of doubtful quality available
Extremely
There is only one study of inadequate quality available
serious(2) Inconsistency: Inconsistency may already have been solved by pooling orsummarizing the results of subgroups of studies with similar results and provide
overall ratings for these subgroups. In this case, one doesn’t need to downgrade.
If no explanation for inconsistency is found, the review team can decide not topool or summarize results, and rate the results as ‘inconsistent’. In this case, no34

COSMIN manual for systematic reviews of PROMs
quality level for the evidence will be given. However, an alternative solution
could be to rate the pooled or summarized result (e.g. based on the majority of
results) as sufficient or insufficient and downgrade the quality of the evidence forinconsistency with one or two levels. The reviewers should also decide what will
be considered as serious (i.e. ‐1 level) or very serious (i.e. ‐2 levels)
inconsistency, because this is context dependent. It is up to the review team todecide which seems to be the best solution in each situation.
(3) Imprecision refers to the total sample included in the studies. We recommend todowngrade with one level when the total sample size of the pooled orsummarized studies is below 100, and with two levels when the total sample size
is below 50. Note that this principle should not be used for measurement
properties in which a sample size requirement is already included in the COSMIN
Risk of Bias box, i.e. content validity, structural validity, and cross‐culturalvalidity.
(4) Indirectness can occur if studies are included in the review that were (partly)
performed in another population or another context of use than the populationor context of use of interest in the systematic review. For example, if only part ofthe study population consists of patients with the disease of interest, the review
team can decide to downgrade with one or two levels for serious or very serious
indirectness. One can consider downgrading for indirectness for constructvalidity or responsiveness when the evidence is considered weak. For example
when the evidence is only based on comparisons with PROMs measuring
different constructs or only based on differences between extremely different
groups. How to decide on what should be considered as serious or very serious
indirectness is context dependent, and should be decided on by the review team.
To determine the grading for the quality of evidence, the concerns about the quality ofthe evidence should be added up. Therefore, it is helpful to consider the GRADE factors
one by one by using the consecutive order as specified in Table 6. First, the risk of bias isconsidered (see Table 7). For example, when three studies are found with sufficient (i.e.
‘+’) results, but all of doubtful quality, the level of evidence will be downgraded for riskof bias from high to moderate (i.e. ‐1). Second, further downgrading for other factors
should be considered. After risk of bias, inconsistency should be considered. If theresults of the three studies in the example above are all rated as sufficient, nodowngrading for inconsistency is required. Otherwise, downgrading should beconsidered. Next, the sample size should be taken into account. For example, when thesample size of the three studies together is more than 100, there will be no further
downgrading. If the (total) sample size is between e.g. 50‐100, one should downgradewith ‐1. Lastly, the evidence could be downgraded because of indirectness. For example,
consider a systematic review that focusses on pain and comfort scales for infants, andthe inclusion criteria is ‘children between 0‐18 years’ because a lack of studies in infants
only (i.e. below 1 year) was expected. Studies including children of all ages, including
infants, may lead to downgrade the quality of evidence by one level, and studies
including only children between 4 and 12 years may even lead to downgrade the quality
by two levels, due to indirectness of the results. If, in our example, the three studies
found all include children between 0‐4, but only very few infants, one may decide todowngrade one level (i.e. from moderate to low). In this example, the overall quality of35

COSMIN manual for systematic reviews of PROMs
the evidence is now considered as ‘low’, so the conclusion will be that there is low
quality evidence that the measurement property of interest is sufficient.
We recommend that grading is done by two reviewers independently and thatconsensus among the reviewers is reached, if necessary with help of a third reviewer.

36

COSMIN manual for systematic reviews of PROMs
3.2. Step 5: Evaluating content validity
Content validity (i.e. the degree to which the content of a PROM is an adequate reflection of
the construct to be measured (21)) is considered to be the most important measurement
property, because it should be clear that the items of the PROM are relevant,comprehensive, and comprehensible with respect to the construct of interest and study
population (7). The evaluation of content validity requires a subjective judgment by thereviewers. In this judgement, the PROM development study, the quality and results ofadditional content validity studies on the PROMs (if available), and a subjective rating of
the content of the PROMs by the reviewers is taken into account. As this step is very
important but rather extensive we developed a separate users’ manual for guidance on
how to evaluate the content validity of PROMs. This manual can be found on the COSMIN
website(33).
If there is high quality evidence that the content validity of a PROM is insufficient, thePROM will not be further considered in Steps 6‐8 of the systematic review and one candirectly draw a recommendation for this PROM in Step 9 (i.e. recommendation ‘C’:
“PROMs that should not be recommended (i.e. PROMs with high evidence of insufficient
content validity)). In all other cases, the PROM can be further taken into consideration in
the systematic review.

37

COSMIN manual for systematic reviews of PROMs
3.3 Step 6. Evaluation of the internal structure of PROMs: structural validity,
internal consistency, and cross‐cultural validity\measurement invarianceInternal structure refers to how the different items in a PROM are related, which isimportant to know for deciding how items might be combined into a scale or subscale(6). This step concerns an evaluation of structural validity (including unidimensionality)
using factor analyses or IRT or Rasch analyses; internal consistency; and cross‐culturalvalidity and other forms of measurement invariance (using differential item functioning(DIF) analyses or Multi‐Group Confirmatory Factor Analyses (MGCFA)). Here we arereferring to testing of existing PROMs; not further refinement or development of new
PROMs. These three measurement properties focus on the quality of items and therelationships between the items in contrast to the remaining measurement properties
discussed in Step 7, who mainly focus on the quality of scales or subscales. We
recommend to evaluate internal structure directly after evaluating the content validityof a PROM. As evidence for structural validity (or unidimensionality) of a scale orsubscale is a prerequisite for the interpretation of internal consistency analyses (i.e.
Cronbach’s alpha’s), we recommend to first evaluate structural validity, to be followed
by internal consistency and cross‐cultural validity\measurement invariance.
Step 6 is only relevant for PROMs that are based on a reflective model that assumes that
all items in a scale or subscale are manifestations of one underlying construct and are
expected to be correlated. An example of a reflective model is the measurement ofanxiety; anxiety manifests itself in specific characteristics, such as worrying thoughts,
panic, and restlessness. By asking patients about these characteristics, we can assess thedegree of anxiety (i.e. all items are a reflection of the construct) (23). If the items in ascale or subscale are not supposed to be correlated (i.e. a formative model), theseanalyses are not relevant and Step 6 can be omitted. In other words, if factor analysis, or
IRT or Rasch analysis is performed on a PROM based on a formative model, these results
can be ignored, as the results are not interpretable.
If it is not reported whether a PROM is based on a reflective or formative model, thereviewers need to decide on the content of the PROM whether it is likely based on areflective or a formative model. Unfortunately, it is not always possible to decide
afterwards if the instrument is based on a reflective or formative model and thus
whether structural validity is relevant. If a study included in the review presents a factor
analysis, IRT or Rasch analysis and you are in doubt whether the (sub) scale is reflective,
or whether it might be mixed (both reflective and formative items within a subscale), we
recommend to consider it a reflective model. Subsequently, the quality of the study andthe quality of the results are rated.
The evaluation of structural validity, internal consistency and cross‐culturalvalidity\measurement invariance consists of three sub steps as described in Chapter 3.1(see Figure 4). First, the risk of bias in each study on structural validity, internalconsistency and cross‐cultural validity\measurement invariance is assessed using the
COSMIN Risk of Bias checklist (see Chapter 5 for detailed instructions for how to rate thestandards in each box). Second, data is extracted on the characteristics of the PROM(s),on characteristics of the included patient sample(s), and on the results of themeasurement properties (see Appendices 3‐7). The result per study is then evaluatedagainst the criteria for good measurement properties. Third, all results per
measurement property of a PROM are quantitatively pooled or qualitatively
38

COSMIN manual for systematic reviews of PROMs
summarized, and this pooled or summarized result is again evaluated against thecriteria for good measurement properties to get an overall rating (Table 4). Finally, thequality of the evidence is graded using the modified GRADE approach, as described in
Chapter 3.1.3.
The risk of bias in a study on internal consistency depends on the available evidence forstructural validity because unidimensionality is a prerequisite for the interpretation ofinternal consistency analyses (i.e. Cronbach’s alpha’s). Therefore, the quality of evidence
for internal consistency cannot be higher than the quality of evidence for structuralvalidity. Note that in a systematic review of PROMs, the conclusion about structuralvalidity may come from different studies, even from studies that were conducted afterstudies on internal consistency.
We recommend to take the quality of evidence for structural validity as a starting point
for determining the quality of evidence for internal consistency, and to downgrade
further for risk of bias, inconsistency (i.e. unexplained inconsistency of results acrossstudies), imprecision (i.e. total sample size of the available studies), and indirectness if
needed.
A Cronbach’s alpha based on a scale which is not unidimensional is difficult to interpret.
We therefore recommend to ignore results of studies on internal consistency of scales
when there is evidence that these scales are not unidimensional. For example, if results
on structural validity show that a scale has three factors, internal consistency of each of
those three subscales is relevant. Cronbach’s alpha’s on a total score can be ignored (andclinicians and researchers should be encouraged not to use these total scores) unlessthere is prove of unidimensionality of the total score, e.g. by a higher order or bi‐factor
CFA.

39

COSMIN manual for systematic reviews of PROMs
3.4 Step 7. Evaluation of the remaining measurement properties: reliability,
measurement error, criterion validity, hypotheses testing for construct validity,
and responsiveness
Subsequently, the remaining measurement properties (reliability, measurement error,
criterion validity, hypotheses testing for construct validity, and responsiveness) shouldbe evaluated. Unlike content validity and internal structure, the evaluation of thesemeasurement properties mainly assess the quality of the scale or subscale as a whole,
instead of the items.
Issues regarding validity to be decided upon a priori by the review teamFor assessing the quality of hypotheses testing for construct validity and for theconstruct approach for responsiveness, the review team should a priori decide on two
issues:First, the review team should decide whether there is a gold standard available formeasuring the construct of interest in the target population. If there is no gold standard(in principle, there is no gold standard available of a PROM, only the long version of ashortened PROM can be considered as a gold standard), the review team should not use
the box for criterion validity and the box for the criterion approach for responsivenessin the systematic review. All evidence for validity presented in the included studies will
be considered as evidence for construct validity or the construct approach forresponsiveness. If an outcome measurement instrument is considered a gold standard,studies comparing the PROM to this particular instrument are considered as evidence
for criterion validity or criterion approach for responsiveness.
Second, for interpreting the results of studies on hypotheses testing for constructvalidity, and on studies using a construct approach for evaluating responsiveness, thereview team should formulate a set of hypotheses about expected relationships between
the PROM under review and other well‐defined and high quality comparator
instruments which are used in the field. If possible, also some hypotheses about
expected differences between subgroups can be formulated in advance. This way, thestudy results are compared against the same hypotheses.
The expected direction (positive or negative) and magnitude (absolute or relative) of thecorrelations or differences should be included in the hypotheses (for examples we referto other publications(34‐37)). Without this specification it is difficult to decide
afterwards whether the results are in accordance with the hypothesis or not. For
example, review team could state that they expect a correlation of at least 0.50 between
the PROM under study and the comparator instrument that intend to measure the same
construct. Or that they expected that chronically ill patients score on average score 10
points higher compared to acute patients on the PROM under consideration. The
hypotheses may also concern the relative magnitude of correlations. For example, thereview team could state that they expect that the score on PROM A correlates at least
0.10 points higher with the score on instrument (e.g. a PROM) B than with the score on
instrument C. In Table 8 some generic hypotheses are presented that could beconsidered as a starting point for formulating hypotheses for the review (23).

40

COSMIN manual for systematic reviews of PROMs
Table 8. Generic hypotheses to evaluate construct validity and responsiveness
Generic hypotheses*:
1.
Correlations with (changes in) instruments measuring similar
constructs should be ≥0.50.
2.
Correlations with (changes in) instruments measuring related, butdissimilar constructs should be lower, i.e. 0.30‐0.50.
3.
Correlations with (changes in) instruments measuring unrelated
constructs should be <0.30.
4.
Correlations defined under 1, 2, and 3 should differ by a minimum of
0.10.
5.
Meaningful changes between relevant (sub)groups (e.g. patients withexpected high vs low levels of the construct of interest)
6.
For responsiveness, AUC should be ≥ 0.70
*Reproduced with approval from De Vet et al. (23)
AUC = Area Under the Curve with an external measure of change used as the ‘gold
standard’
Hypotheses can be based on literature (including the included studies in the review) and(clinical) experiences of the review team members. In the original COSMIN checklist, oneof the standards included in the box on hypotheses testing for construct validity andresponsiveness was about whether hypotheses were stated in the study, which often
lead to an ‘inadequate’ rating. By recommending the review team to formulate
hypotheses in the new methodology, the methodological quality of an included study is
no longer dependents upon whether or not hypotheses were formulated in thesestudies. All results found in included articles can now be compared against this same setof hypotheses, and more results can be used to build a conclusion about the constructvalidity of a PROM.
Additional hypotheses may need to be formulated during the review process, dependingon the observed results in the included studies. We consider it not a problem if the
hypotheses are not all defined a priori, as long as the hypotheses are reported in thereview, making the methodology transparent.
If it is impossible to formulate a good hypothesis for a specific analysis (e.g. about an
expected magnitude of difference between two subgroups), and the authors of the study
did not state their expectations, the review team can consider ignoring these results, as
it is unknown how the results should be interpreted.
When assessing responsiveness, one of the most difficult tasks is formulating
challenging hypotheses. By challenging hypotheses we aim to show that the instrument
truly measures changes in the construct(s) it purports to measure. This means that the
instrument should measure changes in the purported construct(s), but also that itshould measure the right amount of change, i.e. it should not under‐ or overestimate thereal change in the construct that has occurred. This latter aspect is often overlooked inassessing responsiveness. For example, the hypotheses can concern expected meandifferences between changes in groups or expected correlations between changes in thescores on the instrument and changes in other variables, such as scores on other
instruments, or demographic or clinical variables. Hypotheses about expected effect size(ES) or similar measures such as standardized response mean (SRM) can also be used,
but only when an explicit hypothesis (and rationale) for the expected magnitude of the41

COSMIN manual for systematic reviews of PROMs
effect size is given. The hypotheses may also concern the relative magnitude ofcorrelations, for example a statement that change in instrument A is expected tocorrelate higher with change in instrument B than with change in instrument C.
Evaluating the measurement properties and grading the quality of the evidence
The evaluation of the measurement properties consists of the three sub steps, as
described earlier in Chapter 3.1 (Figure 4).
First, the risk of bias in each study on these measurement properties is assessed using
the COSMIN Risk of Bias checklist (see Chapter 5 for detailed instructions for how to rate
the standards).
Second, data is extracted on the characteristics of the PROM(s), on characteristics of the
included study population(s), and on results on the measurement properties (seeAppendices 3‐6), and the result per study is evaluated against the criteria for goodmeasurement properties (see Table 4).
Third, all results per measurement property of a PROM are quantitatively pooled or
qualitatively summarized, and this pooled or summarized result is evaluated against thecriteria for good measurement properties to get an overall rating for the measurement
property (Table 4). Finally, the quality of the evidence is graded using the modified
GRADE approach, as described in Chapter 3.1.
Specific aspects to consider for measurement error
When applying the criteria for good measurement error, information is needed on the
Smallest Detectable Change (SDC) or Limits of Agreement (LoA), as well as on the
Minimal Important Change (MIC). This information may come from different studies.
The MIC should have been determined using an anchor‐based longitudinal approach(38‐41). The MIC is best calculated from multiple studies and by using multiple anchors.
It is up to the review team to decide whether the quality of the evidence should be
downgraded (e.g. one level) when there is information available on the MIC value fromonly one study, or when the study on the MIC is not adequately performed (e.g.
insufficient validity of the anchor)(42, 43). When a MIC value is determined using a
distribution‐based method, in fact, the MIC value does not reflect what patients consider
important, but is rather an indicator of the measurement error of the PROM. Results
found in such studies should either be ignored or considered as evidence on
measurement error (23). If not enough information is available to judge whether the
SDC or LoA is smaller than the MIC, we recommend to just report the information that isavailable on the SDC or LoA without grading the quality of evidence (note that
information on the MIC alone provides information on the interpretability of a PROM,
see Chapter 4.1).
Specific aspects to consider for hypotheses testing and responsivenessIn studies on construct validity and responsiveness, different study designs may havebeen used, i.e. comparisons with other outcome measurement instrument, comparisons
between groups, or comparisons before and after an intervention. Therefore, the boxes
Hypotheses testing for construct validity and Responsiveness both consist of two andfour parts, respectively (see Chapter 5.7 and 5.8). The methodological quality of eachpart will be rated separately, using the “worst score counts” method (12).
In general, each result (e.g. a correlation between scores of two outcome measurement
instruments, or an ES) could be considered as a single study. When for example, thePROM under study is compared to four different outcome measurement instruments,

42

COSMIN manual for systematic reviews of PROMs
four correlations are computed, and this could be considered as four single ‘studies’.
Basically, the box Hypotheses testing should be completed four times. However, wheneach standard for all four ‘studies’ will be rated the same (e.g. the constructs measured
by all four comparison outcome measurement instruments are clear and all comparison
instruments are of good quality) the ratings can be combined into one risk of bias rating(i.e. the methodological quality of the studies is ‘very good’). Next, each result iscompared against the criterion (i.e. whether or not the hypothesis was confirmed), andreported in the results table (see Appendix 7). If the methodological quality of each‘study’ is not similar, it could be assessed separately. In Appendix 7, an example is given(called ‘ref 6’ under hypotheses testing) of an article in which three hypotheses were
tested in adequate studies and the results are in accordance with the hypotheses, andthree other hypotheses were tested in inadequate studies, and one of the results was inaccordance with the hypotheses and two results were not in accordance with the
hypotheses.
Moreover, in one article both hypotheses about comparison with other instruments
could be tested as well as hypotheses about expected differences between subgroups. In
that case, different parts of the box Hypotheses testing for construct validity will be used(part a and b, respectively). However, when the ratings are the same (for methodologicalquality), this could be reported together.
Next, the quality of the evidence should be determined. We recommend to determine
this similarly as for the other measurement properties. In this step, we consider all
results reported in one article together as one study. We do not grade the evidence per
hypothesis or per study design, because otherwise high evidence for hypotheses testing
can easily be reached.
An example is provided in Appendix 7. Here, 11 out of 13 results are rated as sufficient.
We conclude that we have consistent findings for sufficient hypotheses testing forconstruct validity, as 85% of the results are in line with the hypotheses. We have onevery good study. Therefore, we do not downgrade for risk of bias. Depending on
imprecision (i.e. total sample size of the available studies), and indirectness we could
downgrade, if needed.
In studies on construct validity and responsiveness some results may provide stronger
evidence than other results. For example, correlations with PROMs measuring similar
constructs (i.e. convergent validity) can be considered as providing more evidence thancorrelations with PROMs measuring dissimilar constructs. This can be taken into
account when determining the pooled or summarized result for construct validity,
especially when results are inconsistent. For example, when results about comparisonswith PROMs measuring similar constructs are consistently not in accordance with the
hypotheses, while results about comparison with dissimilar constructs tend to be inaccordance with the hypotheses, the review team could decide to put more emphasis on
the results of the hypotheses concerning comparisons with instruments measuring
similar constructs.

43

COSMIN manual for systematic reviews of PROMs

4. Part C steps 8‐10: Selecting a PROM
Part C steps 8‐10 concerns the description of data on interpretability and feasibility of
the PROMs (step 8), formulating recommendations based on all evidence (step 9) andthe reporting of the systematic review (step 10).
4.1 Step 8: Describe interpretability and feasibility
Interpretability is defined as the degree to which one can assign qualitative meaning(that is, clinical or commonly understood connotations) to a PROM’s quantitative scoresor change in scores (21). Both the interpretability of single scores and theinterpretability of change scores is informative to report in a systematic review. Theinterpretation of single scores can be outlined by providing information on the
distribution of scores in the study population or other relevant subgroups, as it may
reveal clustering of scores, and it can indicate floor and ceiling effects. Theinterpretability of change scores can be enhanced by reporting MIC values andinformation on response shift (referring to changes in the meaning of one's self‐
evaluation of a target construct as a result of: (a) a change in the patient's internal
standards of measurement (i.e. scale recalibration); (b) a change in the patient's values(i.e. the importance of component subdomains constituting the target construct); or (c) aredefinition of the target construct (i.e. reconceptualization)(44)) (see Appendix 5).
Feasibility is defined as the ease of application of the PROM in its intended context ofuse, given constraints such as time or money (45), for example, completion time, cost of
an instrument, length of the instrument, type and ease of administration (see Appendix
6 for all aspects of feasibility) (26). Feasibility applies to patients completing the PROM
(self‐administered) and researchers or clinicians who interview or hand over the PROM
to patients. The concept ‘feasibility’ is related to the concept ‘clinical utility’, whereas
feasibility focusses on PROMs, clinical utility refers to an intervention (46).
Interpretability and feasibility are not measurement properties because they do notrefer to the quality of a PROM. However, they are considered important aspects for awell‐considered selection of a PROM. In case there are two PROMs that are very difficult
to differentiate in terms of quality, it is recommended that feasibility aspects (e.g. time
aspects and budget restrictions) should be taken into consideration in the selection of
the most appropriate instrument. Floor and ceiling effects can indicate insufficient
content validity, and can result in insufficient reliability.
Interpretability and feasibility are only described, and not evaluated, as these are noformal measurement properties (i.e. they do not tell us something about the quality of
an instrument, rather they are important aspects that should be considered in theselection of the most suitable instrument for a specific purpose).

44

COSMIN manual for systematic reviews of PROMs
4.2 Step 9: formulate recommendations
Recommendations on the most suitable PROM for use in an evaluative application areformulated with respect to the construct of interest and study population. To come to anevidence‐based and fully transparent recommendation, we recommend to categorize theincluded PROMs into three categories:(A) PROMs with evidence for sufficient content validity (any level) AND at least low
quality evidence for sufficient internal consistency;(B) PROMs categorized not in A or C.
(C) PROMs with high quality evidence for an insufficient measurement property
PROMs categorized as ‘A’ can be recommended for use and results obtained with these
PROMs can be trusted. PROMs categorized as ‘B’ have potential to be recommended foruse, but they require further research to assess the quality of these PROMs. PROMs
categorized as ‘C’ should not be recommended for use. When only PROMs categorized as‘B’ are found in a review, the one with the best evidence for content validity could be theone to be provisionally recommended for use, until further evidence is provided.
Justifications should be given to as why a PROM is placed in a certain category, anddirection should be given on future validation work, if applicable. To work towards
standardization in outcome measurement (e.g. core outcome set development) and to
facilitate meta‐analyses of studies using PROMs, we recommend to subsequently advise
on one most suitable PROM (5). This recommendation does not only have to be based on
the evaluation of the measurement properties, but may also depend on interpretability
and feasibility aspects.
4.3 Step 10: report the systematic review
In accordance with the PRISMA Statement (18), we recommend to report the following
information:(1) the search strategy (for example on a website or in the (online) supplemental
materials to the article at issue), and the results of the literature search and selection of
the studies and PROMs, displayed in the PRIMSA flow diagram (including the final
number of articles and the final number of PROMs included in the review) (Appendix 2);(2) the characteristics of the included PROMs, such as name of the PROMs, reference tothe article in which the development of the PROM is described, constructs being
measured, language and study population for which the PROM was developed, intended
context(s) of use, available language version of the PROM, number of scales or subscales,
number of items, response options, recall period, interpretability aspects, and feasibility
aspects (Appendix 3);(3) the characteristics of the included study populations, such as geographical location,
language, important disease characteristics, target population, sample size, age, gender,and setting (Appendix 4);(4) the methodological quality ratings of each study per measurement property per
PROM (i.e. very good, adequate, doubtful, inadequate), the results of each study, and the45

COSMIN manual for systematic reviews of PROMs
accompanying ratings of the results based on the criteria for good measurement
properties (sufficient (+) / insufficient (‐) / indeterminate (?)) (Appendix 7). This table
could be published for example as Appendix or supplemental material on the website of
the journal only;(5) a Summary of Findings (SoF) table per measurement property, including the pooledor summarized results of the measurement properties, its overall rating (i.e. sufficient(+) / insufficient (‐) / inconsistent (±)/ indeterminate (?)), and the grading of the qualityof evidence (high, moderate, low, very low). An example of a SoF table can be found in
Appendix 8. These SoF tables (i.e. one per measurement property) will ultimately be
used in providing recommendations for the selection of the most appropriate PROM fora given purpose or a particular context of use. Note that these tables can be used in thedata extraction process throughout the entire review.

46

COSMIN manual for systematic reviews of PROMs

5. COSMIN Risk of Bias checklist
In this chapter we elaborate on all standards included in boxes 3 to 10 of the COSMIN
Risk of Bias Checklist. An elaboration of all standards included in box 1 PROM
Development and box 2 Content validity is provided in the users’ manual for guidanceon how to evaluate the content validity of PROMs. This manual can be found on the
COSMIN website (33).
Each box contains an item asking if there were any other important othermethodological flaws that are not covered by the checklist, but that may lead to biased
results or conclusions. For some of the boxes we will provide examples of such flaws
below.
5.1 Assessing risk of bias in a study on structural validity (box 3)
Structural validity refers to the degree to which the scores of a PROM are an adequate
reflection of the dimensionality of the construct to be measured(21) and is usuallyassessed by factor analysis.
Box 3. Structural validity
Does the scale consist of effect indicators, i.e. is it based on a reflective model?
Does the study concern unidimensionality or structural validity?
validity

yes / nounidimensionality / structural

The box on structural validity starts with two questions that can help the reviewer todecide whether this measurement property is relevant for the instrument under study(i.e. question on reflective model), or to become aware of the specific purpose of thestudy (i.e. question on unidimensionality or structural validity). Both questions are notstandards and they are not used in the worst score counts method.
Structural validity is only relevant for instruments that are based on a reflective model.
A reflective model is a model in which all items are a manifestation of the sameunderlying construct. These kind of items are called effect indicators. These items are
expected to be highly correlated and interchangeable. Its counterpart is a formativemodel, in which the items together form the construct. These items do not need to becorrelated. Therefore, structural validity is not relevant for items that are based on aformative model(24, 47, 48). For example, stress could be measured by asking about theoccurrence of different situations and events that might lead to stress, such as job loss,
death in a family, divorces etc. (49). These events do not need to be correlated, thus
structural validity is not relevant for such an instrument.
Often, authors do not explicitly describe whether their instrument is based on areflective or formative model. To decide afterwards which model is used, one can do a
simple “thought test”. With this test one should consider whether all item scores are
expected to change when the construct changes. If yes, the construct can be considered areflective model. If not, the PROM is probably based on a formative model (24, 47).

47

COSMIN manual for systematic reviews of PROMs
It is not always possible to decide afterwards if the instrument is based on a reflective or
formative model and thus whether structural validity is relevant. In this case, we
recommend to complete the box to assess the quality of the analyses that were
performed in the included studies.
The measurement property Structural validity refers to the model fit of a factor analysis
on all items in an outcome measurement instrument, e.g. to confirm a 3‐factor model for
an instrument with three subscales. Unidimensionality refers to whether the items in ascale or subscale measure a single construct. It is an assumption for internal consistency
or IRT/Rasch analyses. For the purpose of checking unidimensionality each subscalefrom a multidimensional PROM can separately be evaluated with a factor analysis or
IRT/Rasch analyses. An evaluation of unidimensionality is sufficient for theinterpretation of an internal consistency analysis and IRT/Rasch analysis, but it does notprovide evidence for structural validity as part of construct validity of amultidimensional PROM. That is, because it does not provide evidence, for example, thatan instrument with three subscales indeed measures three different constructs. A
question was added to the box on Structural validity about whether the aim of the study
was to assess unidimensionality or to assess structural validity. Although the standards
for assessing unidimensionality and structural validity are the same, the purpose isdifferent and the conclusion about the PROM can be different and reviewers should takethis into account when reporting the results of such studies in a systematic review. This
question is not a standard and it is not used in the worst score counts method. It is onlya help for the reviewers to be aware of which purpose was served in the study.
Statistical methods
1 For CTT: Was exploratory or
confirmatory factor analysis

very goodadequate

doubtful

Confirmatory Exploratory
factor analysis factor analysis
performed
performed

performed?

inadequateNA

No exploratory Not
or confirmatory applica
factor analysis
bleperformed

Standard 1. To determine the structure of the instrument, a factor analysis is thepreferred statistic when using CTT. Although confirmatory factor analysis is preferredover explorative factor analysis, both could be useful for the evaluation of structuralvalidity. Confirmative factor analysis tests whether the data fit a premeditated factor
structure(50). Based on theory or previous analyses a priori hypotheses are formulated
and tested. Explorative factor analysis can be used when no clear hypotheses exist about
the underlying dimensions (50).
Statistical methods
2 For IRT/Rasch: does the chosen model
fit to the research question?

Chosen model
fits well to theresearchquestion

Assumablethat thechosenmodel fits
well to theresearchquestion

Doubtful if
the chosenmodel fits
well to theresearchquestion

ChosenNAmodel does
not fit to theresearchquestion48

COSMIN manual for systematic reviews of PROMs
Standard 2: An example of a model that does not fit the research question is when
follow‐up data are combined but not analysed using a multi‐level IRT model.
Statistical methods

very goodadequate

doubtfulinadequate

3 Was the sample size included FA: 7 times thenumber ofin the analysis adequate?
items, and≥100

FA: at least 5
the times
number ofitems, and≥100; OR atleast 6 times
the number of
items, but<100

FA: 5 times the FA: < 5 times
number of
the number of
items, but
items
<100

Rasch/1PLmodels: ≥ 200
subjects

Rasch/1PLmodels: 100‐
199 subjects

Rasch/1PLmodels: 50‐99
subjects

Rasch/1PLmodels: < 50
subjects

2PL
parametric IRTmodels OR
Mokken scale
analysis: ≥1000 subjects

2PL
parametric IRTmodels OR
Mokken scale
analysis: 500‐
999 subjects

2PL
parametric IRTmodels OR
Mokken scale
analysis: 250‐
499 subjects

2PL
parametric IRTmodels OR
Mokken scale
analysis: < 250
subjects

NAStandard 3. Factor analyses and IRT/Rasch analyses require large sample sizes.
The proposed requirements were based on Comrey (51), Brown (chapter 10) (52),Embretson and Reise (53), Edelen and Reeve (54), Reise and Yu (55), Reise et al. (56),and Linacre (57).
These sample size requirements are rules of thumb, and are context related. Sample size
requirements increase as a result of the complexity of the mode l (Edelen et al. 2007).
Moreover, depending on the research question different levels of precision may be
acceptable, which relate again to the sample size needed (Edelen et al. 2007). Anotherrelated consideration is the sampling distribution of the respondents. The sample shouldreflect the population of interest and contain enough respondents with extreme scores
so that items even at extreme ends of the construct continuum will have reasonable
standard errors associated with their estimated parameters. (Edelen et al. 2007).
Other
4 Were there any other important flaws
in the design or statistical methods ofthe study?

No otherimportantmethodologic
al flaws

Other
minor
methodolo
gical flaws(e.g.
rotation
methodnotdescribed)

Other
importantmethodolog
ical flaws(e.g.
inappropria
te rotation
method)We have not defined specific requirements for factor analyses, such as the choice of the
explorative factor analysis (principal component analysis or common factor analysis),49

COSMIN manual for systematic reviews of PROMs
the choice and justification of the rotation method (e.g. orthogonal or oblique rotation),or the decision about the number of relevant factors. Such specific requirements aredescribed by e.g. Floyd & Widaman (50) and de Vet et al. (58). When there are serious
flaws in the quality of the factor analysis, we recommend to rate item 4 with “doubtful”
or “inadequate”.
5.2 Assessing risk of bias in a study on internal consistency (box 4)Internal consistency refers to the degree of interrelatedness among the items and isoften assessed by Cronbach’s alpha. Cronbach’s alpha’s can be pooled across studies
when the results are sufficiently similar. For an example, we refer to a study performedby Collins and colleagues (8).
For an appropriate interpretation of the internal consistency parameter, the items
together should form a unidimensional scale or subscale. Internal consistency andunidimensionality are not the same. Internal consistency is the relatedness among theitems (59). Unidimensionality refers to whether the items in a scale or subscale measurea single construct. It is a prerequisite for a clear interpretation of the internalconsistency statistics (59, 60), and can be investigated for example by factor analysis(61) or IRT methods (see structural validity, box 3).
Box 4. Internal consistency
Does the scale consist of effect indicators, i.e. is it based on a reflective model? 1

yes / noThe first question in the Box Internal consistency concerns the relevance of theassessment of the measurement property internal consistency for the PROM understudy. The internal consistency statistic only gets an interpretable meaning, when theinterrelatedness among the items is determined of a set of items that together form areflective model (59, 60). See also box 3 for an explanation about reflective models.
Design requirements
1 Was an internalconsistency statistic
calculated for each
unidimensional scaleor subscale
separately?

very goodInternal
consistency
statistic
calculated for
eachunidimensional
scale or subscaleadequate

Doubtfulinadequate

NA

UnclearInternal consistency
whether scale statistic NOTor sub scale is calculated on
unidimensional unidimensional scale

Standard 1. The internal consistency coefficient should be calculated for each
unidimensional subscale separately. Information on unidimensionality (or structuralvalidity) can be either found in the same study or in other studies. Based on the at least
low quality evidence for structural validity we determine whether or not the items fromthe (sub) scales form an unidimensional scale.
When an internal consistency parameter is calculated per subscale as well as for a
multidimensional total scale (for example the total score of four subscales) this latter
result can be ignored, as it cannot be interpreted. If an internal consistency parameter isonly reported for a multidimensional total scale, this standard should be rated
50

COSMIN manual for systematic reviews of PROMs‘inadequate’. If no information is found in the literature on the structural validity or
unidimensionality of a PROM, this standard can be rated with ‘doubtful’. In this case, we
recommend to report the results found on internal consistency, without drawing aconclusion.
Statistical methods
2 For continuous scores: Was
Cronbach’s alpha or omega
calculated?

Cronbach’s
alpha, orOmega
calculated

3 For dichotomous scores:
Cronbach’s
Was Cronbach’s alpha or KR‐ alpha or KR‐
20 calculated?
20 calculated
4 For IRT‐based scores: Was
standard error of the theta(SE (θ)) or reliability
coefficient of estimatedlatent trait value (index of(subject or item) separation)
calculated?

SE(θ) orreliability
coefficient
calculated

Only item‐
total
correlations
calculated

No Cronbach’s alphaand no item‐total
correlations
calculated

Only item‐
total
correlations
calculated

No Cronbach’s alphaNa
or KR‐20 and no item‐
total correlations
calculatedSE(θ) or reliability
coefficient NOT
calculated

Na

NaStandards 2 and 3. Studies based on CTT should report Cronbach’s alpha or Omega
values (62).
Standard 4. Several reliability indices are available that are based on IRT/Raschanalyses. These indices are based on one score. Examples are the standard error oftheta, and index of person or subject separation.
5.3 Assessing risk of bias in a study on cross‐cultural validity\ measurement
invariance (box 5)
Cross‐cultural validity refers to the degree to which the performance of the items on atranslated or culturally adapted instrument are an adequate reflection of theperformance of the items of the original version of the instrument. To assess thismeasurement property data from at least two different groups is required, e.g. a Dutch
and an English sample, or males and females. We recommend to complete this box for
studies that have evaluated cross‐cultural validity for PROMs across culturally different
populations. We interpret ‘culturally different population’ broadly. We do not only
consider different ethnicity or language groups as different cultural populations, butalso other groups such as different gender or age groups, or different patient
populations. Cross‐cultural validity is evaluated by assessing whether the scale ismeasurement invariant or whether or not Differential Item Functioning (DIF) occurs.
Measurement Invariance (MI) and non‐DIF refer to whether respondents from different
groups with the same latent trait level (allowing for group differences) respondsimilarly to a particular item.

51

COSMIN manual for systematic reviews of PROMs

Box 5. Cross‐cultural validity\Measurement invariance
Design requirements
1 Were the samples similar for
relevant characteristics except
for the group variable?

very goodadequate

doubtfulinadequate

Evidence
provided thatsamples weresimilar for
relevant
characteristics
except groupvariableStated (but noevidence
provided) thatsamples weresimilar for
relevant
characteristics
except groupvariable

Unclearwhethersamples weresimilar for
relevant
characteristics
except groupvariable

Samples were
NOT similar
for relevantcharacteristics
except groupvariableNAStandard 1. When evaluating cross‐cultural validity, scores of two (or more) groups are
directly compared in one statistical model. This could be language groups (e.g. when the
Dutch version of a PROM is being compared to the English version of the same PROM),or groups based on another variable, such as males versus females, or healthy andchronically ill people. Except from the group variable, the two groups should be similar
for relevant characteristics, such as disease severity, age, etc. In one study gender could
be the group variable, and in another study (e.g. comparing two language groups)
gender is considered a relevant characteristic of the groups and is expected to be
similarly distributed across the groups. It is up to the review team to judge whether all
relevant characteristics are similarly distributed across the groups.
Statistical methods
2 Was an appropriate approach used
to analyse the data?

A widely
Assumable
recognized that theor wellapproachjustifiedwasapproach appropriate,
was used but notclearly
described

Not clearapproach
Not
what
NOT
applicableapproachappropriate
was used or
doubtfulwhethertheapproachwasappropriateStandard 2. Adequate approached for assessing cross‐cultural validity using CTT areregression analyses or confirmatory factor analysis (CFA). For an explanation of the useof ordinal regression models, we refer to Crane et al. (63) or Petersen et al. (64). For an
explanation of CFA to investigate measurement invariance we refer to Gregorich (65).
An adequate approach for assessing cross‐cultural validity using IRT methods is
Differential Item Functioning (DIF) analyses (66).

52

COSMIN manual for systematic reviews of PROMs

Statistical methods
3 Was the sample size
included in the analysis
adequate?

Regression
analyses or
IRT/Rasch based
analyses: 200
subjects pergroup

150 subjects
per group

100
subjects
per group

MGCFA*: 7 times
the number of
items, and ≥100

5 times thenumber ofitems, and≥100; OR 5‐7
times thenumber ofitems, but<100

5 times the <5 times thenumber of number of
items, but items
<100

< 100
subjects per
group

Standard 3. CFA, IRT analysis or regression analysis require high sample sizes to obtainreliable results. As the results of studies cannot be pooled, a sample size requirement
was included as a standard. These recommendations are based on a publication by Scottet al. (67).
5.4 Assessing risk of bias in a study on reliability (box 6)
Reliability refers to the proportion of the total variance in the measurements which isdue to ‘true’ differences between patients. The word ‘true’ must be seen in the context of
CTT, which states that any observation is composed of two components – a true scoreand error associated with the observation. ‘True’ is the average score that would beobtained if the scale was administered an infinite number of times to the same person. It
refers only to the consistency of the score, and not to its accuracy (22). Reliability canalso be explained as the ability of a PROM to distinguish between patients. Within ahomogeneous group, it is hard to distinguish between patients (23). An important
assumption made in a reliability study (and in a study on measurement error) is thatpatients are stable on the construct to be measured between the repeatedmeasurements.
Box 6. Reliability
Design requirements

very good

Evidence
1 Were patients stable in the interim
period on the construct to be measured? provided
thatpatients
were stable

adequate

doubtfulAssumable Unclear ifthatpatients
patients
were stablewere stable

inadequate

NA

Patients
were NOT
stable53

COSMIN manual for systematic reviews of PROMs
2 Was the time interval appropriate?

Timeinterval
appropriate

3 Were the test conditions similar for themeasurements? e.g. type ofadministration, environment,
instructions

Test
conditions
were
similar(evidence
provided)

Assumablethat test
conditions
were
similar

Doubtfulwhethertime interval
wasappropriate
or timeinterval was
not statedTimeinterval
NOT
appropriate

Unclear if
test
conditions
were similar

Testconditions
were NOT
similarStandard 1. Patients should be stable with regard to the construct to be measured
between the administrations. What “stable patients” are depends on the construct to bemeasured and the target population. Evidence that patients were stable could be, forexample, an assessment of a global rating of change, completed by patients orphysicians. When an intervention is given in the interim period, one can assume that(many of) the patients have changed on the construct to be measured. In that case, we
recommend to rate Standard 2 as “inadequate”.
Standard 2. The time interval between the administrations must be appropriate. The
time interval should be long enough to prevent recall bias, and short enough to ensurethat patients have not been changed on the construct to be measured. What an
appropriate time interval is, depends on the construct to be measured and the target
population. A time interval of about 2 weeks is often considered appropriate for theevaluation of PROMs (22).
Standard 3. A last requirement is that the test conditions should be similar. Test
conditions refer to the type of administration (e.g. a self‐administered questionnaire,interview, performance‐test), the setting in which the instrument was administered (e.g.
at the hospital, or at home), and the instructions given. These test conditions may
influence the responses of a patient. The reliability may be underestimated if the test
conditions are not similar. However, in clinical practice, different test condition might beused disorderly, and a specific research question could be whether the reliability of a
PROM is still appropriate when using the PROM under different test conditions. In thiscase, the test conditions do not need to be similar as they are supposed to vary as part of
the research aim of the study. In that case, this item can be rated with ‘very good’. See forexample, Van Leeuwen(68).

54

COSMIN manual for systematic reviews of PROMs

Statistical methods
4 For continuous scores: Wasan intraclass correlation
coefficient (ICC) calculated?

ICC calculated
and model orformula of the
ICC isdescribed

5 For dichotomous/nominal/
ordinal scores: Was kappa
calculated?

Kappa
calculated

6 For ordinal scores: Was aweighted kappa calculated?

WeightedKappa
calculated

7 For ordinal scores: Was theWeightingweighting scheme described? scheme
e.g. linear, quadratic
described

ICC calculated but
model or formulaof the ICC notdescribed or not
optimal.
Pearson orSpearmancorrelation
coefficient
calculated withevidence
provided that nosystematicchange has
occurredPearson orSpearmancorrelation
coefficient
calculated
WITHOUT
evidence
provided thatno systematicchange has
occurred or
WITH evidence
that systematicchange has
occurred

No ICC or
Na
Pearson orSpearmancorrelations
calculated

No kappa
calculated

Weightingscheme NOT
described

Unweighted
Kappa
calculated or
not described

Na

Na

Na

The preferred reliability statistic depends on the type of response options.
Standard 4. For continuous scores the intraclass correlation coefficient (ICC) is
preferred (22, 69). To assess the reliability of a PROM, often a straightforward test‐
retest reliability study design is chosen. The preferred ICC model in that case is the two‐way random effects model (70), in which in addition to the variance within patients thevariance (i.e. systematic differences) between time point is taken into account (23). The
use of the Pearson’s and Spearman’s correlation coefficient is considered doubtful when
it is not clear whether there are systematic differences occurred, because these
correlations do not take systematic error into account.
Standards 5, 6, and 7. For dichotomous scores or nominal scores the Cohen’s kappa is
the preferred statistical method (22). For ordinal scales partial chance agreement
should be considered, and therefore a weighted kappa (22, 71, 72) is preferred. A
description of the weights (e.g., linear or quadratic weights (73)) should be given.
Proportion agreement is considered not adequate as it is a parameter for measurement
error (see box 7).
Unweighted kappa considers any misclassification equally inappropriate. However, amisclassification of two adjacent categories may be less erroneous as a misclassificationof categories that are more apart from each other. A weighted kappa takes this into
account. However, the aim of a study could be to assess the reliability of any
misclassification, making a unweighted kappa as an appropriate parameter. In such a
study, Standard 7 can be rated as ‘very good’.
55

COSMIN manual for systematic reviews of PROMs
Other
8 Were there any other important
flaws in the design or statisticalmethods of the study?

No otherimportantmethodologicalflaws

Other minor
Other
methodological important
flaws
methodologicalflawsStandard 8: An example of another important flaw is when the administrations were notindependent. Independent administrations imply that the first administration has not
influenced the second administration. At the second administration the patient should
not have been aware of the scores on the first administration.
Another example could be, when the PROM was administered in an interview. Suppose
one experienced interviewer administers all first administrations of the interviews of all
patients, and the second administration was either done by an experienced interviewer
or by an unexperienced interviewer, and it was not clear which patient was interviewed
by which interviewer. Subsequently, this was not taken into account in the analysis.
Suppose a low ICC was found, it is unknown whether this is due to the different
characteristics of the interviewers (that could be improved by standardizing
requirements for interviewers), or due to an insufficient quality of the PROM.
5.5 Assessing risk of bias in a study on measurement error (box 7)
Measurement error refers to the systematic and random error of an individual patient’s
score that is not attributed to true changes in the construct to be measured.
Box 7. Measurement error
Design requirements

very goodadequate

doubtful

Inadequate

1 Were patients stable in theinterim period on theconstruct to be measured?

Patients werestable(evidence
provided)

Assumable Unclear ifthatpatients were
patients
stablewere
stablePatients were
NOT stable

2 Was the time interval
appropriate?

Time interval
appropriate

Doubtfulwhether timeinterval wasappropriate
or timeinterval was
not stated

Time interval
NOT
appropriate

3 Were the test conditions
similar for themeasurements? (e.g. type ofadministration, environment,
instructions)Test
conditions
were similar(evidence
provided)

Assumable Unclear if test
that test
conditions
conditions were similar
weresimilarNA

Test
conditions
were NOT
similarStandards 1‐3: see reliability

56

COSMIN manual for systematic reviews of PROMs
Statistical methods
SEM, SDC, or
4 For continuous scores: Was
the Standard Error of
LoA
Measurement (SEM), Smallest calculated
Detectable Change (SDC) or
Limits of Agreement (LoA)
calculated?

Possibletocalculate
LoA fromthe data
presented% positive
and negative
agreement
calculated%agreement
calculated

5 For dichotomous/ nominal/
ordinal scores: Was thepercentage (positive andnegative) agreement
calculated?

SEM
calculated
based onCronbach’s
alpha, or on
SD from
another
population% agreement
not calculated

Not
applicableNot
applicableStandard 4. The preferred statistic for measurement error in studies based on CTT is theStandard Error of Measurement (SEM) based on a test‐retest design. Note that thecalculation of the SEM based on Cronbach’s alpha is considered not appropriate, because
it does not take the variance between time points into account (74). Other appropriate
statistics for assessing measurement error are the Limits of Agreement (LoA) and the
Smallest Detectable Change (SDC) (23). Both parameters are directly related to the SEM.
Changes within the LoA or smaller than the SDC are likely to be due to measurement
error and changes outside the LoA or larger than the SDC should be considered as realchange in individual patients. Note that this does not indicate that these changes are alsomeaningful to patients. This depends on what change is considered important, which is
an issue of interpretability.
Standard 5. Often kappa is considered as a measure of agreement, however, kappa is ameasure of reliability (72). An appropriate parameter of measurement error (also called
agreement) of dichotomous/nominal/ordinal PROM scores is percentage agreement.
5.6 Assessing risk of bias in a study on criterion validity (box 8)
Criterion validity refers to the degree to which the scores of a PROM are an adequate
reflection of a ‘gold standard’. In a systematic review, the review team should determine
what reasonable ‘gold standards’ are for the construct to be measured. All studies
comparing a PROM to this accepted gold standard can be considered a study on criterion
validity.
Box 8. Criterion validity
Statistical methods

very good

1 For continuous scores: Were
correlations, or the area under thereceiver operating curve calculated?
2 For dichotomous scores: Weresensitivity and specificity
determined?

adequate

doubtfulinadequate

NA

Correlations
or AUC
calculated

Correlations
or AUC NOT
calculatedNa

Sensitivity andspecificity
calculated

Sensitivity and Na
specificity
NOT
calculated

57

COSMIN manual for systematic reviews of PROMs
Standards 2 and 3. When both the PROM and the gold standard have continuous scores,
correlation is the preferred statistical method. When the instrument scores arecontinuous and scores on the gold standard are dichotomous the area under the receiver
operating characteristic (ROC) is the preferred method, and when both scores are
dichotomous sensitivity and specificity are the preferred methods to use.
Other
3 Were there any other important flaws
in the design or statistical methods ofthe study?

No otherimportantmethodologicalflaws

Other minor
Other
methodological important
flaws
methodologicalflawsStandard 3: Bias may occur, for example, when a long version of a questionnaire iscompared to a short version, while the scores of the short version were computed using
the responses obtained with the longer version. In that case, this standard could be
rated as inadequate.
5.7 Assessing risk of bias in a study on hypotheses testing for construct validity(box 9)
Hypotheses testing for construct validity refers to the degree to which the scores of a
PROM are consistent with hypotheses (for instance with regard to internal relationships,
relationships to scores of other instruments, or differences between relevant groups)
based on the assumption that the PROM validly measures the construct to be measured.
Hypotheses testing is an ongoing, iterative process (34). The more specific the
hypotheses are and the more hypotheses are being tested, the more evidence isgathered for construct validity. Many types of hypotheses can be tested to evaluate
construct validity of a PROM. In general, these hypotheses concern comparisons withother outcome measurement instruments (that are not being considered as a gold
standard), or differences in scores between ‘known’ groups (e.g. patients with multiple
sclerosis (MS) and spasticity were expected to score statistically significantly higher
on an arm and hand functioning scale, as compared with patients with MS without
spasticity, because spasticity is a causative for activity limitations due to impairments of
the arm and hand (68)). The box on hypotheses testing is therefore structured in two
sections. To assess the risk of bias of studies comparing the PROM to comparison
instruments, part a of the box (items 1‐4) should be completed. For assessing the risk ofbias of known group validity studies, part b of the box (items 5‐7) should be completed.
The risk of bias for studies in which MI or DIF was investigated can be assessed using
Box 5 Cross‐cultural validity.
The overall rating is the lowest rating given on any applicable item per part of the box. If
in an article the PROM under study is being compared to both (a) comparison
instrument(s), as well as different subgroups are compared, the box should becompleted twice, and results are handled as two sub studies. However, when the ratings
are the same, they can be combined. For example, in Appendix 7 (overview Table)reference 5 tested 5 hypotheses, and was rated as ‘adequate’. These 5 hypotheses
concerned both comparisons with other instruments as well as expected differences
between groups.

58

COSMIN manual for systematic reviews of PROMs

Box 9. Hypotheses testing for construct validity
9a. Comparison with other outcome measurement instruments (convergent validity)
Designrequirements

very good

1 Is it clear what
the comparator
instrument(s)measure(s)?

Constructs
measured by thecomparator
instrument(s) isclear

2 Were themeasurement
properties of
the comparator
instrument(s)sufficient?

Sufficientmeasurement
properties of thecomparator
instrument(s) in a
population
similar to thestudy population

adequate

doubtfulinadequate

NA

Constructs
measured by thecomparator
instrument(s) is not
clear
Sufficientmeasurement
properties of thecomparator
instrument(s) but
not sure if these
apply to the study
populationSome
information on
measurement
properties of thecomparator
instrument(s) inany study
population

No information on
the measurement
properties of thecomparator
instrument(s), OR
evidence of
insufficientmeasurement
properties of thecomparator
instrument(s)Standards 1 and 2: When the PROM under review is compared to another instrument, it
should be known what the construct is that the comparator instruments measures, andthe instrument itself should be of sufficient quality. If not, it is difficult to interpret theresults of (e.g.) correlations.
This information can be reported in the article included in the review. However, when
this is not described, the review team could try to find additional information to decideon rating these standards.
When multiple comparator instruments are being used in a study, and for one
instrument the construct is clearly described, and for the other instrument it is notclearly described, we recommend to consider each analyses as a separate study, andcomplete the box multiple times. We recommend to do the same in case some of thecomparator instruments are of adequate quality, while other are of doubtful orinadequate quality.
Statistical methods
3 Were design andstatistical methods
adequate for the
hypotheses to betested?

Statistical
Assumable thatmethods applied statistical methods
appropriate
were appropriate

Statisticalmethods
applied NOT
optimalStatisticalmethods applied
NOT appropriateStandard 3. Appropriate statistical methods could be correlations between the PROM
and the comparator instrument. When, for example, Pearson correlations were
calculated, but the distribution of scores or mean scores (SD) were not presented, it
could be considered ‘assumable’ that the statistical methods were appropriate. P‐values
should not be used in testing hypotheses, because it is not relevant to examine whether
59

COSMIN manual for systematic reviews of PROMs
correlations statistically differ from zero (75). The validity issue is about whether the
direction and magnitude of a correlation is similar to what could be expected based on
the construct(s) that are being measured.
Another example of an appropriate method is when CFA or Structural Equation
Modeling (SEM) is performed over multiple scales or subscales which are considered tomeasure similar or different constructs to examine whether subscales measuring similar
constructs are more related to each other than subscales measuring different constructs.
9b. Comparison between subgroups (discriminative or known‐groups validity)
Design requirements
5 Was an adequate
description provided ofimportant characteristicsof the subgroups?

very goodadequate

doubtful

Adequate
description of
the importantcharacteristicsof thesubgroups

Adequate
description of mostof the importantcharacteristics ofthe subgroups

Poor of no
description of
the importantcharacteristicsof thesubgroups

inadequate NA

Standard 5. When PROM scores between two subgroups are compared, importantcharacteristics of the groups should be reported, and it should be clear on whichcharacteristics the groups differ, for example age, gender, disease characteristics,
language etc..
Statistical methods
6 Were design and statistical Statisticalmethods adequate for the methods
hypotheses to be tested?
appliedappropriate

Assumable thatstatistical methods
were appropriate

Statisticalmethods
applied NOT
optimalStatisticalmethods
appliedNOT
appropriateStandard 6. Many different hypotheses can be formulated and tested. The users of the
COSMIN Risk of Bias checklist have to decide whether or not the statistical methods
used in the article are adequate for testing the stated hypotheses. P‐values should be
avoided in testing hypotheses, because it is not relevant to examine whether
correlations statistically differ from zero (75). The validity issue is about whether the
direction and magnitude of a correlation is similar to what could be expected based on
the construct(s) that are being measured. When assessing differences between groups, it
is also less relevant whether these differences are statistically significant (which
depends on the sample size) than whether these differences are as large as expected.
5.8 Assessing risk of bias in a study on responsiveness (box 10)
Responsiveness refers to the ability of a PROM to detect change over time in theconstruct to be measured. Although responsiveness is considered to be a separate
measurement property from validity, the only difference between cross‐sectional(construct and criterion) validity and responsiveness is that validity refers to thevalidity of a single score, and responsiveness refers to the validity of a change score (21).
Therefore, the standards for responsiveness are similar to the standards of constructand criterion validity. Although the approach is similar to construct and criterion
60

COSMIN manual for systematic reviews of PROMs
validity, we do not to use the terms construct and criterion responsiveness, because
these terms are unfamiliar in the literature.
Similarly as for construct and criterion validity, the design requirements for assessing
responsiveness are divided in situations in which a gold standard is available (part a
standards 1‐5), situations in which hypotheses are tested between change scores on the
PROM under review and change scores on comparator instruments which are not gold
standards (part b standards 6‐9), and situations in which hypotheses are tested about
expected differences in changes between subgroups (part c standards 10‐12). In
addition, standards are included for situations in which hypotheses are tested about the
expected magnitude of an intervention (part d standards 13‐15).
Box 10. Responsiveness10a. Criterion approach (i.e. comparison to a gold standard)
Statistical methods
1 For continuous scores: Were
correlations between change
scores, or the area under the
Receiver Operator Curve(ROC) curve calculated?

very goodadequate

doubtful

Correlations
or Area under
the ROC Curve(AUC)
calculated

2 For dichotomous scales: Were Sensitivity andsensitivity and specificity
specificity(changed versus not changed) calculated
determined?

inadequate

NA

Correlations
or AUC NOT
calculatedna

Sensitivity and na
specificity
NOT
calculatedStandards 1‐2 are similar as the standards for criterion validity.
10b. Construct approach (i.e. hypotheses testing; comparison with other outcome measurement
instruments)
Design requirements

very good

4 Is it clear what thecomparator instrument(s)measure(s)?

Constructs
measured by thecomparator
instrument(s) isclear

5 Were the measurement
properties of thecomparator instrument(s)sufficient?

Sufficientmeasurement
properties of thecomparator
instrument(s) ina population
similar to thestudy population

adequate

doubtfulinadequate

NA

Constructs
measured bythecomparator
instrument(s)is not clear
Sufficientmeasurement
properties of
the comparator
instrument(s)but not sure if
these apply tothe study
populationSome
information on
measurement
properties ofthecomparator
instrument(s)in any study
populationNO
information onthemeasurement
properties ofthecomparator
instrument(s)OR evidence of
insufficientquality ofcomparator
instrument(s)61COSMIN manual for systematic reviews of PROMs
Statistical methods
6 Were design and statistical Statisticalmethods adequate for the methods applied
hypotheses to be tested?
appropriate

Assumable thatstatisticalmethods wereappropriate

Statisticalmethods
applied NOT
optimalStatisticalmethods
applied NOT
appropriate

Other
7 Were there any otherimportant flaws in thedesign or statisticalmethods of the study?

No otherimportantmethodologicalflaws

Other minor
Other
methodological important
flaws
methodologicalflawsStandards 4‐7 are similar as the standards for hypotheses testing for construct validity
part a.
10c. Construct approach: (i.e. hypotheses testing: comparison between subgroups)
Design requirements
8

Was an adequate
description provided ofimportant characteristicsof the subgroups?

very goodadequate

doubtful

Adequate
description of theimportantcharacteristics ofthe subgroups

Adequate
description ofmost of theimportantcharacteristicsof thesubgroups

Poor or no
description of
the importantcharacteristicsof thesubgroups

Statisticalmethods applied
appropriate

Assumable thatstatisticalmethods wereappropriate

Statisticalmethods
applied NOT
optimalinadequate

NA

Statistical methods
9

Were design andstatistical methods
adequate for the
hypotheses to be tested?

Statisticalmethods
applied NOT
appropriateStandards 8 – 9 are similar as the standards for hypotheses testing for construct validity
part b.
10d. Construct approach: (i.e. hypotheses testing: before and after intervention)
Design requirements
11 Was an adequate
description provided of
the intervention given?

very goodadequate

Adequate
description of theintervention

doubtfulinadequate

Poor
description oftheinterventionNO
description oftheintervention

Statisticalmethods
applied NOT
optimalStatisticalmethods
applied NOT
appropriate

NA

Statistical methods
12 Were design andstatistical methods
adequate for the
hypotheses to be tested?

Statisticalmethods applied
appropriate

Assumable thatstatisticalmethods wereappropriate

62

COSMIN manual for systematic reviews of PROMs
An effect size only has meaning as a measure of responsiveness if we know (or assume)
beforehand what the magnitude of the effect of the intervention is. If, for example, we
expect a large effect of the intervention on the construct measured by the PROM we cantest the hypothesis that the intervention results in an effect size of 0.8 or higher. But if
we expect a small effect of the intervention, we would not expect such a high effect size.
This example shows that a high effect size does not necessarily indicates a good
responsiveness. So, if the effectiveness of an intervention or the change in health statuson the construct of interest is known, one could use effect sizes (mean change score / SDbaseline)(76), and related measures, such as standardised response mean (mean change
score / SD change score) (77), Norman’s responsiveness coefficient (σ2 change/ σ2change + σ2 error) (78), and relative efficacy statistic ((t‐statistic1 / t‐statistic2)2) (79) toevaluate responsiveness.
When several instruments are compared in the same study, this could give evidence forthe relative responsiveness of the instruments. But again, only if a hypothesis is being
tested including the expected magnitude of the treatment effect. Let us propose that wehave three measurement instruments (A, B, and C), all measuring the same construct.
The intervention given is expected to moderately affect the construct measured by thethree instruments. Results show that instrument A has an effect size of 0.8, instrument Bof 0.40 and instrument C of 0.15. Based on our hypothesis of a moderate effect weshould conclude that instrument B appears to best measure the construct of interest.
Instrument A seems to over‐estimates the treatment effect (e.g. because it shows change
in persons who do not really change), and instrument A seems to under‐estimates it.
This example shows that it may not always be appropriate to conclude that the
instrument with the highest effect size is the most responsive.
Inappropriate measures for responsivenessGuyatt’s responsiveness ratio (MIC/SD change score of stable patients)(80) is
considered to be an inappropriate measure of responsiveness, because it takes theminimal important change into account. Minimal important change concerns theinterpretation of the change score, not the validity of the change score. The paired t‐testis also considered to be an inappropriate measure of responsiveness because it is ameasure of significant change instead of valid change, and it is dependent on the sample
size of the study (75).

63

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs
Appendix 1. Example of a search strategy (81)1) construct#1: (Depressive disorder[mh] OR depression[mh] OR (depress*[tiab] NOT medline[sb]))(2) population#2: (Diabet*[tiab])(3) sensitive search filter developed by Terwee et al. (27)64

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs

Appendix 2. Example of a flowchart

PubMedn = ...

EMBASEn =...

….
n =...

Additional record
identified
n = ...

Records screened
after removing
duplicatesn=Articles selectedbased on title andabstractn=Reasons for exclusion:
Validation not aim of study (n = )Different constrct measures (n = )Different study population (n = )Different type of outcome measure (n = )Other (n = )Total included in the review:n articles describing x PROMs

65

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs
Appendix 3. Example of a table on characteristics of the included PROMs
PROM*(reference
to first
article)Construct(s)Targetpopulation

Mode ofadministration(e.g. self‐report,interview‐based,
parent/proxy
report etc)

Recallperiod(Sub)scale(s) (numberof items)

Response
options

Range ofscores/scoring

Original
language

Availabletranslations* Each version of a PROM is considered a separate PROM.

Other characteristics which may be extracted are: ‘administration time’; ‘year of development’; ‘conceptual model used’; ‘recommended by standardization
initiatives for [a specific patient population or for the construct to be measures]’, ‘number of studies evaluating the instrument’; ‘completion time (minutes)’; ‘Full
copy available’; ‘licensing information and costs’.

66

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs
Appendix 4. Example of a table on characteristics of the included study populations
Population
PROM

Ref

A

1
2
3
1

B

N

Age
Mean (SD,range) yr

Disease characteristics
Gender% female

Disease

Diseasedurationmean (SD) yr

Instrument administration
Diseaseseverity

Setting

Country

Language

Response
rate

Other characteristics which may be extracted are ‘study design’, ‘patient selection’.

67

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs
Appendix 5. Information to extract on interpretability of PROMs
The content of this table is based on the Box Interpretability from the original COSMIN Checklist (1)PROM (ref)Distribution ofscores in the
study
population

Percentage of
missing items
and percentageof missing total
scoresFloor andceiling effects

Scores andchange scoresavailable forrelevant(sub)groups

Minimal importantchange (MIC) or
minimal importantdifference (MID)

Information on
response shift

PROM A (ref 1)
PROM A (ref 2)
PROM A (ref 3)
PROM B (ref 1)…

68

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs
Appendix 6. Information to extract on feasibility of PROMs
The content of this table is based on the guideline for selecting PROMs for Core Outcome Sets (5)Feasibility aspects
Patient’s comprehensibility
Clinician’scomprehensibility
Type and ease of

PROM A

PROM BPROM CPROM D

administration
Length of the instrument
Completion time
Patient’s required mental
and physical ability level
Ease of standardization
Ease of score calculation
Copyright
Cost of an instrument
Required equipment
Availability in different
settings
Regulatory agency’s
requirement for approval

69

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs

Appendix 7. Table on results of studies on measurement properties
Fictional example of results of the measurement properties
Country(language) inwhich thequestionnaire
was evaluatedStructural validity

Internal consistencynResult (rating)n

PROM A (ref 1)

US

878

Unidimensiona
l scale (CFI
0,97, TLI 0,97)
(+)

878

PROM A (ref 2)

Dutch

573

PROM A (ref 3)

Spanish43

PROM A (ref 4)
PROM A (ref 5)

PROM (ref)

Meth
qual

Result (rating)

84

Very
good

Dutch

113

UK

78

adequ
te
inade
quate

ICC
(agreement) =0.85 (+)
ICC = 0.93 (+)

878

1 factor (+)

1494

Meth
qual
Very
good

Result(rating)
Cronb.
Alpha =
0.92 (+)

Very
good
Very
good

PSI = 0.94
(+)
Cronb
alpha =
0.91 (+)

ReliabilitynPooled or summary result(overall rating)

Meth
qual
Very
good

Cross‐culturalvalidity\
measurement
invariancen
Meth
Resultqual(rating)

0.91‐0.94(+)

275

Spearman rho= 0.94 (+)
0.85‐0.94 (+)

70

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs

PROM

PROM A (ref 2)

Country(language) inwhich thequestionnaire
was evaluated
Dutch

PROM A (ref 5)
PROM A (ref 6)

Measurement errornMethResultqual(rating)

Criterion validityn
Meth
Resultqual(rating)

Hypotheses testingn
Meth
Result (rating)
qual
573

Very
good

UK

78

adequ
ate

US

154

Adeq
uate
inade
quate

Pooled or summary result(overall rating)805

Responsiveness
n
Meth
qual

Result(rating)

Results in linewith 2 hypo’s(2+)
Results in linewith 5 hypo’s(5+)
Results in linewith 3 hypo’s(3+)
Result in linewith 1 hypo
(1+)
Results not inline with 2
hypo’s (2‐)
11+ and 2‐
overall (+)

71

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs

Appendix 8. Summary of Findings Tables
Structural validity

Summary or pooled result

Overall rating

Quality of evidence

PROM A

Unidimensional scoresufficientHigh (as there is one very good study
available)

PROM BPROM CInternal consistency

Summary or pooled result

Overall rating

Quality of evidence

PROM Asummarized Cronbach alpha / PSI =sufficient

High: multiple very good studies,

0.91‐0.94; total sample size: 1494

consistent results

PROM BPROM C
Cross‐cultural

Summary or pooled result

Overall rating

Quality of evidence

No info available

No info available

Reliability

Summary or pooled result

Overall rating

Quality of evidence

PROM AICC range 0.85 – 0.93; consistentsufficient

High: one very good study, andvalidity\measurement invariance
PROM A
PROM BPROM C

72

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs
results; sample size: 275

consistent results

PROM BPROM C

Measurement error

Summary or pooled result

Overall rating

Quality of evidence

PROM ANo info available

No info available

Hypotheses testing

Summary or pooled result

Overall rating

Quality of evidence

PROM A

11 out of 13 hypotheses confirmed

sufficient

High: as the unconfirmed

PROM BPROM Chypotheses come from inadequate
studies, we ignore these results.
PROM BPROM C

Responsiveness

Summary or pooled result

Overall rating

Quality of evidence

PROM A
PROM BPROM C73

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs

6. References
1.
Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The
COSMIN checklist for assessing the methodological quality of studies on measurement
properties of health status measurement instruments: an international Delphi study. Qual Life
Res. 2010;19(4):539-49.
2.
Prinsen CA, Mokkink LB, Bouter LM, Alonso J, Patrick DL, Vet HC, et al. COSMIN
guideline for systematic reviews of outcome measurement instruments. Qual Life Res.
2018;[Epub ahead of print].
3.
Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al.
Increasing value and reducing waste in research design, conduct, and analysis. Lancet.
2014;383(9912):166-75.
4.
Walton MK, Powers JA, Hobart J, al. e. Clinical outcome assessments: A conceptual
foundation – Report of the ISPOR Clinical Outcomes Assessment Emerging Good Practices
Task Force. Value Health. 2015;18:741-52.
5.
Prinsen CA, Vohra S, Rose MR, Boers M, Tugwell P, Clarke M, et al. How to selectoutcome measurement instruments for outcomes included in a "Core Outcome Set" - apractical guideline. Trials. 2016;17(1):449.
6.
Mokkink LB, Vet HC, Prinsen CA, Patrick D, Alonso J, Bouter LM, et al. COSMIN
Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures. Qual
Life Res. 2018;[Epub ahead of print].
7.
Terwee CB, Prinsen CA, Chiarotto A, Vet HC, Westerman MJ, Patrick DL, et al.
COSMIN standards and criteria for evaluating the content validity of health-related PatientReported Outcome Measures: a Delphi study. 2018.
8.
Collins NJ, Prinsen CA, Christensen R, Bartels EM, Terwee CB, Roos EM. Knee
Injury and Osteoarthritis Outcome Score (KOOS): systematic review and meta-analysis ofmeasurement properties. Osteoarthritis Cartilage. 2016;24(8):1317-29.
9.
Gerbens LA, Chalmers JR, Rogers NK, Nankervis H, Spuls PI, Harmonising Outcome
Measures for Eczema i. Reporting of symptoms in randomized controlled trials of atopic
eczema treatments: a systematic review. Br J Dermatol. 2016;175(4):678-86.
10.
Chinapaw MJ, Mokkink LB, van Poppel MN, van MW, Terwee CB. Physical activity
questionnaires for youth: a systematic review of measurement properties. Sports Med.
2010;40(7):539-63.
11.
Speksnijder CM, Koppenaal T, Knottnerus JA, Spigt M, Staal JB, Terwee CB.
Measurement Properties of the Quebec Back Pain Disability Scale in Patients With
Nonspecific Low Back Pain: Systematic Review. Phys Ther. 2016;96(11):1816-31.
12.
Terwee CB, Mokkink LB, Knol DL, Ostelo RW, Bouter LM, de Vet HC. Rating themethodological quality in systematic reviews of studies on measurement properties: a scoring
system for the COSMIN checklist. Qual Life Res. 2012;21(4):651-7.
13.
Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL, et al. The
COSMIN checklist for evaluating the methodological quality of studies on measurement
properties: a clarification of its content. BMC Med Res Methodol. 2010;10:22.
14.
Mokkink LB, Terwee CB, Stratford PW, Alonso J, Patrick DL, Riphagen I, et al.
Evaluation of the methodological quality of systematic reviews of health status measurement
instruments. Qual Life Res. 2009;18(3):313-33.
15.
Terwee CB, Prinsen CA, Ricci Garotti MG, Suman A, de Vet HC, Mokkink LB. The
quality of systematic reviews of health-related outcome measurement instruments. Qual Life
Res. 2016;25(4):767-79.

74

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs
16.
Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions
Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from:www.handbook.cochrane.org.
17.
Cochrane Hanbook for Systematic reviews of Diagnostic Test Accuracy Reviews 2013
[Available from: http://methods.cochrane.org/sdt/handbook-dta-reviews.
18.
Peterson DAB, P.; Jabusch, H. C.; Altenmuller, E.; Frucht, S. J. Rating scales formusician's dystonia: the state of the art. Neurology. 2013;81(6):589-98.
19.
Herr KB, K.; Decker, S. Tools for assessment of pain in nonverbal older adults withdementia: A state-of-the-science review. Journal of Pain and Symptom Management.
2006;31(2):170-92.
20.
GRADE. GRADE Handbook - Handbook for grading the quality of evidence and thestrength of recommendations using the GRADE approach. 2013.
21.
Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The
COSMIN study reached international consensus on taxonomy, terminology, and definitions ofmeasurement properties for health-related patient-reported outcomes. J Clin Epidemiol.
2010;63(7):737-45.
22.
Streiner DL, Norman G. Health Measurement Scales. A practical guide to theirdevelopment and use. 4th edition ed. New York: Oxford University Press; 2008.
23.
de Vet HC, Terwee CB, Mokkink L, Knol DL. Measurement in Medicine: a practicalguide: Cambridge University Press; 2010.
24.
Fayers PM, Hand DJ, Bjordal K, Groenvold M. Causal indicators in quality of life
research. Qual Life Res. 1997;6(5):393-406.
25.
Elbers RG, Rietberg MB, van Wegen EE, Verhoef J, Kramer SF, Terwee CB, et al.
Self-report fatigue questionnaires in multiple sclerosis, Parkinson's disease and stroke: asystematic review of measurement properties. Qual Life Res. 2012;21(6):925-44.
26.
Griffiths C, Armstrong-James L, White P, Rumsey N, Pleat J, Harcourt D. A
systematic review of patient reported outcome measures (PROMs) used in child andadolescent burn research. Burns. 2015;41(2):212-24.
27.
Terwee CB, Jansma EP, Riphagen, II, de Vet HC. Development of a methodological
PubMed search filter for finding studies on measurement properties of measurement
instruments. Qual Life Res. 2009;18(8):1115-23.
28.
Clanchy KMT, S. M.; Boyd, R. Measurement of habitual physical activity
performance in adolescents with cerebral palsy: a systematic review. DevMed Child Neurol.
2011;53(6):499-505.
29.
PRISMA Statement. http://prisma-statement org/ [Internet]. 2016 10/24/2016.
Available from: http://prisma-statement.org/.
30.
Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al.
Quality criteria were proposed for measurement properties of health status questionnaires. J
Clin Epidemiol. 2007;60(1):34-42.
31.
Ernst MY, Albert J.; Kriston, Levente; Schonfeld, Michael H.; Vettorazzi, Eik;
Fiehler, Jens. Is visual evaluation of aneurysm coiling a reliable study end point? Systematic
review and meta-analysis. Stroke. 2015;46(6):1574-81.
32.
DerSimonian R, Laird N. Meta-analysis in clinical trials revisited. Contemp Clin
Trials. 2015;45(Pt A):139-45.
33.
Terwee CB, Prinsen CA, de Vet HCW, Bouter LM, Alonso J, Westerman MJ, et al.
COSMIN methodology for assessing the content validity of Patient-Reported Outcome
Measures (PROMs). User manual. 2017.
34.
Strauss ME, Smith GT. Construct Validity: Advances in Theory and Methodology.
Annu Rev Clin Psychol. 2008.

75

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs
35.
Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychological
Bulletin. 1955;52:281-302.
36.
McDowell I, Jenkinson C. Development standards for health measures. J Health ServRes Policy. 1996;1:238-46.
37.
Messick S. The standard problem. Meaning and values in measurement andevaluation. . American Psychologist. 1975;oct:955-66.
38.
de Vet HC, Terwee CB. The minimal detectable change should not replace theminimal important difference. J Clin Epidemiol. 2010;63(7):804-5.
39.
de Vet HC, Ostelo RW, Terwee CB, van der Roer N, Knol DL, Beckerman H, et al.
Minimally important change determined by a visual method integrating an anchor-based and a
distribution-based approach. Qual Life Res. 2007;16(1):131-42.
40.
de Vet HC, Terwee CB, Ostelo RW, Beckerman H, Knol DL, Bouter LM. Minimalchanges in health status questionnaires: distinction between minimally detectable change andminimally important change. Health Qual Life Outcomes. 2006;4:54.
41.
Crosby RD, Kolotkin RL, Williams GR. Defining clinically meaningful change inhealth-related quality of life. J Clin Epidemiol. 2003;56(5):395-407.
42.
Yost KJ, Eton DT, Garcia SF, Cella D. Minimally important differences were
estimated for six Patient-Reported Outcomes Measurement Information System-Cancer scales
in advanced-stage cancer patients. J Clin Epidemiol. 2011;64(5):507-16.
43.
van Kampen DA, Willems WJ, van Beers LW, Castelein RM, Scholtes VA, Terwee
CB. Determination and comparison of the smallest detectable change (SDC) and the minimal
important change (MIC) of four-shoulder patient-reported outcome measures (PROMs).
Journal of orthopaedic surgery and research. 2013;8:40.
44.
Sprangers MA, Schwartz CE. Integrating response shift into health-related quality oflife research: a theoretical model. Soc Sci Med. 1999;48(11):1507-15.
45.
Boers M, Kirwan JR, Tugwell P, Beaton D, Bingham CO, III, Conaghan PG, et al.
The OMERACT handbook: OMERACT; 2015.
46.
Smart A. A multi-dimensional model of clinical utility. International journal for
quality in health care : journal of the International Society for Quality in Health Care.
2006;18(5):377-82.
47.
Fayers PM, Hand DJ. Factor analysis, causal indicators and quality of life. Qual Life
Res. 1997;6(2):139-50.
48.
Streiner DL. Being inconsistent about consistency: when coefficient alpha does anddoesn't matter. J Pers Assess. 2007;80:217-22.
49.
Wirth RJ, Edwards MC. Item factor analysis: Current approaches and future
directions. Psychological Methods. 2007;12:58-79.
50.
Floyd FJ, Widaman FK. Factor analysis in the development and refinement of clinicalassessment instruments. . Psychological Assessment. 1995;7:286-99.
51.
Comrey AL, Lee HB. A first course in factor analysis. 2nd ed. ed. Hillsdale, NJ:
Erlbaum; 1992 1992.
52.
Brown T. Confirmatory Factor Analysis for applied research. New York: The Guilford
Press; 2015.
53.
Embretson SE, Reise SP. Item response theory for psychologists. Mahwah, New
Jersey: Lawrence Erlbaum Associates; 2000.
54.
Edelen MO, Reeve BB. Applying item response theory (IRT) modeling toquestionnaire development, evaluation, and refinement. Qual Life Res. 2007;16 Suppl 1:5-18.
55.
Reise SPY, J. Parameter recovery in the graded response model using MULTILOG.
JEM. 1990;27:133-44.

76

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs
56.
Reise SPM, T.M.; Maydeu-Olivares, A. Targeted bifactor rotations and assessing theimpact of model violations on the parameters of unidimensional and bifactor models. Journalof Educational and Psychological Measurement. 2011;71:684-711.
57.
Linacre JM. Sample size and item calibration stability. Rasch Measurement
Transactions. 1994;7(4):328.
58.
de Vet HC, Ader HJ, Terwee CB, Pouwers F. Are factor analytical techniques
appropriately used in the validation of health status questionnaires? A systematic review on
the quality of factor analyses of the SF-36. . Quality of Life Research. 2005;14:1203-18.
59.
Cortina JM. What is coefficient alpha? An examination of theory and applications.
Appl Psychology. 1993;78:98-104.
60.
Cronbach LJ. Coefficient Alpha and the Internal Structure of Tests. Psychometrika.
1951;16:297-334.
61.
Streiner DL. Figuring out factors: The use and misuse of factor analysis. Can J
Psychiatry. 1994;39:135-40.
62.
Revelle WZRE. Coefficients Alpha, Beta, Omega, and the GLB: Comments on
Sijtsma. Psychometrika. 2009;74(1):145-54.
63.
Crane PK, Gibbons LE, Jolley L, Van Belle G. Differential item functioning analysis
with ordinal logistic regression techniques. DIFdetect and difwithpar. Med Care. 2006;44(11Suppl 3):S115-23.
64.
Petersen MA, Groenvold M, Bjorner JB, Aaronson N, Conroy T, Cull A, et al. Use ofdifferential item functioning analysis to assess the equivalence of translations of aquestionnaire. Qual Life Res. 2003;12(4):373-85.
65.
Gregorich SE. Do Self-Report Instruments Allow Meaningful Comparisons AcrossDiverse Population Groups? Testing Measurement Invariance Using the Confirmatory Factor
Analysis Framework. Medical Care. 2006;44(11 Suppl 3):S78-S94.
66.
Teresi JA, Ocepek-Welikson K, Kleinman M, Eimicke JP, Crane PK, Jones RN, et al.
Analysis of differential item functioning in the depression item bank from the Patient
Reported Outcome Measurement Information System (PROMIS): An item response theory
approach. Psychol Sci Q. 2009;51(2):148-80.
67.
Scott NW, Fayers PM, Aaronson NK, Bottomley A, De Graeff A, Groenvold M, et al.
A simulation study provided sample size guidance for differential item functioning (DIF)
studies using short scales. Journal of Clinical Epidemiology. 2009;62:288-95.
68.
Van Leeuwen LM, Mokkink LB, Kamm CP, de Groot V, van den Berg P, Ostelo RW,
et al. Measurement properties of the Arm Function in Multiple Sclerosis Questionnaire(AMSQ): a study based on Classical Test Theory. Disabil Rehabil. 2016:1-8.
69.
Shrout PE, Fleiss JL. Intraclass Correlations: Uses in assessing rater reliability.
Psychological Bulletin. 1979;86:420-8.
70.
McGraw KOW, S.P. Forming inferences about some intraclass correlation
coefficients. Psychological Methods. 1996;1:30-46.
71.
Cohen J. Weighted kappa: Nominal scale agreement with provision for scaled
disagreement or partial credit. Psychological Bulletin. 1968;70:213-20.
72.
de Vet HC, Mokkink LB, Terwee CB, Hoekstra OS, Knol DL. Clinicians are right not
to like Cohen's kappa. BMJ. 2013;346:f2125.
73.
Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation
coefficient as measures of reliability. . Educational and Psychological Measurement.
1973;33:613-9.
74.
de Vet HC, Terwee CB, Knol DL, Bouter LM. When to use agreement versus
reliability measures. . Journal of Clinical Epidemiology. 2006;59:1033-9.
75.
Altman DG. Practical statistics for medical research. London: Chapman and Hall;
1991.

77

COSMIN Risk of Bias checklist manual for systematic reviews of PROMs
76.
Cohen J. Statistical power analysis for the behavioural sciences. . 2nd ed. ed.
Hillsdale, NJ: Lawrence Erlbaum Associates; 1988.
77.
McHorney CAT, A.R. Individual-patient monitoring in clinical practice: are available
health status surveys adequate? Qual Life Res. 1995;4:293-307.
78.
Norman GR. Issues in the use of change scores in randomized trials. J Clin Epidemiol.
1989;42(11):1097-105.
79.
Stockler MR, Osoba D, Goodwin P, Corey P, Tannock IF. Responsiveness to change
in health-related quality of life in a randomized clinical trial: a comparison of the Prostate
Cancer Specific Quality of Life Instrument (PROSQOLI) with analogous scales from the
EORTC QLQ-C30 and a trial specific module. European Organization for Research andTreatment of Cancer. J Clin Epidemiol. 1998;51(2):137-45.
80.
Guyatt G, Walter S, Norman G. Measuring change over time: assessing the usefulnessof evaluative instruments. J Chronic Dis. 1987;40(2):171-8.
81.
van Dijk SEA, M.C.; van der Zwaan, L.; Bosmans, J.E.; van Marwijk, H.W.J.; vanTulder, M.W.; Terwee, C.B. Measurement properties of questionnaires for depressive
symptoms in adult patients with type 1 or type 2 diabetes: a systematic review. Qual Life Res.
2018; doi: 10.1007/s11136-018-1782-y. [Epub ahead of print].

78