Review for 10.1186/1471-230X-8-25 (2008)

Reviewer's report
Title:Quality of health care and patient satisfaction in liver disease: the
development of the QUOTE-Liver questionnaire
Version: 1Date:7 November 2007
Reviewer: Giovanni Apolone
Reviewer's report:
General
The paper by Gutteling JJ et al describes the preliminary phases of the
development and validation of a new satisfaction questionnaire aimed at
evaluating the quality of care delivered to out-patients with chronic liver disease.
Briefly, using a well described methodology based on focus groups (11 patiens
involved) and on empirical tests on two further samples of patients (28 and 152),
some information about its structure, internal consistency, content and face
validity are given. In addition, some exprets also rated the questionnaire.
The paper is well organized, methods are agreeable, data and findings are fairly
reported. Nevertheless, there are a few points that should be taken into account
as they hamper the quality and value of the paper in the current version. These
points may be summarized saying that the results are preliminary and further
data are needed to support the Authors Conclusion “ It is a useful instrument that
can be used to identify those aspects of care that need improvement...” (as
stated at the end of the Abstract and in the Discussion).
As a matter of fact, before introducing a new intervention in the health care
setting, whether it is a technology, a new drug, or an evaluative tool, its validity
must be fully tested as the final goal is to improve patients (quality of) care, as
any change in patients care is potentially associated with benefits, risks and
costs. “Validity” is not a yes/not attribute but the results of the accumulation of
evidence produced through a long “validation” process where hypotheses are
formulated and then sequentially challenged with several empirical testings (see
Ref 1 for an example of the entire validation process that is required to support
the validity of a new tool). What so far done is promising, but some further tests
are required before claiming that this instrument is “validated” and useful. In
concusion, the new questionnaire should be tested in further and independent
settings, including longitudinal and comparative studies. So far the demonstration
of the content/face validity of the scales and the statistical association with an
external criterion can assure us that the instrument is promising but such
evidence does not tell us anything about the true/incremental value of the new
tool when compared to using “nothing” or “something else”.
On other words, the paper ends where it is supposed to start.
-------------------------------------------------------------------------------
Major Compulsory Revisions (that the author must respond to before a decision
on publication can be reached)
1)Given the preliminary nature of results, I recommend to amend from the paper
all the “over-interpretation” about the validity or the value of the questionnaire,
given the partial and scarce data furnished.
2)In the background, Authors fail to demonstrate the need to create a brand new
(disease and setting specific) satisfaction with care questionnaire. Most
references are old and not up-date. In addition, it is not clear why in this setting a
generic measure is not appropriate. In The Netherlands a generic version of the
QUOTE questionnaire is already available. Before starting the quite long process
of developping and validating a new tool, this fact should be proved or fully
discussed.
3)As to items inclusion in the final version of the questionnaire, are all items
actually required to describe the phenomenon under evaluation? And is it true for
each of the concepts under evaluation?
4)It is nor clear in the paper how the final scores are assembled and the potential
implications of the criteria and rules used to group items and put them together.
Are Authors sure that all items pertain to the same domain (satisfaction), and
may they be equally scored according to the identified concept
(importance,performance, etc)? And what about the methods used to assemble
the patients answers: did they test alternative (weighted, hierarchical) methods?
5) Reliability has been measured using the Cronbach alpha estimate. This is
appropriate in the first preliminary phase of the study to assess the
appropriateness and relevance of items inclusion,as well as the items internal
consistency; nevertheless, some evaluations of the questionnaire
stability/reproducibility is needed. It implies the use of the test-retest methods.
6) Concurrent validity was done using, as external indicator (criterion) of
satisfaction, a VAS mono-item question. This is strange and not appropriate as,
also according to the Authors in the “Background”, the VAS approach in this
setting is not reccomended. In addition, it is curious that in order to validate a
new tool, a non-validated measure is used as gold standard (what about the
validity, reliability of the VAS question?). This latter analysis can be considered
either a sort of tautology or an inappropriate analysis as we do not have anything
to support the validity, reliability, meaning and interpretability of the external
criterion adopted.
7)Do Authors think that a correlation of 0.69 is good enough to rate/judge as
“good” the questionnaire construct validity? To me it appears quite low, given the
concept under evaluation, the type of patients recruited and the statistics used.
Their point-of-view should be substantianed with literature.
8)The lack of sensitivity and the ceiling effect is very disturbing. A satisfaction
with care tool should be able to capture all the levels of satisfaction/quality of
care, pick up (discriminate) differences between groups and detect differences
over time in the same sample, after the implementation of changes in health
care. The present version of the questionnaire does not have at all these
important characteristics. May be considered useful a questionnaire that is only
focussed on high/top levels of the concept under evaluation (quality/satisfaction)
and does not have demonstrated capabilities to detect differences?
-------------------------------------------------------------------------------
Minor Essential Revisions (such as missing labels on figures, or the wrong use of
a term, which the author can be trusted to correct)
1) The preliminary and partial nature of available empirical findings should be
fully recognized and acknowledged across the entire paper. This limitation could
be also present in the title.
2) The above points raise also doubts about the generalizability of the
questionnaire validity outside the setting in which it has been developed and
tested. Some warnings about the need of further independent tests, the need to
also test other dimensions of “validity” such as the responsiveness (in
longitudinal studies, eventually with a before/after design), and the evaluation of
its robustness in other settings are required.
2) Finally, readers should be strongly advised to be cautious in generalizing
present results to other linguistic, cultural and health care settings.
-------------------------------------------------------------------------------
Discretionary Revisions (which the author can choose to ignore)
What next?: Reject because scientifically unsound
Level of interest: An article of insufficient interest to warrant publication in a
scientific/medical journal
Quality of written English: Acceptable
Statistical review: No, the manuscript does not need to be seen by a
statistician.
Declaration of competing interests:
I declare that I have no competing interests