Reviewer's report
Title:A systematic review of tests for lymph node status in primary endometrial
cancer
Version: 1Date:30 November 2007
Reviewer: Rob J Scholten
Reviewer's report:
This systematic review addresses the comparative accuracy of four index tests
for assessing the lymph node status in women with primary endometrial cancer.
To me, this is a well written, concise, and easy to follow review. I have the
following comments and suggestions. (NB: I donâ##t have sufficient knowledge
of the subject matter of this review, so my comments pertain to the methods and
presentation of results only.)
Major Compulsory Revisions
1. Study quality: the authors applied sound criteria for assessment of study
quality. To me, the adaptations that the authors made to various QUADAS
criteria, seem to be adequate and the applied procedure for quality assessment
(2 independent authors) is sound. However, on average, the component studies
seem to be of very low quality (Table 1) with indications of major verification bias
(inflating the sensitivities) and review bias (inflating both sens and spec). In
addition, there are quite a few studies with â##indeterminateâ## results, which
may influence the test accuracies considerably. The authors, however, donâ##t
seem to address the lack of methodological quality properly. The authors might
wish to degrade their conclusions, which to me are much too strong (â##This
review provides a robust summaryâ#¦.â##) in respect of this lack of quality of
component studies.
2. Data synthesis: the authors cite a few out-of-date papers (e.g. 5, 8 and 9) that
describe (meta-analytic) methods that are no longer considered to be
state-of-the-art. Direct pooling of sensitivities and specificities separately is not a
valid method, because it ignores the correlation (and covariance) of these two
measures. In addition, pooling of those measures usually leads to incorrect pairs
of summary estimates that donâ##t fit with the range of observed values (see
reference 9 and Gatsonis, AJR 2006;187:271-281 for an explanation). In
addition, pooling positive and negative likelihood ratios separately without
addressing them in combination can be done only in a few exceptional situations,
e.g. absence of a threshold effect (see reference 7). To me the existence of a
threshold effect cannot be excluded (see ROC space plots) and relying on
statistical tests for this threshold effect with such small studies (large chance of
type II error) can be dangerous. In addition, separate pooling of LRs can lead to
pairs of LRs that have impossible combinations of sensitivities and specificities
(see Zwinderman AH, Bossuyt PM. We should not pool diagnostic likelihood
ratios in systematic reviews. Stat Med 2007 Jul 5 [Epub ahead of print]. I would
strongly encourage the authors to apply meta-analytic methods (HSROC of
bivariate normal method) that address the above-mentioned issues (see
Gatsonis, AJR 2006;187:271-281 and Reitsma, J Clin Epidemiol
2005;58:982-990). Those methods will summarize the data either as an
estimated summary ROC-curve (which enables comparison of the accuracy of
the various index tests) or a summary estimate of the mean sensitivity and
specificity (if the data indicate that it is reasonable to present those measures).
From those pairs of summary sensitivities and specificities likelihood ratios can
be derived. It should be noted that all sensitivities and the specificities of
CT-scanning are quite heterogeneous, which can be caused by chance variation
(small samples), threshold effects, methodological flaws or clinical variation.
Therefore, the calculation of summary estimates might not be indicated.
Minor Essential Revisions
3. Comparing index tests: from Figure III I conclude that all index tests are
investigated in isolation. When making inferences about the comparative
accuracy of the various tests, the authors should bear in mind that they base
their inferences on indirect comparisons. Extraneous factors (e.g. the spectrum
of disease) may influence the accuracy of the various tests: when CT scanning
and SN would have been applied to patients with advanced and less advanced
disease, respectively, itâ##s difficult to judge the comparative accuracy of CT
and SN. The authors might wish to address the indirectness of their comparisons
in the discussion section.
4. If itâ##s reasonable to calculate summary estimates, the authors might wish to
discuss the effect of false positives and false negatives for clinical practice in the
discussion section.
Discretionary Revisions
5. Please, write ASTEC and ESGO in full.
What next?: Unable to decide on acceptance or rejection until the authors have
responded to the major compulsory revisions
Level of interest: An article of importance in its field
Quality of written English: Acceptable
Statistical review: Yes, and I have assessed the statistics in my report.
Declaration of competing interests:
I declare that I have no competing interests.