Review for 10.1186/s12876-021-02082-z (2022)

Reviewer’s report
Title: 
A case for improved assessment of gut permeability – a meta-analysis
quantifying the lactulose:mannitol ratio in coeliac and Crohn’s disease
Version: 
1 
Date: 
23 Dec 2020
Reviewer's report:
I would like to thank Gan et al for their revisions to the manuscript and response
to my comments. The manuscript has improved. However, I have some concerns
that have not been adequately addressed.
1. Regarding the definitions of active vs. inactive Crohn's disease and how these
groups were handled in the analysis
My original comment:
In a spot check of selected studies, the threshold of 150 on the CDAI is
incorrectly used to differentiate between active and inactive disease:
* The authors state that Haas et al 2009 compares patients with inactive Crohn's
disease to controls. However, 8 of the 29 Crohn's disease patients included in the
study were on steroids and 8 were acutely ill at the time of the study. While the
mean CDAI among included patients was 127 (less than the threshold of 150),
the maximum CDAI among included patients was >300 (a score indicative of
moderate-to-severe Crohn's disease). This, therefore, is not a true comparison of
inactive Crohn's disease and controls.
* The Wild et al 2003 study is a comparison of patients in an acute disease flare
before and after receiving a course of steroids - the authors state that this is a
comparison of active and inactive Crohn's disease. However, only 22 of 30
patients included in the study reached steroid-induced clinical remission. This
means that 8 patients continued to have active disease and the comparison
presented by the authors is not truly a comparison of active and inactive Crohn's
disease.
The authors' response:
The reviewer has raised a good point about how the grouping of patients with
inactive and active Crohn's disease in these studies. For the Wild et al study, we
have excluded the 8/30 patients who did not experience remission and only
analysed the LMR for the remaining 22 who did recover after the course of
steroids. In the Haas et al study, only one LMR value was reported for the entire
group. Thus we were not able to remove any patients highlighted by the reviewer
in this case. We have included additional text in the manuscript to address this.
We acknowledge the limitations in the papers mentioned and have discussed this
in the 'discussion' section. In the Haas et al study, we took the LMR from a group
with a mean CDAI of less than 150 and explained in table S1 that the group was
heterogenous where 8 patients were acutely ill. In the same table S1, we have
also highlighted the different definitions used to categorise Crohn's disease, and
discussed this as a limitation as well. We have corrected the issues highlighted
by the reviewer in the case of the Wild et al study and have explicitly discussed
the limitations in the case of Haas et al study (where removal of the highlighted
data was not possible)
My additional concern:
Thank you for clarifying the exclusion of individuals from the Wild study and
that patients with active and inactive disease could not be analyzed separately for
the Haas study. The exclusion of patients from the Wild study should be clearly
stated in Table 2 and Table S1. Table 2 reports that 30 patients with active
Crohn's disease were included in the analysis and 11 patients with inactive
Crohn's disease were included. This is different than the 22 patients mentioned
by the authors. These numbers should be clarified.
I am still concerned that the comparison reported in the Haas study is reported as
a comparison of inactive Crohn's disease and healthy controls. Given that
patients with active and inactive Crohn's disease could not be separated in this
study, I would recommend that this study be excluded from the pooled
comparison of inactive Crohn's disease and healthy controls since the
comparison made in this study is not the same as the comparisons being made in
other studies.
2. Regarding the use of weighted vs. standardized mean differences
My original comment:
Given the differences in the methods used to estimate the LMR across studies,
the authors should report the pooled standardized mean difference, rather than
the weighted mean difference. This estimate should be less vulnerable to
differences across assays.
The authors' response:
As per the reviewer's suggestion, we have included the pooled standardised mean
difference in LMR across studies. (See updated figures in supplementary
information)
My additional concern: 
Thank you for adding the standardized mean differences to the text. Given the
heterogeneity in the methodology across studies, I would recommend the
standardized mean differences be the primary outcome, rather than the weighted
mean difference. Since the standardized mean differences are based on Z-scores
rather than absolute values, the data are more comparable when presented as
standardized mean differences.
3. Regarding the LMR values used in the calculation of sensitivity and
specificity calculations
My original comment:
In order to calculate a sensitivity and specificity for the LMR in diagnosing
coeliac disease, a threshold would be required, whereby an LMR above this
threshold would be considered to correspond to a coeliac diagnosis while an
LMR below this threshold would not be considered coeliac. Given the
heterogeneity in values and methodology observed across the included studies, it
is likely that these thresholds varied across studies. Additional information about
these thresholds is required. This corresponds to item 14 on the PRISMA-DTA
checklist (extension of PRISMA for diagnostic accuracy tests:
http://www.prisma-
statement.org/documents/PRISMA%20DTA%20Checklist.pdf)
The authors' response:
The reviewer has raised a good point, and we will explain in our 'Methods'
section that the threshold that we used was coeliac diagnosis by intestinal biopsy
which is used as the established gold standard in diagnosing coeliac disease. 
My additional concern:
Thank you for clarifying the gold standard used in these diagnostic accuracy
studies. My question, however, relates to the choice of a cut point above which
the LMR test is indicative of a celiac diagnosis and below which is indicative of
a healthy individual. The LMR test would need to be dichotomized in order to
calculate sensitivity and specificity. Considering the heterogeneity in the LMR
tests and the differences between celiac and healthy controls, the cut points
selected in included studies likely varied. An ROC curve depicts how sensitivity
and specificity vary according to values of the LMR test by plotting the false
positive rate (1-specificity) on the x-axis and sensitivity on the y-axis. The
authors should specify the specific LMR values used to calculate the sensitivity
and specificity values pooled in Figure S5.
4. Regarding the description of 'more significant' when describing the
comparison between the 2:1 and 5:2 solute ratios.
My original comment:
The authors state that "Differences in LMR were more significant when using
the 2:1 ratio than the 5:2 ratio." While the magnitude of the difference in the
LMR for untreated coeliac and healthy controls was greater for the 2:1 ratio and
the p-value was slightly smaller, the effect estimates and confidence intervals for
these estimates are quite similar. Further, there was high heterogeneity in most of
these comparisons and no statistical test was conducted that would allow authors
to reach this conclusion. The authors should be cautious about over-interpreting
their findings.
The authors' response:
We agree with the reviewer that it is difficult to directly compare the different
solute ratio due to the high data heterogeneity and we discussed this as a
limitation under the discussion section. We have also added text to the sentence
referenced by the reviewer to clarify this. 
My additional concern:
I still do not understand the 'more significant' description used by the authors.
The effect estimates and confidence intervals for the two solute ratios. In
addition, the presence of high heterogeneity for the pooled estimates make it
unlikely that these ratios were significantly different, and no statistical test was
performed to indicate a significant difference between the two ratios. Therefore,
one should therefore not describe one ratio as being 'more significant' than the
other.
5. Regarding different measures of variability across studies.
My original comment:
Did all included studies report means and standard deviations? If some reported
medians with either interquartile ranges or ranges, how were these handled? How
were paired data handled in the analysis (e.g., for Wild et al 2003, where LMR
measurements were repeated for each individual before and after a course of
steroids)?
The authors' response:
As the reviewer has pointed out, some studies included medians, 95% confidence
intervals, interquartile ranges, ranges and standard errors of mean. We converted
the standard errors, confidence intervals, interquartile ranges and ranges into
standard deviation using known statistical formulae. Where only median values
were available, we simply took these as the average LMR values (ie medians and
means were treated in the same way). Regarding paired data, we grouped them
according to inflammation levels or disease activity. In Wild et al 2003's paper
for example, the LMR before steroids in patients with CDAI >200 was put into
the 'active Crohns' group, and LMR after steroids in patients with CDAI <150
was put into the 'inactive Crohns' group. This effectively means that paired data
were treated in the same manner as unpaired data. We have added text into the
method section to clarify this. 
My additional concern:
Thank you for providing this additional detail in your response to my comments.
Can The values reported in Tables 1 and 2 do not explicitly state which studies
reported means and which reported medians. This should be clarified. In
addition, you report standard deviations for all studies even though you mention
here that some of these estimates are based on other measures of variation. The
data reported should accurately reflect this - either by specifying which values
were calculated or by providing the data as originally published in the included
studies. In addition, the Methods section should reference the methodology used
to convert measures of variation to standard deviations.
6. Regarding the 'weighted mean' calculations within study groups.
My original comment:
Can the authors clarify the 'weighted mean' and 'standard mean' LMR values for
each group? Only a single pooled mean LMR is reported for each group.
The authors' response:
We have now clarified the distinction between standard mean and weighted
mean and calculated overall LMRs in our methods section. This now reads as:
'Stata 15 (StataCorp, College Station, Texas) was used for the statistical analysis
reported in this study. All analyses were performed using a random effects
model in response to expected heterogeneity of data collected. The Weighted
Mean Difference (WMD) and Standard Mean Difference (SMD) of LMR were
calculated between controls and treated coeliac, controls and untreated coeliac,
treated and untreated coeliac, controls and inactive Crohn's disease, controls and
active Crohn's disease, and active and inactive Crohn's disease. The definition of
active Crohn's and inactive Crohn's in individual papers are listed in Table S1.
The weighted mean differences refer to the pooled estimate, with the 'weighted'
component referring to the different weights applied to each study in the overall
estimate. Meta-analysis of summary estimates of proportions was also calculated
for overall LMRs and sensitivities and specificities. All results for pooled
estimates were presented with 95% confidence intervals (CIs).' 
My additional concern: 
Thank you for clarifying the comparison between groups. Can the authors please
comment on the approach used to pool mean LMR values within study groups.
7. Regarding the description of 'weighted mean differences' for within study
comparisons. 
My original comment:
The authors report weighted mean differences for within-study comparisons.
Weighted mean differences typically refer to the pooled estimate, with the
'weighted' component referring to the different weights applied to each study in
the overall estimate. The authors should correct this terminology.
The authors' response:
Many thanks for this comment. We have now updated the methods section to
describe Weighted Mean Difference according to the reviewer's suggestion.
My additional concern: 
The weighted mean difference refers to the 'weights' used to pool studies. The
data reported in Tables 1 and 2 are mean differences, not weighted mean
differences. This should be corrected.
8. Regarding the interventions studied:
My original comment:
It would be helpful to add a description of the interventions study participants
were subjected to in Tables S1 and S2 for the RCTs and observational
interventional studies (e.g., Wild et al 2003 where LMR was measured before
and after a course of steroids in patients with Crohn's disease) included in this
review.
The authors' response:
We have done this and it is now reflected in Tables S1 and S2.
My additional concern: 
Thank you for adding this detail to Tables S1 and S2. It would be additionally be
beneficial for readers if a comment was added indicating when the comparison
of LMR values was made before and after the described intervention.
9. Regarding the Vilela studies.
My original comment: 
Vilela et al 2008 is included in Figure 1 twice.
The authors' response:
This is because there are two studies by Vilela and colleagues in the same year
pertaining this subject. 
My additional concern:
Thank you for clarifying that this was two separate studies. Can the authors
please double check the reference lists to confirm that both are referenced
correctly. Reference 18 is listed as Vilela et al and reference 28 is listed as
Garcia Vilela.
Are you able to assess any statistics in the manuscript or would you
recommend an additional statistical review?
If an additional statistical review
is recommended, please specify what aspects require further assessment in your
comments to the editors.
I am able to assess the statistics
Declaration of competing interests
Please complete a declaration of competing
interests, considering the following questions:
1
. 
Have you in the past five years received reimbursements, fees, funding, or
salary from an organisation that may in any way gain or lose financially
from the publication of this manuscript, either now or in the future?
2
. 
Do you hold any stocks or shares in an organisation that may in any way
gain or lose financially from the publication of this manuscript, either now
or in the future?
3
. 
Do you hold or are you currently applying for any patents relating to the
content of the manuscript?
4
. 
Have you received reimbursements, fees, funding, or salary from an
organization that holds or has applied for patents relating to the content of
the manuscript?
5
. 
Do you have any other financial competing interests?
6
. 
Do you have any non-financial competing interests in relation to this
paper?
If you can answer no to all of the above, write 'I declare that I have no competing
interests' below. If your reply is yes to any, please give details below.
I declare that I have no competing interests.
Statement on potential review bias
 Please complete a 
statement on potential
review bias, considering the following questions:
1. Did you co-author any publication with an author of this manuscript
in the last 5 years? 2. Are you currently or recently affiliated at the
same institution as an author of this manuscript?
If you can answer no to all of the above, write 'I declare that I did not publish
with these authors in the last 5 years and also meet the affiliation criteria”. If
your reply is yes to any, please give details below.
I declare that I did not publish with these authors in the last 5 years and also
meet the affiliation criteria.
Reviewer Publication Consent.
 I agree for my report to be made available
under an Open Access Creative Commons CC-BY License (
http://creativecommons.org/licenses/by/4.0
) if this manuscript is accepted for
publication. Any comments that I do not wish to be included in the published
report have been included as confidential comments to the editor, which will not
be published.
If you are not happy for us to publish this report, please contact the
editorial office before completing the review.
If you wish, you can include your
name in your published report. Please note you must decide whether to include
your name at the start of the process and confirm this decision whenever
prompted. If you change your decision later, we will not be able to publish your
name.
I agree to the terms of the CC-BY 4.0 license; please 
do not
 publish my name
with my report. (default)
Publons Reviewer Recognition
. Springer Nature can send verification of this
review directly to Publons (a subsidiary of Clarivate Analytics). If you would
like to take advantage of this service, please click on the “Yes” option below.
Your name, email address, title of the reviewed manuscript, name of the journal,
and date of your review submission (the “Review Data”) will then be transmitted
to Publons upon publication of the manuscript. If you have already registered at
Publons, they will notify you of the receipt of this review and update your profile
as per your settings and their policy. If you are not registered with Publons, you
will receive an email from them asking you to register in order for them to be
able to recognize your review on your new profile page. Publons may use the
Review Data to generate derivative metadata for the benefit of Publons and you
as a reviewer, carefully considering the sensitivity of such information. For
example, Publons may verify your record as a reviewer by updating your profile
published on its webservice if you have registered for such service or help
editors to identify candidate reviewers. Please find the details of processing in
Publons’ privacy policy 
https://publons.com/about/terms
No
Is the study design appropriate to answer the research question (including the use
of appropriate controls), and are the conclusions supported by the evidence
presented?
1
Are the methods sufficiently described to allow the study to be repeated?
2
Is the use of statistics and treatment of uncertainties appropriate?
2
Is the presentation of the work clear?
2
Are the images in this manuscript (including electrophoretic gels and blots) free
from apparent manipulation?
1