Title: Comparative Diagnostic Accuracy of the IOTA SRR and LR2 Scoring Systems for
Discriminating Between Malignant and Benign Adnexal Masses by junior physicians in Chinese
Patients: A Retrospective Observational Study
Version: 0 Date: 02 Aug 2023
Reviewer's report:
Dear authors thanks for conducting avaluable study that compares the diagnostic accuracy of two
ultrasound-based predictive scoring systems, the IOTA SRR and LR2, in distinguishing between
benign and malignant adnexal masses in Chinese patients. Their results demonstrate that both
scoring systems exhibit high sensitivity and specificity in the hands of junior doctors. The
study contributes to the existing literature on adnexal masses and diagnostic models, providing
valuable insights into the performance of these scoring systems in a Chinese patient population.
However, there are points that can be addressed to strengthen the study further.
1.Title suggestion:
I can advice to change the submitted manuscript title into:
"Comparative Diagnostic Accuracy of IOTA SRR and LR2 Scoring Systems for Discriminating
Between Malignant and Benign Adnexal Masses in Chinese Patients: A Retrospective
Observational Study"
2.Abstract notes:
1. The study was conducted by two junior doctors, which might raise concerns about
interobserver variability and potential bias in the interpretation of ultrasound images and scoring
system evaluations. It would be beneficial to have a more experienced and skilled group of
doctors or specialists to ensure the accuracy and consistency of the assessments.
2. The study does not provide information about the specific training or experience of the junior
doctors in using the IOTA SRR and LR2 scoring systems. Understanding their familiarity and
proficiency with these systems could impact the results.
3. The study lacks information on the reproducibility of the scoring systems within the same
observer (intraobserver variability). Evaluating this aspect would have provided additional
insights into the consistency of the scoring systems' performance.
4. Although the sample size is reasonable, the study was conducted at a single center, which
might limit the generalizability of the results to other populations or regions.
5. The study only focuses on Chinese patients, and it may not be possible to extrapolate
these findings to other ethnic populations due to potential variations in disease patterns and
characteristics.
6. The study does not provide information about the prevalence of specific types of adnexal
masses in the cohort. It would be valuable to understand the distribution of different mass types
to assess the scoring systems' performance in specific subgroups.
3.Introduction notes:
1. Lack of specific research objectives: The introduction does not clearly state the research
objectives or the specific aims of the study. While it is mentioned that the study aims to compare
the diagnostic accuracy of the SRR and LR2 scoring systems in Chinese patients with adnexal
masses, this should be explicitly stated in the introduction.
2. Justification for focusing on Chinese patients: The introduction briefly mentions that the
existing prediction models were based on data from European populations and that no studies
have compared these models in Chinese patients. However, it lacks a clear justification for
focusing on the Chinese population. Providing specific reasons for this choice, such as potential
differences in disease characteristics, patient demographics, or variations in the performance of
the models in diverse ethnic populations, would strengthen the rationale for the study.
3. Lack of information on the study design: The introduction does not provide information about
the study design (retrospective, prospective, or a combination) or the source of patient data.
Adding details about the data collection process and patient inclusion criteria would enhance
the clarity of the study design.
4. Citations and recent references: The introduction contains several references to support the
discussion, but some of them are outdated (e.g., references [1], [2], [3]). It is recommended to
update the references with more recent and relevant studies to strengthen the current review.
5. Lack of information on the significance of the study: The introduction should emphasize
the importance of the research by highlighting the potential clinical implications of comparing
the two scoring systems in the Chinese population. Mentioning the potential impact on patient
outcomes, treatment decisions, and healthcare resource utilization would enhance the study's
significance.
4.Methods notes:
1. Junior doctors' training: The methods mention that the two junior doctors received theoretical
and practical training on both the SRR and LR2 systems before the study. It would be helpful
to include additional details about the training, such as the duration, content, and method of the
training, to assess its adequacy and potential impact on the study's results.
2. Lack of information on lesion classification: The methods mention that the SRR risk was
classified as low (#20%) and high (>20%), but there is no mention of how the LR2 probability
was categorized. Providing this information is essential for understanding the study's findings
and comparing the performance of the two scoring systems more effectively.
3. Intra-reader agreement values: The methods state that intra-reader agreements for the SRR
and LR2 systems were reported as # values of 0.92 and 0.87, respectively. However, the methods
do not mention the number of cases used to calculate these agreement values. Including the
total number of cases used for intra-reader agreement assessment would add transparency to
the analysis.
4. Data collection process: While the methods mention data collection, there is no specific
mention of how clinical data (e.g., age, menopausal status, and CA125 levels) were obtained.
Information on the sources of these data and how they were recorded would be helpful in
understanding the study's data integrity and reliability.
5. Missing information on inter-reader agreement: The study involves two junior doctors
independently evaluating each mass using the SRR and LR2 systems. While intra-reader
agreement values are reported, there is no information on the inter-reader agreement between
the two junior doctors. It is crucial to assess inter-reader agreement, as it provides insight into
the consistency of the scoring systems' interpretation between different observers.
6. Validation of US characteristics: The study uses IOTA terminology to describe the US
characteristics of each mass, but it does not provide information on the validity or reliability of
these US characteristics in the context of the study population. It would be helpful to mention
any previous studies or research that have validated the use of IOTA terminology for adnexal
masses in the Chinese patient population.
5.Results notes
1. Inconsistency in the number of patients and masses: In the methods section, it is mentioned
that the study included 128 patients who met the inclusion criteria. However, in the results
section, the number of patients is reported as 144. This discrepancy needs to be addressed and
clarified to avoid confusion.
2. Missing information on bilateral adnexal lesions: The results mention that there were 16
patients with bilateral adnexal lesions, but there is no further information or analysis provided
for these cases. It would be helpful to include a separate analysis or discussion on these bilateral
lesions to understand their impact on the results.
3. Explanation of diagnostic performance metrics: While the results provide the diagnostic
performance metrics (sensitivity, specificity, PPV, NPV, and Youden index) for both the SRR
and LR2 systems, there is no accompanying explanation or interpretation of these values. Adding
a brief discussion on the significance and clinical implications of these metrics would enhance
the understanding of the study findings.
4. Inadequate statistical comparison: The results mention that DeLong's test was used to compare
the AUCs of the ROC curves for the two scoring systems. However, the results do not include
the actual p-value of this comparison. Providing the p-value would help determine the statistical
significance of the difference between the AUCs.
5. Use of kappa value to assess agreement: The study uses the kappa (#) value to assess the
agreement between the SRR and LR2 scoring systems. While # values provide a measure of
agreement beyond chance, they can be influenced by the prevalence of the condition being
assessed. Therefore, it is essential to interpret the # value cautiously, considering the prevalence
of malignant adnexal masses in the study population.
A.Tables notes:
1. Inconsistent number of patients and masses: In Table 1, the number of patients is reported as
128, while in Table 2, the number of masses is reported as 144. This inconsistency needs to be
addressed and clarified to avoid confusion in the interpretation of the results.
2. Additional information for bilateral lesions: Table 1 mentions that 16 patients had bilateral
adnexal involvement, but no further information or analysis is provided for these cases in
subsequent tables. It would be helpful to include a separate column or row in Tables 2 and 3 to
present data specifically for bilateral lesions.
3. Format of diagnostic validity metrics: In Table 3, the diagnostic validity metrics (AUC,
sensitivity, specificity, PPV, NPV, and Youden index) for the two scoring systems are presented
for overall patients, premenopausal women, and postmenopausal women. It would be more
informative to include these metrics for each scoring system in separate rows or columns, making
it easier to compare the results side by side.
4. Lack of p-values for AUC comparison: Table 3 provides the AUC values for both scoring
systems, but it does not include the p-values from DeLong's test to compare the AUCs. Adding
the p-values would help determine the statistical significance of any differences in diagnostic
accuracy between the two systems.
5. Clarification of sensitivity and specificity values: The sensitivity and specificity values in
Table 3 appear to have a higher level of precision (e.g., 97.5%, 90.0%) compared to the
reported sample size (e.g., overall patients: n = 144). It would be helpful to clarify whether
these sensitivity and specificity values are calculated based on the total number of masses or the
number of patients to ensure accurate interpretation.
Suggestions:
1. Clarify and reconcile the inconsistency in the number of patients and masses reported in Table
1 and Table 2, respectively.
2. Include a separate column or row in Tables 2 and 3 to present data specifically for bilateral
adnexal lesions.
3. Reformat Table 3 to present diagnostic validity metrics for each scoring system in separate
rows or columns for better comparison.
4. Provide the p-values from DeLong's test in Table 3 to determine the statistical significance
of any differences in diagnostic accuracy between the two scoring systems.
5. Clarify whether the sensitivity and specificity values in Table 3 are calculated based on the
total number of masses or the number of patients to ensure accurate interpretation.
B. Statistical Analysis notes:
1. Lack of information on missing data: The statistical analysis section does not mention how
missing data were handled, if any. It is important to address any missing data and describe the
method used to handle them to ensure the validity of the results.
2. Need for more detailed explanation of ROC analysis: While the section mentions that
ROC curves were used to evaluate the diagnostic accuracy of the two scoring systems, it
lacks a detailed explanation of ROC analysis. Adding a brief description of ROC analysis and
how sensitivity and specificity were plotted on the ROC curve would make the results more
understandable.
3. Missing p-value for DeLong's test: The results mention the use of DeLong's test to compare the
AUCs of the ROC curves for the two scoring systems, but the actual p-value of this comparison
is not provided. Including the p-value is necessary to determine the statistical significance of
the difference between the AUCs.
4. Interpretation of kappa value: The agreement between the SRR and LR2 scoring systems
was assessed using the kappa (#) value, but there is no explanation or interpretation of this
value in the statistical analysis section. A brief discussion on the significance of the #-value,
especially considering the prevalence of malignant adnexal masses in the study population,
would be beneficial.
5. No mention of sample size calculation: The statistical analysis section does not mention
whether a sample size calculation was performed to determine the required number of patients
for the study. If sample size calculation was done, it should be mentioned, along with the
rationale for the chosen sample size.
6.Discussion notes:
1. Limited generalizability: The discussion mentions that the study data were collected from
a single center, which may limit the generalizability of the findings to a broader population.
It would be helpful to discuss the implications of this limitation and consider the need for
multicenter studies to confirm the results in different patient populations.
2. Missing information on clinical parameters: The discussion acknowledges that important
clinical parameters were missing in the study. It would be beneficial to discuss the potential
impact of this missing information on the study's findings and suggest possible ways to address
this limitation in future research.
3. Interpretation of sensitivity and specificity values: The discussion states that both scoring
systems had high sensitivity and specificity but does not provide a deeper interpretation of these
findings. Discussing the clinical implications of high sensitivity and specificity in the context of
ovarian cancer diagnosis and treatment could add value to the discussion.
4. Need for larger sample size: The discussion mentions the limitation of a relatively low sample
size and suggests the need for a larger sample to further validate the results. However, it does
not elaborate on the potential impact of the small sample size on the study's statistical power
and reliability of the findings.
7.Conclusion notes:
1. Lack of discussion on the clinical implications: The conclusion does not discuss the clinical
implications of the study's findings. It would be valuable to provide insights into how these
results could impact clinical decision-making, patient management, and potential improvements
in the accuracy of adnexal mass diagnosis.
2. Limited generalizability: The conclusion does not address the limitation of the study being
conducted in a single center and the potential implications for generalizability. Acknowledging
this limitation and discussing the need for further research in diverse patient populations would
enhance the conclusion's completeness.
3. Reiteration of results: The conclusion largely reiterates the study's findings without
introducing any new information or broader implications. To improve the conclusion's impact, it
could highlight how the study contributes to the existing body of knowledge and suggest avenues
for future research.
8.References notes:
1. Recent studies missing: The references listed are up to date until 2022, but the study was
conducted in 2023. It would be beneficial to include more recent studies published in 2023 or
close to the current year to ensure the most up-to-date information is presented.
2. Diverse geographical representation: The majority of the references appear to be from studies
conducted in China or other specific regions. To enhance the generalizability of the findings,
the reference list could include studies from a more diverse geographical representation.
3. Limited number of references: While the references provided are relevant, the total number
seems limited. To strengthen the study's foundation, additional references from high-impact
journals and well-established studies in the field could be included.