Title: Construction and validation of a novel nomogram for prediction of lymph node metastasis
in HER2-positive breast cancer: based on the optimal number of examined lymph nodes for
accurate nodal staging
Version: 0 Date: 13 Nov 2024
Reviewer's report:
PEER REVIEWER ASSESSMENTS:
OBJECTIVE - Full research articles: is there a clear objective that addresses a testable research
question(s) (brief or other article types: is there a clear objective)?
Yes - there is a clear objectiveYes - there is a clear objective
DESIGN - Is the current approach (including controls and analysis protocols) appropriate for
the objective?
No - there are major issuesNo - there are major issues
EXECUTION - Are the experiments and analyses performed with technical rigor to allow
confidence in the results?
No - there are major issuesNo - there are major issues
STATISTICS - Is the use of statistics in the manuscript appropriate?
No - there are issues with the statistics in the studyNo - there are issues with the statistics in
the study
INTERPRETATION - Is the current interpretation/discussion of the results reasonable and not
overstated?
No - there are major issuesNo - there are major issues
OVERALL MANUSCRIPT POTENTIAL - Is the current version of this work technically
sound? If not, can revisions be made to make the work technically sound?
No - manuscript has some fundamental flaw(s)No - manuscript has some fundamental flaw(s)
PEER REVIEWER COMMENTS:
GENERAL COMMENTS: The manuscript presents a study focused on developing a nomogram
to predict lymph node metastasis in HER2-positive breast cancer using data from the SEER
database. However, several methodological and reporting issues impact its validity. The
choice of data from 2010-2015 raises questions about its current relevance. Additionally, the
predictors and outcome variables are not clearly defined, and inconsistencies in reporting,
such as inconsistency in timeframes, are evident. Potential biases due to exclusion criteria,
inadequate handling of missing data, and the absence of external validation limit the model's
generalizability. The heavy reliance on AUC as the sole performance metric, without considering
other measures, constrains the assessment of the nomogram’s effectiveness. Overall, the study
lacks the necessary clarity, robustness, and clinical applicability. I give my detailed comments
below.
REQUESTED REVISIONS:
1. The study uses data from the SEER database for patients diagnosed between 2010-2015.
However, the SEER database includes data up to 2024, raising questions about why more recent
data were not included. Additionally, Figure 1 shows a time period of 2010-2017, creating an
inconsistency that needs clarification.
2. The exclusion criteria applied in Figure 1 removed a large number of patients (48,747 out of
52,834), potentially introducing bias. For instance:
• Excluding patients with fewer than one examined lymph node (ELN) or those with incomplete
clinical data could bias the study toward patients with more comprehensive documentation.
• Excluding patients without radiation therapy or chemotherapy may exclude cases that represent
different clinical profiles, thereby skewing the results.
•
3. The manuscript does not clearly define predictors and outcome variables in a dedicated
section. Instead, they are dispersed throughout the text, making it difficult for readers to clearly
understand which variables were used and how they were defined.
4. The study does not explicitly describe how missing data were addressed. This is a critical
issue, as improper handling of missing data could introduce selection bias.
5. The manuscript does not adequately consider potential confounding factors, such as patient
comorbidities, which could independently affect lymph node metastasis rates.
6. The nomogram includes several variables (ELN, radiotherapy, chemotherapy, T-stage, and
grade), which increases the risk of overfitting, especially given the relatively small size of the
training set (n=2,828) and validation set (n=1,212). This may limit the model's generalizability.
7. The study heavily relies on logistic regression for both univariate and multivariate analyses.
However, this approach might not fully capture potential non-linear relationships between
predictors (e.g., chemotherapy, radiation) and outcomes (metastasis).
8. The study does not address how it handled potential class imbalance, especially since
patients with metastatic lymph nodes are likely to be fewer than those without metastasis. Class
imbalance could impact metrics like AUC (reported in Figures 3 and 5).
9. Although AUC values are reported, other important metrics like precision, recall, F1 score,
and confusion matrix are missing. Relying solely on AUC may be misleading, especially in
imbalanced datasets.
10. Odds ratios for variables such as chemotherapy, T-stage, and grade show significant
discrepancies between the training and validation cohorts (Tables 2 and 3). This inconsistency
is unexpected given the random split, suggesting potential instability in the model's predictive
power.
11. While calibration plots show good alignment between predicted and observed probabilities,
there is limited explanation of the results. Additionally, the manuscript does not address potential
overfitting that might result from overly optimistic calibration.
12. The manuscript lacks a detailed interpretation of the decision curve analysis results. The
clinical significance of the net benefit at different thresholds is not thoroughly explored.
13. The study concludes that examining exactly 13 lymph nodes is optimal for staging, but this
conclusion is based solely on internal data. Without external validation, these findings may not
be reliable for clinical recommendations.
14. Although some limitations are acknowledged, such as the absence of detailed clinical
data (e.g., recurrence rates, surgical techniques) from the SEER database, there is insufficient
discussion on how these limitations might impact the study’s conclusions and its applicability
to clinical practice.
15. The manuscript claims that the nomogram outperforms using ELNs alone, but it does
not quantify the additional benefit of including variables like radiotherapy or chemotherapy.
Furthermore, decision analysis to demonstrate the impact on clinical decision-making compared
to current standards is lacking.The manuscript presents a study focused on developing a
nomogram to predict lymph node metastasis in HER2-positive breast cancer using data from
the SEER database. However, several methodological and reporting issues impact its validity.
The choice of data from 2010-2015 raises questions about its current relevance. Additionally,
the predictors and outcome variables are not clearly defined, and inconsistencies in reporting,
such as inconsistency in timeframes, are evident. Potential biases due to exclusion criteria,
inadequate handling of missing data, and the absence of external validation limit the model's
generalizability. The heavy reliance on AUC as the sole performance metric, without considering
other measures, constrains the assessment of the nomogram’s effectiveness. Overall, the study
lacks the necessary clarity, robustness, and clinical applicability. I give my detailed comments
below.
REQUESTED REVISIONS:
1. The study uses data from the SEER database for patients diagnosed between 2010-2015.
However, the SEER database includes data up to 2024, raising questions about why more recent
data were not included. Additionally, Figure 1 shows a time period of 2010-2017, creating an
inconsistency that needs clarification.
2. The exclusion criteria applied in Figure 1 removed a large number of patients (48,747 out of
52,834), potentially introducing bias. For instance:
• Excluding patients with fewer than one examined lymph node (ELN) or those with incomplete
clinical data could bias the study toward patients with more comprehensive documentation.
• Excluding patients without radiation therapy or chemotherapy may exclude cases that represent
different clinical profiles, thereby skewing the results.
•
3. The manuscript does not clearly define predictors and outcome variables in a dedicated
section. Instead, they are dispersed throughout the text, making it difficult for readers to clearly
understand which variables were used and how they were defined.
4. The study does not explicitly describe how missing data were addressed. This is a critical
issue, as improper handling of missing data could introduce selection bias.
5. The manuscript does not adequately consider potential confounding factors, such as patient
comorbidities, which could independently affect lymph node metastasis rates.
6. The nomogram includes several variables (ELN, radiotherapy, chemotherapy, T-stage, and
grade), which increases the risk of overfitting, especially given the relatively small size of the
training set (n=2,828) and validation set (n=1,212). This may limit the model's generalizability.
7. The study heavily relies on logistic regression for both univariate and multivariate analyses.
However, this approach might not fully capture potential non-linear relationships between
predictors (e.g., chemotherapy, radiation) and outcomes (metastasis).
8. The study does not address how it handled potential class imbalance, especially since
patients with metastatic lymph nodes are likely to be fewer than those without metastasis. Class
imbalance could impact metrics like AUC (reported in Figures 3 and 5).
9. Although AUC values are reported, other important metrics like precision, recall, F1 score,
and confusion matrix are missing. Relying solely on AUC may be misleading, especially in
imbalanced datasets.
10. Odds ratios for variables such as chemotherapy, T-stage, and grade show significant
discrepancies between the training and validation cohorts (Tables 2 and 3). This inconsistency
is unexpected given the random split, suggesting potential instability in the model's predictive
power.
11. While calibration plots show good alignment between predicted and observed probabilities,
there is limited explanation of the results. Additionally, the manuscript does not address potential
overfitting that might result from overly optimistic calibration.
12. The manuscript lacks a detailed interpretation of the decision curve analysis results. The
clinical significance of the net benefit at different thresholds is not thoroughly explored.
13. The study concludes that examining exactly 13 lymph nodes is optimal for staging, but this
conclusion is based solely on internal data. Without external validation, these findings may not
be reliable for clinical recommendations.
14. Although some limitations are acknowledged, such as the absence of detailed clinical
data (e.g., recurrence rates, surgical techniques) from the SEER database, there is insufficient
discussion on how these limitations might impact the study’s conclusions and its applicability
to clinical practice.
15. The manuscript claims that the nomogram outperforms using ELNs alone, but it does
not quantify the additional benefit of including variables like radiotherapy or chemotherapy.
Furthermore, decision analysis to demonstrate the impact on clinical decision-making compared
to current standards is lacking.