Methodological Issues in the Development of a Prediction Tool for First‐Trimester Outcomes

To the Editor: Doubilet and colleagues have presented a Web-based calculator to predict the firsttrimester outcome when a very early gestational sac (intrauterine saclike structures without a yolk sac or embryo) is seen on ultrasound imaging. Although an accurate prognostic tool for the first-trimester prognosis could be useful, we would caution against the use of this calculator for the following main reasons. First, the study performed a weak and inappropriate validation of the model, such that its performance is unknown. The area under curve reported by the authors is inadequate, as it is based on the exact same data that were used to develop the model, and this will tend to exaggerate the performance. The authors have omitted a crucial step, known as internal validation, which allows this bias to be quantified. Furthermore, the accuracy of the risk estimates themselves (referred to as calibration) was not estimated and thus remains unknown. However, good calibration is crucial for making appropriate clinical decisions and to counsel patients. Second, the authors noted that the proportion of miscarriages in their sample was much higher than is usually observed, and that the patient characteristics were not representative of typical patients. One consequence of this is that the tool is unlikely to give an accurate prediction in new patients. In particular, it appears that the risk of miscarriage may well be overestimated for more typical populations. Third, an important predictor, “human chorionic gonadotropin (hCG) rise,” was not measured in 465 (79%) participants in this retrospective study. There is therefore a considerable risk that the relationship between hCG rise and outcome will differ in different settings, particularly when contrasting measurement protocols are in place. Moreover, there is no description of how and when these hCG measurements should be taken. It should be noted that the sample size (345 miscarriages and 245 successful outcomes) was reasonable for the 9 parameters that were considered for inclusion in the final model (see Table 1 in the article). However, the authors used inappropriate statistical methods, which lead to poorly performing and inaccurate models. These include screening variables for significance using univariable tests, followed by using a stepwise regression approach on the set of significant variables, which are known to lead to unstable models. Furthermore, the authors dichotomized hCG rise when it was available, rather than using its actual value in the model. This loss of information will contribute to a deterioration in predictive performance. Finally, it is highly recommended for prediction model studies to follow the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline to maximize the value of the work. An estimate of the risk of miscarriage has an impact on the parents, and inaccurate estimates will lead to poor decisions. We therefore believe that it is premature to recommend this model and to state that “we have encoded the model in an online calculator that may help providers make decisions about the need for and timing of follow-up testing and also give patients realistic expectations about the prognosis of their pregnancy. ” The authors should perform a meaningful evaluation of the model: namely, an internal validation and also an external validation on new and representative data, before recommending its use. We are nonetheless concerned that the data source and statistical methods used here are problematic for optimal risk prediction.