Discussion on “Assessing the goodness of fit of logistic regression models in large samples: A modification of the Hosmer‐Lemeshow test” by Giovanni Nattino, Michael L. Pennell, and Stanley Lemeshow

I congratulate Nattino, Pennell, and Lemeshow (NPL) for creating a practically oriented goodness-of-fit statistic that can be used in large samples. As statistical estimation and nullhypothesis statistical testing originated in the realm of small samples, most procedures are designed for small samples, with even special extensions for small-sample corrections (eg, AICc). At the same time, there has been much less development of statistics or tests for the case of large samples. Yet, the issue of statistical tests with large samples has become a serious problem, both in research and in practice (Lin et al., 2013). Researchers and practitioners used to applying standard statistical inference using p-values are often unaware of the sample size issue. In fields such as information systems and management, it is now common to see statistical tests applied to samples of over 100 000 observations, with resulting minuscule p-values. While the danger in tests where the null hypothesis is “no effect” (eg, Fand t-tests in regression models) is the discovery of statistically significant but practically meaningless effects, in goodness-of-fit tests, where the null hypothesis is “perfect fit,” the danger is discarding useful, but statistically insignificant models. NPL are also commended for making the connection between a statistical performance measure (“goodness-offit”) and its practical use (“practically useful probabilities”). A major criticism of our field is that we often fail to link the statistical, or data analytical, goal, and utility to their domain counterparts. Such gaps lead to confusion and sometimes wrong usage in practice, where a statistical measure is optimized without considering the actual utility of the analysis. The information quality (InfoQ) framework by Kenett and Shmueli (2014) posits that the quality of information derived in an empirical study depends on four components and their interactions: goal, data, analysis, and utility. Goal and utility both require a mapping from domain to analytics. By presenting the real example of predicting postneonatal mortality, NPL demonstrated how to tie data and analysis with the medical goal and utility, and how those link to the proposed statistical performance measure. When dealing with statistical tests for large samples, a question I have often been asked by nonstatistician colleagues is when is the sample size considered “big”? While for simple cases like a t-test, we have guidelines such as “if n > 30 you can use the z-test instead,” for significance testing with regression and other complex statistical models, it is not clear when the inflated p-value issue “kicks in. ” In Lin et al. (2013), we suggested authors should look at charts of their p-values of interest as a function of sample size, but clearly that is intended for detecting the issue, not as a solution. Along these lines: when should a researcher consider the modified Hosmer Lemeshow test (HL test) in place of the HL test? NPL offer a few insights based on their simulation study: (a) “the proposed test is always more conservative than the traditional HL test,” (b) “the difference between the proposed modification and the traditional test increases in larger samples,” (c) “For moderate sample sizes, such as n = 10, 000, the power of the HL test is reasonable and the rejected models are likely to be poorly calibrated,” and finally (d) “Figure 2 implies that the proposed testing procedure can be applied to any scenario where using the HL test would be appropriate. ” Given these insights, what is the guideline that a researcher should follow in terms of choosing between HL and modified-HL in terms of sample size? NPL emphasize that calibration statistics and