Review for 10.1186/s12876-024-03224-9 (2024)

Title: Identifying novel acute pancreatitis sub-phenotypes using total serum calcium trajectories
Version: 0 Date: 02 Oct 2023
Reviewer:  Eran Bellin
Reviewer's report:
The authors use Group Based trajectory modelling (traj1,2) to identify distinct calcium trajectory
patterns discernable in the first 24 hours of a pancreatitis hospitalization in a large number of
admissions (~500,00 admissions, 250K patients) at Beth Israel hospital Mimic Iv 2008-2019
and Mimic III (2001-2007).
I found their effort of great interest and with a number of additions and enrichment of discussion
and analysis might find this a worthwhile methods paper describing the challenges of such an
effort.
The stated purpose is to identify defined patterns of clinical evolution of calcium to be used as
predictors for the ultimate outcome of death.
I will first go to the central issue of prediction and then explore some specific issues in the paper.
In any prediction effort, it is critical to describe the point of prediction and its outcome at
the individual patient level.  While some methodology on the average might detect a certain
mortality for the defined group in the aggregate, the specific attribution of mortality to the
specific patient is what is called for in order to use this tool to predict at the individual level or
to “adjust” for outcomes when you are looking at interventions and multiple confounds.
This means that once you have decided that the predictive model will predict at hour 24
hour (a choice that excludes about 25% of the entire sample), you must then summarize the
behavior in a meaningful fashion.  The authors use odds ratios as they are convenient to use
with logistic regression and to adjust for multiple variables (table 3- models 1,2,3)  (age,
gender,race,congestive heart failure….).  Of note, is that these “adjusters” do not dramatically
change the odds ratios and their confidence intervals from their crude Model 1 counterparts.
This suggests that describing with absolute and relative risks as well at least in the paper  would
not compromise the message a great deal (having shown the minimal impact on odds).
More important, however, is the failure of the authors to produce metrics that are meaningful in
the predictive universe.  What is the positive predictive value of a positive finding for phenotype
I, II, III , IV?  Similarly what is the negative predictive value?  (include metrics of sensitivity and
specificity).  How do these metrics of prediction compare to the other predictive models to which
they allude in the paper.  If it is not possible to create the comparator for lack of information in
the dataset, they should say that explicitly and then just report the predictive behavior of their
model.
The history of the traj model comes from the sociology, psychology universe where the model
was used to describe the outcome.  What are the outcome patterns of  juvenile delinquency over
years 11-15?  What sort of cohort patterns do we have in arrests as people age.   The purpose as
I understand it was to find the baseline variables that would predict the “trajectory phenotype”.
The method ostensibly also provided an ability to put in a time varying variable in the trajectory
path to see if an intervention might alter the trajectory.
Interestingly, in its origins in the sociology and psychology literature, the trajectory path was the
outcome variable with predictive modelling at baseline using static measurables at time zero.
This was advanced to include within trajectory time dependent interventions to see how the
trajectories would change, but the assignment to trajectory outcome was dependent on baseline
measures.
In the present paper the authors are using the characterized trajectory in the first 24 hours as a
“phenotypic exposure variable” to predict the outcome of death.
The use of the trajectory determined from two measures to claim to be able to predict the future
seems to be outside of the realm of the original intent although the authors provide two other
examples of a similar use3,4.  However, both  Bhavani3 and Bhatraju4 explicitly identified
a validation set (testing set), applied the model, and looked for “outcomes” an activity not
undertaken by the current authors.
Even if the authors presented the positive and negative predictive value, they would be guilty of
overfitting as they built their phenotypes on the same dataset that they are testing its predictive
performance.  Now that does not free them from actually doing the calculations, but it would
require them to admit when they do it that the behavior of the prediction may be  optimistic
unless they split the data set in half (training testing) or did some cross validation work (hold
out one tenth to test, using 9/10th to build the model.
The authors have told us that there are phenotypic trajectories with different likelihoods of death
and therefore (and this is a leap) we should be able to use this knowledge to predict outcome.
The problem is they do not tell us how to assign patients to the 4 phenotypes for predictive
purposes.
Let’s say we have Patient A and patient A has findings at time 0 and time 6, to which phenotype
is this patient to be assigned?  Without this assignment we can not use the Phenotype as an
exposure variable to calculate the positive, negative predictive value, sensistivity or specificity.
I am actually quite curious as to how just two measures can allow us to sort patients into the
four phenotypes for practical use, but the authors do not tell us how to do this.
Moreover, in providing the predictive behavior, they would have to distinguish between
predictive behavior of patients who had two values within the first 4 hours 4 hours apart,   6
hours  - 6 hour apart, I would leave this to them, but trajectory might be better identified with
different delta in time between measures.
Specific comments:
1. Authors do not acknowledge their responsibility to their intended non statistically literate
audience to explain some of their terms and meaning.  Table s2 provides for the different
phenotypes the average posterior probability.  But do not tell us what this means.  Is this
the posterior probability of survival?  It would also be helpful to reidentify the Names of the
phenotypes so the reader can easily ask himself does this make clinical sense?  Also, in looking at
the posterior probabilities we note that the confidence intervals all overlap (strange for a tool that
is to distinguish between trajectories) and also include values greater than 100%.  The authors
need to address these issues if they are to honestly engage their readers.
2. An underlying problem inherent is the large time horizon.  Has the management of acute
pancreatitis changed over this period of time?
3. The impact of calcium infusion as part of therapy is acknowledged by at least one analysis with
calcium infusion but unfortunately no diagram without calcium infusion.   In addition, the very
act of calcium infusion is a marker of non random management decision potentially segregating
patients into very different cohorts based on “clinical differences” perceived by the clinicians
and manifesting in the use of calcium that then changes the very trajectory you are trying to
use as a pure signal of what is the clinical state of the patient.  Again, this is not disqualifying
for publication if this is a methods paper trying to explore the challenge of large datasets that
because of the limited coverage of a single hospital forces the use of long time periods with
changing meaningfulness of calcium infusion as an identifier of the “sickliness of the patient”
or “clinical sophistication or attentiveness of the physician”.
4. I am struck by some of the very high calcium’s present and suspect that some must be from
the calcium infusion, some from osteoclastic damage from metastatic cancer to bone, but I think
some discussion of the extraordinarily high caclicums might te in order.
I enjoyed the paper and think that as a methods paper with some of the methodologic
enhancements described, it would help the readership understand some of the strengths and
weaknesses of analyzing such datasets, an important contribution as  today’s  twitter-age
clinicians are in a swoon about generative AI (large language modeling) and now more than ever
need to understand what real analytic challenges are beyond playing with stochastic parrots.
Blbliography
1. Nagin DS, Odgers CL. Group-based trajectory modeling in clinical research. Annu Rev Clin
Psychol. 2010;6:109-138.
2. Jones BL, Nagin DS. A Note on a Stata Plugin for Estimating Group-based Trajectory Models.
Sociological Methods & Research. 2013;42(4):608-613.
3. Bhavani SV, Carey KA, Gilbert ER, Afshar M, Verhoef PA, Churpek MM. Identifying
Novel Sepsis Subphenotypes Using Temperature Trajectories. Am J Respir Crit Care Med.
2019;200(3):327-335.
4. Bhatraju PK, Mukherjee P, Robinson-Cohen C, et al. Acute kidney injury subphenotypes
based on creatinine trajectory identifies patients at increased risk of death. Crit Care.
2016;20(1):372.