SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We describe the architecture used to model this in full detail in the SI; however, what is more essential for the utility of the ESGPT software library is that this API for specifying internal dependencies, and the associated non-architectural changes required to support it (e.g., enabling generation to respect the multi-stage process implied by the intra-event dependency graph) can be used across a variety of internal transformer architectures. Thus, this provided model class can serve as a basis for further model development that relies on internal event dependencies in future research without invalidating the benefits of the other tools in ESGPT. While the NestedAttention model builds on pre-existing works, to the best of our knowledge this precise architecture is novel.
Event Stream GPT provides generation capabilities for unsupervised, zero-shot evaluation. A critical use-case for foundation models is the ability to generate new, unseen predictions for the future of an input. To the best of our knowledge, ESGPT is the first pipeline to bring that capability in a systemized fashion to foundation models over event stream data like EHR systems. To do so, we introduce several key modifications from the traditional Hugging Face generation API, including the ability to support complex emission distributions during generation, dynamic computation of time-dependent input features for newly generated time-points, and generating events in sequence while respecting the causal path. Full details can be found in the SI.
5 Evaluating Foundation Models
5.1 The Problem
Even after building a foundation model on this non-standard domain, one still has to evaluate that model, both as a concrete representation learning system and in its possible viability as an early stage foundation model. In domains of complex, timeseries of events, like medical record datasets, there are a number of unique challenges that must be addressed in evaluation. We foresee the following metrics to be critical points to assess in early stage foundation models in such domains: (1) Performance as a raw generative model, (2) Viability as a foundation model, (3) Practical utility vs. current ML systems, (4) Performance disparities across subject sub-populations, and (5) Privacy risks.
5.2 Current State-of-the-art
Existing models in this space have significant gaps in their evaluation strategies to date. Table 4, in addition to summarizing model modalities, also summarizes evaluation methods used. While some techniques like few-shot fine-tuning analyses are well represented, others are under-explored. In
particular, at present there are no existing strategies to reliably assess general, zero-shot performance of GPTs over event stream datasets, a gap which we fill here.
5.3 Contributions of Event Stream GPT
ESGPT focuses on providing usable, powerful tools to enable rapid evaluation of GPTs over event stream datasets, focused currently on the assessment of a model's generative performance and in its viability as a foundation model in few and zero-shot evaluation tasks. While further efforts on the other evaluation criteria would be highly impactful, given the dearth of existing utilities just on assessing these basic performance metrics, we leave the rest to future work. In addition to these evaluation utilities, we also provide pre-built scripts for running distributed Bayesian hyperparameter optimization sweeps over foundation models through Weights and Biases (WandB) [2].
Event Stream GPT assesses generative model performance & hyperparameter tuning To make assessing generative performance easier, ESGPT comes pre-equipped with a PyTorch Lightning [7] module for running pre-training models which tracks a variety of metrics to assess generative model performance, including traditional classification and regression metrics over all component internal event covariates. These are logged by default to WandB, permitting easy exploration of output model performance. These metrics are further tracked and can be assessed visually in WandB reports during hyperparameter tuning as well, which is run via a WandB sweep with a provided default hyperparameter search space. A template report for assessing the output of these sweeps is also provided alongside the code for our working example over MIMIC-IV (accessible in the SI), allowing users to easily design their own dashboards to monitor their model tuning runs.
Event Stream GPT assesses foundation model viability via few and zero-shot performance. While many systems already explore few-shot performance (and ESGPT comes pre-equipped with lightning modules and utilities for performing similar assessments), assessing zero-shot performance across general models has not been previously studied. Enabling zero-shot evaluation for foundation models trained on event stream data requires three things: First, a labeled dataset corresponding to a well-defined task which can be used to evaluate zero-shot produced labels; second, a method to perform unsupervised forecasting of a patient's record given an input window, which ESGPT provides; third, a function to translate generated samples into empirical predictions.
The ESGPT system then provides a PyTorch Lightning utility to take as input a dataset, task, pretrained model, and labeling function, and perform zero-shot evaluation over a specified data cohort. This system reports performance metrics based on the output of this labeling function over the empirical predictions generated for each patient's input. This provides any model trained using the ESGPT API the ability to assess zero-shot performance in a meaningful way, out of the box. For examples of these evaluation utilities on MIMIC-IV, see the SI.
5.4 Example ESGPT Models over MIMIC-IV
To demonstrate the evaluation capabilities of ESGPT, in this section we provide examples of evaluating pre-trained models or training supervised models from scratch on two canonical fine-tuning tasks over MIMIC-IV: In hospital mortality prediction and 30-day readmission prediction (though all of these capabilities are fully extensible to any binary classification task over an ESGPT dataset). Full details of these experiments and results can be found in the online documentation for the MIMIC-IV tutorial<sup>4</sup>, but here we highlight three capabilities of ESGPT: (1) producing baseline performance numbers for fine-tuning tasks over ESGPT systems, (2) performing traditional fine-tuning, and (3) performing zero-shot evaluation of a pre-trained model.
In particular, with regard to baselines, Event Stream GPT provides utilities to hyperparameter tune baselines across both the same neural network architectures as are used during generative pre-training and scikit-learn [26] classifiers. On our proof-of-concept MIMIC-IV cohort for the in-hospital mortality prediction task, for example, hyperparameter tuned random forest classification pipelines can obtain a held-out AUROC of approximately 0.67, and for the readmission risk prediction task, it obtains a performance of approximately 0.65. On the other hand, training neural network architectures from scratch yields AUROCs of 0.79 and 0.57 for the two tasks, respectively.
${}^{4}$  eventstreamml.readthedocs.io/en/dev/MIMIC_IV_tutorial/index.html
When fine-tuning a model from a pre-trained model, we support limited hyperparameter tuning over just non-fixed parameters (e.g., learning rate, batch size, etc.). Doing this for the in-hospital mortality and readmission risk prediction tasks achieves AUROCs of approximately 0.8 and 0.62, respectively, in both cases yielding a lift over from-scratch training due to the generative pre-training at the  $12\mathrm{k}$  patient dataset scale. We can also assess performance via zero-shot performance. While our proof of concept models do not show better than chance AUROC on these measures, it is nonetheless a significant strength of ESGPT that these analyses can now be performed in a systematic manner.
Limitations While ESGPT provides significant benefits, it also has a number of limitations. Firstly, it lacks some more advanced pre-processing capabilities that other systems support, such as unit conversion, ontological aggregation, or support for a wide variety of optional pre-processing algorithms. Secondly, defining downstream tasks is currently only possible through manual creation / user-defined code, as opposed to via a configuration language. Next, while the utilities ESGPT offers do significantly enhance evaluation, even more is needed on that front, including dedicated metrics for fairness and privacy sensitivity, methods to assess embedding space structure, and methods to detect emergence of foundation model capabilities more cost efficiently. As a mitigation for these issues, we recommend that current users of these pipelines rigorously assess model performance for any downstream tasks across relevant patient subgroups to ensure bias or fairness implications are appropriately measured. Further research on this point, extending upon efforts such as those explored in [5, 37] is needed. Privacy mitigation strategies also need significantly more exploration before models can be safely released. While prior work on NLP models have found some risks due to memorization, the literature is far from clear on the risks or ways to address them for these kinds of models [12, 19]. Relatedly, any downstream users of ESGPT who wish to release datasets pre-processed with ESGPT should also ensure they take all appropriate precautions to ensure ethical, safe dataset release, and include sufficient documentation of the dataset's provenance, composition, and viable use-cases or pit-falls to ensure effective, impactful downstream use [9]. Thirdly, ESGPT currently only supports categorical or continuous valued event stream data modalities; however, this pipeline could also be extended to support additional modalities such as wearable and signals data, with similar abilities as those described in [21, 8]. Finally, ESGPT provides primary value as a data pre-processing pipeline and set of modeling APIs, not as a static set of model architectures or pre-trained weights. As such, we leave detailed comparisons of different model architectures that can be built using the APIs provided by ESGPT across a variety of benchmark tasks to future work.
Related Work As listed in Tables 2 & 4, there are a number of existing processing pipelines and models of relevance here. Beyond those already discussed, there are also a number of models that leverage nongenerative losses that warrant highlighting, such as contrastive methods like [1, 20, 23]. These provide a valuable alternative to generative foundation models in various domains. Additionally, other pipelines also exist in this and adjacent spaces, such as time series specific pipelines and tabular data embedding methodologies [14, 10, 4]. GPTs have also been used very successfully in other domains outside of event stream data. These include both examples of foundation models outside of either NLP or event stream data, such as on protein sequences [27] or point clouds [36], but also other examples of temporal point processes outside of the typical "foundation model" framework [29, 38].
In this paper, we introduced Event Stream GPT (ESGPT), a new, open source software library that fills a critical gap in the foundation model literature: namely, technological support for building generative, pre-trained transformers (GPTs) over "event stream data"—datasets consisting of continuous time sequences of complex events with multiple modalities and internal dependencies. We showed that Event Stream GPT can reliably address all aspects of the research pipeline for this end, from data pre-processing at scale, model construction in a manner accounting for the unique properties of event stream data, and evaluation across all aspects of the foundation model pipeline. We believe that ESGPT will significantly accelerate, standardize, and improve the space of research into these models and will help identify the best architectures and settings where the strengths of GPTs as seen in other domains can be replicated in new domains.
MBAM is supported by a Berkowitz Postdoctoral Fellowship at Harvard Medical School. BN completed this work while in the Department of Computer Science at the University of Toronto. This work has benefited from the advice of Dr. Oleksander Shchur. The All of Us Research Program is supported by the National Institutes of Health, Office of the Director: Regional Medical Centers: 1 OT2 OD026549; 1 OT2 OD026554; 1 OT2 OD026557; 1 OT2 OD026556; 1 OT2 OD026550; 1 OT2 OD 026552; 1 OT2 OD026553; 1 OT2 OD026548; 1 OT2 OD026551; 1 OT2 OD026555; IAA #: AOD 16037; Federally Qualified Health Centers: HHSN 263201600085U; Data and Research Center: 5 U2C OD023196; Biobank: 1 U24 OD023121; The Participant Center: U24 OD023176; Participant Technology Systems Center: 1 U24 OD023163; Communications and Engagement: 3 OT2 OD023205; 3 OT2 OD023206; and Community Partners: 1 OT2 OD025277; 3 OT2 OD025315; 1 OT2 OD025337; 1 OT2 OD025276. In addition, the All of Us Research Program would not be possible without the partnership of its participants.
[1] Monica N Agrawal, Hunter Lang, Michael Offin, Lior Gazit, and David Sontag. Leveraging time irreversibility with order-contrastive pre-training. In International Conference on Artificial Intelligence and Statistics, pages 2330–2353. PMLR, 2022.
[2] Lukas Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.
[3] Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F. Stewart, and Jimeng Sun. Doctor ai: Predicting clinical events via recurrent neural networks. In Finale Doshi-Velez, Jim Fackler, David Kale, Byron Wallace, and Jenna Wiens, editors, Proceedings of the 1st Machine Learning for Healthcare Conference, volume 56 of Proceedings of Machine Learning Research, pages 301-318, Northeastern University, Boston, MA, USA, 18-19 Aug 2016. PMLR.
[4] Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W. Kempa-Liehr. Time series feature extraction on basis of scalable hypothesis tests (tsfresh - a python package). Neurocomput., 307(C):72-77, sep 2018.
[5] Natalie Dullerud, Karsten Roth, Kimia Hamidieh, Nicolas Papernot, and Marzyeh Ghassemi. Is fairness only metric deep? evaluating and addressing subgroup gaps in deep metric learning. In International Conference on Learning Representations, 2022.
[6] Joseph Enguehard, Dan Busbridge, Adam Bozson, Claire Woodcock, and Nils Hammerla. Neural temporal point processes for modelling electronic health records. In Machine Learning for Health, pages 85-113. PMLR, 2020.
[7] William Falcon and The PyTorch Lightning team. PyTorch Lightning, March 2019.
[8] Simon Föll, Martin Maritsch, Federica Spinola, Varun Mishra, Filipe Barata, Tobias Kowatsch, Elgar Fleisch, and Felix Wortmann. FLIRT: A feature generation toolkit for wearable data. Computer Methods and Programs in Biomedicine, 212:106461, 2021.
[9] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86-92, 2021.
[10] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932-18943, 2021.
[11] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):96, 2019.