Value-aware transformers for 1.5d data
Authors: James F Cann
Timothy J Roberts
Amy R Tso
Amy Nelson
Parashkev Nachev
Venue: ICLR 2022 Submitted (2022)

Abstract: Sparse sequential highly-multivariate data of the form characteristic of hospital in-patient investigation and treatment poses a considerable challenge for representation learning. Such data is neither faithfully reducible to 1d nor dense enough to constitute multivariate series. Conventional models compromise their data by requiring these forms at the point of input. Building on contemporary sequence-modelling architectures we design a value-aware transformer, prompting a reconceptualisation of our data as 1.5-dimensional: a token-value form both respecting its sequential nature and augmenting it with a quantifier. Experiments focused on sequential in-patient laboratory data up to 48hrs after hospital admission show that the value-aware transformer performs favourably versus competitive baselines on in-hospital mortality and length-of-stay prediction within the MIMIC-III dataset.

Under review as a conference paper at ICLR 2022
VALUE-AWARE TRANSFORMERS FOR 1.5D DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Sparse sequential highly-muliine multivaratee data of the form characteristic of hospital in- in-patient investigation and treatment poses a considerable challenge for represen- tation learning. Such data is neither faithfully reducible to ID nor dense enough to constitute multivariate series. Conventional models compromise their data by requiring these forms at the point of input. Building contemporary sequence- modelling architectures, we design a value-aware transformer, prompting recon- ceptualisation of our data dis-dimensional: token-value value form both respecting its sequential nature and augmenting it with a quantifier. Experiments focused on sequential in nn-patient laboratory data up to 48hrs after hospital admission show that the awlue- transformer performs favourably versus competitive baselines on in-hospital mortality and length-of-stay prediction within the MIMIC-III dataset.
INTRODUCTION
Modelling the sequence of events in electronic health records (EHRs) poses fundamental challenges contemporary sequence models owing the complex, heterogeneous nature of the data. Even choosing good input representation for such data presents difficult trade-offs. The variety of data shapes encountered within the records for different patients and across different sections of their visit is substantial. While there are some variables, such as patient demographics, that one might reasonably expect all patients to have an entry for, the heterogeneity the patient's ss individual health status is more intricately explored. During admission to hospital, a series of laboratory tests (hereafter referred to as 'labs' for succinctness) may be conducted on patient: the number of possible types of test might be large (in the hundreds), these may be measured repeatedly, once, or not at all. Hospital stays may vary significantly in length and this variation is not necessarily indicative of severity. Very ill patients may have short survival, be artificially sustained on life support in expectation of a recovery that fails to arrive. There are further complications: some tests may have been conducted in advance attributed to their historical health record; or recorded in different modality of their health record (e.g. notes, charts, etc.). The vast complexity and heterogeneity of EHR data presents representation learning with fertile ground for innovation.
structure sets in as soon as
may
1.1 OUR CONTRIBUTIONS
This paper seeks address a particular data regime which is under-studied: that of sparse sequential and multivaault data, where the total number of variables typically outstrips the total number of measurements for given data instance. Table 6 illustrates this for the MIMICIIII lab-events dataset, where the mean number of completely unmeasured variables for a given visit is 697 out of a possible 753. As we shall describe, conventional approaches to modelling this data make severe simplifying assumptions, or rely heavily imputation. Many contemporary works restrict both the variables selected and the population studied, so as to reduce missingness. To address these issues, we:
Present a data encoding that effectively captures sparse sequential highly-multivariate data while avoiding considerable imputation.
Propose neat adaptations to make a standard transformer architecture value-aware, equip- ping to learn and generate this data encoding.
Provide an empirical study on real-world dataset demonstrating that a pre-trained value- aware transformer performs competitively challenging downstream predictive tasks.

---

Under review as a conference paper at ICLR 2022
1.2 RELATED WORK
This work sits on the intersection of a number of exciting areas of research: representation learning for healthcare, transformer architectures, sparse sequence modelling. Here we try to do justice to some of the many relevant works.
Representation learning for EHRs. Researchers seeking to make secondary use of the signal within EHRs have paid particular attention to the developments in deep learning (Shickel et al., 2017 Si et al. 2020), with applications ranging from unsupervised patient stratification Landi et al. (020 to learning personalised comorbidity networks (Qian et al. (020). The wave of excitement set in motion by transformer architectures (Vaswani et al. (017) quickly propagated
the health- care world. Large language models such as BERT (Devlin et al.) (018)) have been subjected to rounds of secondary pre-training on medical corpora (Alsentzer et al. 2019), with the aim of har- nessing signal locked up in the idiosyneratic natural language formats such as clinical notes. By encoding events as tokens, transformers can also be utilised for studying EHR event sequences. When arranged to constitute sentence, these tokens capture the pattern of patient's 's stay. This approach can be pursued at various levels of granularity, for instance Li et al. (2020) use BERT to model sequences of diagnoses over multiple visits. Applying the same methodology at finer grain presents natural model for within-visit medical event sequences such as labs. But while knowledge of the performance of lab test conveys information, it is the lab test together with its value that is the more complete indicator of the health status of patient. [hoi et al. (2020) made progress on integrating labs and their values using a graph convolutional transformer. modelling the graphical and hierarchical structure of EHRs in an exciting piece of work. However, their method was not designed to model repeat measurements of labs a common phenomenon within the natural course of medical investigation and treatment, motivating our work.
Healthcare benchmarks. In order to objectively compare systems such as those described above, EHR-specific benchmark tasks have been proposed, many based on the MIMIC dataset[Jonnoo et al. (0116). Of particular relevance here are the prediction tasks for in mossspitall mortality and length-of- stay proposed in Harutyunyan et al. (2019); Purushotham et al. (2018). In related work, and with a view to further increasing reproducibility, Wang et al. (2020) provided a pre-processing pipeline that addresses concerns such as unit conversion, outlier filtering and aggregation that are common in these tasks. The considerable heterogeneity of structure in EHRs means that the benchmarking papers need to make many design and preprocessing choices for the cohorts selected, the variables used, and the definitions of prediction targets. Not all of these choices are desirable for applications: e.g. in Harutyunyan et al. (2019) labels are defined in such way that the events of interest could occur within the data visible to the model rather than being restricted to predicting future events, while in Purushotham et al. (2018) their largest feature set includes only 11 of the approx. 700 raw features available in the MIMIC-III lab tests, chosen due to their minimal missingness rate. Some of these considerations (targets labels) we would like to refine so as to provide more utility, others (forced variable selection and aggregation) do not align with the purpose of this work, which is to remove the need for cherry-picking of variables. This makes direct comparison with these works challenging a point which other authors have highlighted, even when using largely similar cohorts, variables and identical prediction targets (Che et al. 2018).
Irregular times series. A closely related problem is that of studying irregularly samllee multi- variate time-series. (Shukla & Marlin (2020) detailed three different types of input representations: series-based, besedr-based and set-based, capturing time, variable and value for a range of scenarios. Horn et al. (2020) focus on the set-based approach considering each times series to be a bag-of- events, proposing a model which applies a set function together with single attention mechanism between elements of the set, topped with a classifier. In their experiments, an out-of-the-box trans- former is -o-te-----h-arrr for the Physionet 2012 prediction task (Goldberger et al. (2000). and per- forms competitively on MIMIC-III mortality prediction task (based on Harutyunyan et al. (2019). Another approach is elicited in (Che et al. 2018), where they introduce a GRU-based architecture, designed with inbuilt decay mechanisms. Their method is attuned to time-series variables such as flow data or regularly-sampled labs and relies on the imputation of missing values, which, for our data regime would constitute on average 98% of the input (see Table 6).
2

---

Under review as a conference paper at ICLR 2022
2 DATA GREATER THAN ID
Sequences of words such as those found in natural language neatly represented as ID data using tokens. Sequences of lab results such as those found early EHRs are neatly represented as (token, value) pairs, which might suggest that the ideal representation of this data should multivariate ries with one dimension per lab type. To model it in this form would require considerable imputation due to the data's s sparsity. However, the presence missingness of particular labs sequence informative: they are requested by healthcare professionals guided by clinical intuition that the EHR may not capture. By imputing the data, this valuable signal weakened. Perhaps we can make do with just a ID form and if so, how should that ID be constituted? These are the key questions that a practitioner hoping to model in our data regime faces. data is sparse and of lab-result sequences there may be only few or a large number of measurements with multiple repeat samples of the same lab characteristies which may contain important information.
we will discuss this section, when the the choice is not clear priori. For the motivational problem
We close the section by proposing variation on the conventional approaches, which we refer to as "1.5 11.5-ddmmensionl"" data: a form preserving the sequential nature of the lab orders, together with information about the values, but without the need for full multivariate or sparse vector-based sequence representations and the imputation issues that would ensue.
2.1 CHARACTERISTICS OF LAB-RESULT SEQUENCES
In-patient lab-result sequences are generated by a process of hospital investigation and treatment which imbue them with particular characteristics. Relevance: the labs pertinent to the health investigation being conducted on the patient. Only a small proportion of the all of the labs available might be obtained during any given visit. Repeats: these relevant measurements may be taken once, or repeated many times. Pragmatism: difficult or inconvenient labs will not be taken unless absolutely necessarily. This leads rare-token problem also commonly encountered in natural language. If we believe that these characteristics deliver valuable information, then the way that we represent our data should not compromise them.
2.2 COLLAPSING DATA DOWN TO ID
A common way to deal with this difficult data dimensionality is to 'collapse' it to 1D. There are two natural ways of doing this: the horizontal collapse and the vertical collapse, illustrated in Figure []]
Horizontal collapse. The horizontal collapse retains the sequential structure of the data, and re- moves the value aspect. One form of the horizontal collapse is to ignore the value from the token- value pair. We then have sequences of lab orders, and the premise is that there is sufficient informa- tion contained in the presence of lab order to inform the underlying state of patient. This capi-
Time
Lab 1
Lab 2
vertical
Lab 3
collapse
Lab 4
Lab 5
1D horizontal collapse
[2]
[2]
[4]
[
[4]
[6]
[5]
[6] [5]
Figure I: The ID horizontal data collapse forms a sequence of lab tokens while ignoring their values; ID vertical collapse takes summary statistic (e.g. the mean) for each lab sequence.

---

Under review as a conference paper at ICLR 2022
talises recent advancements in NLP and has formed the basis for some of the medical sequence- modelling applications mentioned in Section 1.2
Vertical collapse. An alternative form of ID collapse is collate repeated values of a variable using a summary measure (such as the mean, median, max, most-recent) and then use these summary measures one per variable the input model. This representation respects the unique value scales on a per-variable basis, however it compromises the sequential nature of the data and collapses repeated measurements.
2.3 IMPUTING FOR MULTIVARIATE SEQUENCES
An enticing candidate for the representation of (token, value) pairs is as a multivariate sequence with one sequence dimension per token. This retains each variable's independent value-scale and the innate temporally-orderer nature of the data. However, the sparsity of our specific data regime problematic, affecting its suitability: a program of severe imputation required order to use this input form. This commonly takes the form of sequential forward- or backward backward-filling filling of data; per-variable mean imputation; or the use of sentinel values, the choice which will necessarily depend the variable in question (so as monly attained by true data points). Imputation is not necessarily evil, but does require careful application, even in relatively dense data regime. Not all missingness is made equal and even the most advanced general-purpon imputation schemes rely on assumptions (MCAR or MAR: see Mohan & Pearl (202T) which are often unrealistic in healthcare setting. Unrealistic imputation can have unexpected downstream implications (Caruana (2021) which can be dangerous for critical applications. Even a sparse vector- based multivariate representation such as that detailed in Shukla & Marlin 2020) would require on average 98% of its input to be imputed for our data (see Table
inappropriately assign sentinel value that is com-
2.4
1.5D DATA
Both the ID collapse and the multivariate sequence approaches introduce their own issues or simpli- fications for sparse sequential highly-multivariate data. We propose interpolate these data regimes and use an alternative data format: [[OO],, [VAL] ), where [TOK] is the name of the variable and where [VAL] its value, which has been transformed and discretised in such a way that it mean- ingfully comparable across all variables. Such an approach:
retains the sequential nature of the data: it is ID sequence tokens.
removes the need for considerable imputation (cf. Tables I and 6
reduces the heterogeneous value dimension common scale, contextualised by its token.
For numeric labs, the value tokenisation can be achieved via quantilisation: conditional population statistics for the lab results can be used to attribute category to each of the values, depending on which quantile the value lies in. If there are no prior distributional statistics available, they can be extracted from the training set data (for simplicity and generality we took this approach in our analysis). For categorical labs we can encode them in any manner which is consistent. Where category can be associated from those mentioned previously, a sentinel category can used this may be necessary in the where result is not recorded, or the lab itself is not within the training set.
Table 1: Comparison of number of inputs required for ID, 1.5D and regularly-sampled multivariate sequence representations, where: is # variables, is number of sequence steps of the model, m is # data entries particular sequence. See T for an analysis of our dataset.
Data representation
# Inputs
Imputations
Signal compromised
1D (vertical)
E [7 m. 1
sequence structure & repeat measures values of measurements raw value-scales investigative pattern
(horizontal)
n m
1.5D
2n
2(n m)
Multivariate sequences
I n m
4

---

Under review as a conference paper at ICLR 2022
The discrete categories chosen should be chosen dependent the application. Since our variables are physiological measurements we choose to use a quantilisation that accentuates the extremities of the distributions, assigning the following discretisation:
[XLOW]: : the value lies below the 10% percentile
[LOW] the value lies between the 10% and 25% percentiles
[MID] the value lies between the 25% and 75% percentiles
[HIGH] the value lies between the 75% and 90% percentiles
[XHIGH] the values lies above the 90% percentile.
The token [UNK] was assigned the lab did not feature the train set, or if there was no numeric lab value associated with the lab. This specific quantilisation should not be read as definitive: other natural candidates are using quartiles, deciles, or a normal/abnormal discretisation, and should be informed by the modelling application.
Multiple events might approximately coincide: to break these ties we used consistent ordering of labs. For our application the order of coincident tests should have direct bearing the informa- tiveness of the tests themselves, but a random ordering would make harder for the model to learn these patterns. This be more explicitly dealt with using temporal embeddings (see Appendix D).
The [VAL] tokens should contextualised by their associated [TOK]. The sequence of [VAL] s, while plausibly carrying some autoregressive signal, not tof-thhessellss readily interpretable as the sequential data of single variable. They augment a ID sequence of tokens with values, hence the apt moniker 11.5D data".
VALUE-AWARD TRANSFORMERS
Transformer architectures in their various guises have shown exceptional performance tokenised sequence modelling tasks, but they lack principled approach for learning (token, value) sequences. The 1.5D data form lays foundation for doing so. We outline guiding principles which provide a basis for utilising this input form based on the discussion in Section 2.4 then design a neat and minimalistic transformer architecture for learning and generating it.
3.1 DESIGN PRINCIPLES
1. Tokens are primary. First, there were tokens, and then came their values.
Principle makes a statement about precedence. The sequence of values [VAL], by themselves, are bereft of meaning and without clear autoregressive interpretation. To benefit from these qualities the model must contextualise them using their respective tokens.
2. Values should be loosely-coupled to their token.
Loose-coupling is most easily defined by contrast with a strong-coupling where each token [TOK] and its quantifier, the value [VAL], is explicitly concatenated during preprocessing to form new token [TOK- [. Strong-coupling drops a fundamental symmetry present the raw data: the information that the pairs ( [[TOK), [VAL1]) and (ITOK), [VAL2]) of the same variable is not provided to the model using this representation. This concatenation approach scales poorly with the number of distinct tokens and values [okens]] At the other extreme there is no-coupling. No-coupling is to treat the tokens and values as completely separate entities. Modelling them in this way both causes the 1.5D form lose its meaning.
3. Values should have the opportunity to influence the autoregressive learning of the token sequence.
While the core autoregressive properties of the 1.5D sequence lie amongst the tokens, there may be useful information in their associated values which should aim capitalise
For further comparison with strong-coupling and empirical experiments Appendix
5

---

Under review as a conference paper at ICLR 2022
3.2 TAILORING VALUE-AWARK TRANSFORMER
The temporal nature of our sequence data make a decoder-only style transformer a compelling candi- date architecture. Large decoder-only models such as GPT-3 (Brown et al. 2020) have demonstrated astonishing performance, especially for natural language generation, and while data generation is not the primary use-case here, the aature-masking inductive bias is an appropriate one for event data. We highlight three core components that adapt a plain decoder-only transformer to the 1.5D data:
Embeddings. To respect principle and ensure loose-coupling of the ( ([TOK), [VAL]) pairs we use separate embedding layers for the tokens and the values. These two embeddings are then con- catenated. Only after this concatenation is a positional embedding added, so that both parts have their position consistently encoded.
For our experiments the number of tokens R 700 and the number of values 10 we map the tokens to R 100 and the values to R to force some compression. Deciding the optimal dimensionality of embedding is an open problem. but many embedding algorithms do not suffer much loss of performance due to parameterriiaiionnn Yin & Shen (2018).
The loosely-coupled embedding concatenation means that information about the values is included within the embeddings that are propagated through the attention layers. This is key that principle is respected. We use a now established bag of tricks (Wang) 202I) with the normalise- feeend-fedforward (with residuals) structure for the attention layers constituting value-aware transformer.
Attention.
Output layers. The choice of autoregressive prediction head for the transformer is another cru- cial design consideration. Both values and tokens should be predicted, and how this done has a significant effect on the overall learning of the model because of its proximity to the loss calculation.
Splitting the final attention layer's output embedding into two parts proved to work better than sharing the entire emhedding by some margin. In respect of principle the lion's share is dedicated to next-token prediction and the remaining part of the embedding used for next-value prediction. Dedicated pekennppediction and value-prediction layers are then attached their respective segment of the embedding. To respect the priority of tokens, the value- prediction layer is conditional on the predicted token, explicitly including it whether correct or incorrect.
Finally, we control the relative weight given to the cross-entropy losses for the tokens [ and the values L [VAL.) using hyperparameter 7 E [0;
C = [ + (1 - [VAL)
3.2.1 PRE-TRAINING
With view to later fine-tuning tasks we include one type of special token: an [EOS] token for the ond-of-sequence. The [EOS] has a distinguished role as the token upon whose output embedding perform classification or regression. There is different [EOS] for the tokens (E IR000) and for the values (E ) The last special token we include is a [PAD] to fill up empty slots in the sequence. The [PAD] class is not used when calculating the cross-entropy losses L [(rek] and L [vaL] During pre-training the model seeks to minimise the total loss, equation
3.2.2 FINE-TUNING
Fine-tuning a value-awar transformer for classification task consists of replacing the autoregres- sive output layers with a classification head, directly harnessing the embeddings output from the top attention layer. There is a small but important difference approach from common bidirectional language models. There, the first token is often denoted [CLS] and is imbued with special clas- sification function, with a classifier head being attached to it (Devlin et al. 2018). Because we are using (future-masked) decoder-style transformer, classification token would be useless: during fine-tuning, the model would see exactly the same input for each data instance, hence is unable to learn anything. In our setting classification is instead done the
prepended start-of-sequence
"hhse are always concatenated together during the input embedding stage.

---

Under review as a conference paper at ICLR 2022
Pre-training
Fine-tuning
To Value
Prediction
Hierarchical Autoregresive Head
Flosting EOS
Tolion
Prediction Head
[EOS] Embedcing
Partition
Attendion Layers*
Attention Layers*
Positional Emb
Positional Emb
Concatenate
Concatenate
Concatenate Embed
Token Emb
Token Emb
[FOS]
[E05]
[EOS]
[EOS]
Muture-masled
Figure Architecture diagrams for pre-training and fine-tuning a value-aware transformer
floating classification head which attaches itself at this position, taking its input the output embedding of [EOS]. We use simple but flexible one-hidden-layer ferd-forward neural net with output layer the number of classes relevant for the classification problem. For regression tasks we use exactly the same set up except with a single class output and a further non-linearity. For fine-tuning the model seeks to minimise problem-dependent loss. For periments we used a soppensity-eiighee cross-entropy loss for imbalanced classification problems and MSE loss for the regression problem.
[EOS] ] token:
EXPERIMENTS
MIMIC-III ('Medical Information Mart for Intensive Care" Flis resource for machine-learning researchers pursuing the advancement of healthcare. It is a large publicly accessible anonymised database comprising comprehensive information relating unit at large US hospital. The heterogeneity of data structure encountered within it are exemplar of the electronic health records healthcare systems across the world.
patients admitted the critical care
4.1 MIMIC-III LAB-EVENTS
Lab-event sequences occurring early within admission to hospital embody the sparse sequential -ighlyytulivaariate data characteristics described above, making the MIMIC-III lab-events an ex- cellent dataset to benchmark We focus experiments the lab-events data which is available up until 48 hours after admission, pre-training a value-aware transformer to learn its autoregressive structure. We then tackle three challenging downstream problems by fine-tuning the value-aware transformer. The first and second are classification tasks: in-hospital mortality prediction, both death within 3 days, and death after days; the third is regression task: predicting the remaining length-of-stay (LOS) at 48 hours.
4.2 EXPERIMENT 1: TRE-TINGNI
In the first experiment we investigated whether incorporating values assists in the prediction of tokens by comparing the value-aware transformer to a plain transformer which uses the same base
Available http://////ppaaoooo..oooooeeeeer

---

Under review as a conference paper at ICLR 2022
Table 2: Pre-training performance: value-aware transformer vs. transformer on Dev. set. L (TOK) indicates the token cross-entropy loss, and Prtok) the corresponding average probability of correct next-token prediction.
Model
Data dim.
L [TKK]
TORJ
[IAL]
PIVAL:
Random
0.0013
0.1250
Transformer (depth 4)
h
1.331
0.2642
VA-transformer (depth 4)
1.5
1.322
0.2674
1.347
0.2600
Transformer (depth 6)
h
1.319
0.2666
VA-transformer (depth
1.5
1.291
0.2750
1.332
0.2639
architecture. We captured their respective token cross-entropy losses L for comparison. We made the competition as fair as possible, choosing the same fixed input sequence length and their respective best learning rate hyperparameters with all other settings kept the same. We let each run until fixed early stopping threshold on the Dev. was met. Further details may be found in Appendix B
The results in Table demonstrate that the value-aware transformer provides promising improve- ment over the transformer, with better average next token prediction for both the small (depth 4) and large (depth 6) versions of the model. Comparing the two depth models, the value-aware trans- former provides 3.2% boost over its transformer sibling in the average probability of correct token prediction, whilst also generating values with considerably greater than random probability.
The successful outcome of Experiment 1 demonstrates that there is both valuable signal in the 1.5D representation, and that the value-aware transformer able exploit The second and third experiments test whether we can channel its superior generative modelling into performance gain on downstream tasks.
4.3 EXPERIMENT 2: IN-HOSPITAL MORTALITY PREDICTION
In-hospital mortality is a challenging clinical outcome to predict because of its highly-imbalanced nature: thankfully a minority of the patients admitted to hospital die. Just 0.84% of visits in our Train set result death within 3 days of admission. Accuracy a poor metric of success here attain 99.16% by always predicting that there is death. Instead optimise for balanced accuracy, using a Train set ceeennsttttwiighehe binary cross-entropy loss. We consider two targets: mortality < 3 days (which has Train propensity of 0.0084), and mortality 7 days (whose Train set propensity is 0.0496).
We compare the performance of fixedspeccffication classifier (a one-hidden layer feed-forward neural network with hidden dimension 100) when trained on various types of ID vertical collapse (listed as 'FFNN'), with the use of the same classifier head for the fine-tuning of the plain trans- former model (using the ID horizontal representation) and the value-aware transformer (using the 1.5D representation).
The value-aware transformers clearly outperform the other methods for these two tasks, as shown Table 3 Interestingly. the FFNNs using the quantilised values perform better than using when using the raw values, likely due to mixture of measurement sparsity and sensitivity to value extremities, indicating that there is some merit to the discretisation scheme we chose in Section 2.4 for this application.
4.4 EXPERIMENT LOS PREDICTION
To round off our experiments choose regression task, and length-of-stay another core problem faced by caregivers. The accurate prediction of LOS enables more efficient delivery of care. When the hospital is under strain run inefficiently there is negative impact patient outcomes.
We use the same sxeeffppcciiiatiin regression network for all models. This has the same archi- tecture for experiment 2, the only differences being single-class output and final soft-plus
8

---

Under review as a conference paper at ICLR 2022
Table In-hospital mortality classification performance on the Test set, with all metrics multiplied by 100 for ease of reading and the mean + std reported over runs.
Model
Data
Mortality Bal-acc
days AUROC
Mortality Bal-ace
days AUROC
FFNN raw values (mean)
57.7 + 4.4
61.0 + 3.3
3.0
66.1
+ 5.4
FFNN quants (mean, one-hot)
1
54.6
6.4
73.6 + 1.4
7.54
77.9 1.5
FFNN quants (mean, integer)
1
60.3 2.8
70.7
6.76
77.2 = 0.6
Transformer (depth 4)
1 1
76.1 1.3
6.65
77.7 0.2
Transformer (depth 6)
1 h
66.3 2.0
75.2 1.3
6..9
77.6 + 0.6
VA- Transformer (depth 4)
1.5
72.5
3.9
82.5 = 2.0
7.63
80.8 = 0.3
VA- PA-Transformer (depth 6)
1.5
71.6
2.5
81.3 = 1.2
72.4 + 0.5
81.0 + 0.3
Table 4: Remaining length-of-stay regression performance Test
Model
Data dim.
Remaining LOS at 48hrs MSE
R²
FFNN
raw values (mean)
I V 158.6 + 6.6
0.091 = 0.038
FFNN
quants (mean, one-hot)
1
131.5 0.9
0.246 0.005
FFNN quants (mean, integer)
1 V
131.6 1.3
0.246 0.007
Transformer (depth 4)
1 h
132.4
0.241 0.010
Transformer (depth 6)
1 h
131.5 $31.5:10 + 1.2
0.246 + 0.007
VA- Transffoorr (depth 4)
1.5
126.9 + 1.4
0.273 + 0.008
VA- Transformer (depth 6)
1.5
125.2 + 3.0
0.282 + 0.017
non-linearity, for the reason that remaining-LOS is a non-negative real number. We compare this regression network's ' performance when the input takes the form of the ID vertically collapsed data with pre-trained transformer trained on the ID horizontally collapsed data and value- aware trans- former trained on the 1.5D representation.
As can be seen from Table both the depth 4 and the depth 6 value-aware transformer surpass the competitor models by a healthy margin on the LOS regression task, providing further compelling evidence that additional information of worth is retained in the 1.5D input representation and that the value-aware transformer is able to convert it into superior downstream performance across range of tasks.
CONCLUSION
This paper is devoted to sparse sequential highly-multivarit data form which is found through- out the healthcare domain and whose value is difficult to unlock. We have analysed conventional approaches to modelling this data using representation learning methods, highlighting the ID vertical and horizontal collapses and the considerable imputation necessitated by models using multivariate sequence input.
Offering fresh perspective, we have reconceptualised the raw data as 1.5-dimensional and pre- sented a token-value form which retains both sequential structure and information about the values, without forcing considerable imputation before the point of input principles for the appropriate modelling of this data form, and based on these principles, made tar- geted adaptations to standard transformer architecture, efficiently tailoring it to the generation of 1.5D data.
model. We have outlined
In series of experiments the MIMIC-III lab-events dataset have demonstrated both the appropriateness of the 1.5D data representation and the suitability of the value-awe aware transformer by clearly surpassing competitive baselines both autoregressive model, and the challenging real-world in-hospital mortality (classification) and length-of-stay (regression) prediction tasks.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The codebase supporting this work has been made available with the submission. Included within it are: the scripts used to pre-process the data; a va-transformess package which is based on the great work of Wang (202I]: the scripts used for the pre-training, fine-tuning and baseline experi- ments. Supplementary details of the pre-processing and experimental settings are outlined in Ap- pendices|Aland The MIMIC-III dataset available at/https: mimicili/ 4/ after suitable completion of training.
REFERENCES
Emily Alsentzer, John R Murphy, Willie Boag. Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv: OrXiv:/904.03323, 2019.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14 14165, 2020.
Rich Caruana. What interpretable machine learning can tell us about missing val- ues. ICML Invited Talk, 2021. URL https:://liddsslvv..eeee
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Edward Choi, Zhen Xu, Yujia Li. Michael Dusenberry, Gerardo Flores, Emily Xue, and Andrew Dai. Learning the graphical structure of electronic health records with graph convolutional transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 606-613 6166613, 2020.
Jacob Devlin, Ming-Wet Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805 2018.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation, 101(2)):215--222, 2000.
Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Mul- titask learning and benchmarking with clinical time series data. Scientific Data, 6(1):96, 2019. ISSN 4463. doi: 10. 1038//41597-019-00000
URL https:: http:///t..orrro..../o
241159--2 01 9- 0103-9
Max Hor,, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4353 4363. PMLR, 13-18 Jul 2020.
Alistair EW Johnson, Tom Pollard, Lu Shen, Lehman Li-Wei, Mengling Feng. Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9,2000
Isotta Landi, Benjamin S Glicksberg, Hao-Chih Lee, Sarah Cherng, Giulia Landi, Mattco Danicletto, Joel T Dudley, Cesare Furlanello, and Riccardo Miotto. Deep representation learning of electronic health records to unlock patient stratification at scale. NPJ digital medicine, 3(1):1-11, 2020.
Yikuan Li, Shishir Rao, Jose Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshiee Behrt: transformer for electronic health records. Scientific reports, 10(1):1-12, 2020.
Karthika Mohan and Judea Pearl. Graphical models for processing missing data. Journal of the American Statistical Association, pp. 1-16, 2021.
10

---

Under review as a conference paper at ICLR 2022
Sanjay Purushotham, Chuizheng Meng, Zhengping Che. and Yan Liu. Benchmarking deep learning models on large healthcare datasets. Journal of Biomedical Informatics, 83:112- 134, 2018. ISSN 1532-0464. doi:
000.0.0..... URL hetps:
/ //www sc encedi rect. com/
Zhaozhi Qian, Ahmed Alaa, Alexis Bellot, Mihaela Schaar, and Jem Rashbass. Learning dynamic and personalized comorbidity networks from event data using deep diffusion processes. In Inter- national Conference on Artificial Intelligence and Statistics, pp. 3295-3305. PMLR, 2020.
Benjamin Shickel, Patrick James Tighe, Azra Bihorac, and Parisa Rashidi. Deep ehr: survey of recent advances in deep learning techniques for electronic health record (ehr) analysis. IEEE 2017.
journal of biomedical and health informatics,
Satya Narayan Shukla and Benjamin M Marlin. A survey principles, models and methods for learning from irregularly sampled time series. arXiv preprint arXiv:2012.00168. 2020.
Yuqi Si, Jingcheng Du, Zhao Li, Xiaoqian Jiang, Timothy Miller, Fei Wang, W Jim Zheng, and Kirk Roberts. Deep representation learning of patient data from electronic health records (ehr): A systematic review. Journal of Biomedical Informatics, pp. 103671, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 019955999099 2017.
Phil Wang- x-transformers: concise but fully-featured transformer, complete with of promising experimental features from various papers. 2021. URL https:://itthbb com/
Shirly Wang, Matthew BA McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C Hughes, and Tristan Naumann. Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-ili. In Proceedings of the ACM Conference on Health, Inference, and Learn- ing. pp. 222-235, 2020.
Zi Yin and Yuanyuan Shen. On the dimensionality of word embedding. arXiv preprint arXiv: 18/2,04224 2018.

---

Under review as a conference paper at ICLR 2022
A SUPPLEMENTARY DATASET INFORMATION
For visit to be included in our dataset it must have lasted at least 48 hours from admission to discharge or death. must have at least ten lab-events recorded and death must not occur at 48 hours or before. Those visits that met these criteria were then randomly split patient identifier into Train, Dev. and Test sets with the proportions 0.8 / 0.1 / 0.1. Some patients have multiple visits so it was important to partition based on the patient so not to introduce data leak. The partition statistics are detailed in 5 For our experiments, minimal processing of the data was performed collapse exploitable signal within the raw data (see Section [B for further details). We use 753 raw lab-event variables, and do not supplement them with static variables such as demographics, since this study is focused the effective utilisation of sequential data.
Table 5: MIMIC-III labs data statistics after preprocessing
Partition
Train
Dev.
Test
Patients
32989
4124
4124
# Visits
41403
5164
5072
Labs/visit mean
142.8
142.3
143.8
# Labs/visit median
127.0
128.0
129.0
Labs/visit 90%
248.0
245.7
250.0
Remaining LOS mean (hrs)
219.3
227.7
226.8
Remaining LOS median (hrs)
125.6
127.3
130.4
Mortality < 3 days
0.00841
0,00833
0.00848
Mortality > 7 days
0.04959
0.04493
0.05205
Table 6: Comparison of the number of imputations required for 1D, 1.5D and multivariate series
data representations for the MIMIC-III lab-events data. where: 1 = 753 is # variables, n 250 is number of sequential steps (cf. Table Vector-based is detailed in Shukla & Marlin (2020) and has varying sized input- we report the mean.
Data representation
# Inputs # Imputations required/visit (Train set) Mean
(% Input) Median (% Input)
ID (vertical)
753
696.9
(92.6%)
699
(92.8%)
ID (horizontal)
250
115.4
(46.1%)
123
(49.2%)
1.5D
500
230.7
(46.1%)
246
(49.2%)
Multivariate sequences
188250
188107.2 2
(99.9%)
188123
(99.9%)
Vector-based
8020.7
7877.9
(98.0%)
6627
(98.1%)
B EXPERIMENTAL SETTINGS & HYPERPARAMETERS
Finetuning for the value-aware transformer. Dropout of probability 0.5 the hidden layer of the classifier, combined with learning rates between the orders of magnitude le-4 to le-5 worked well across different iterations of our datasets and prediction tasks. We always used the same learning rate for model's fine-tuning as it was pre-trained with. We found that some minor pre-epoch exponential learning rate decay (Rx 0.9) lessened the deterioration due to over-fitting after reached its peak performance but the practical effect of this marginal on experiments after just a few epochs, both transformers and value-awan aware transformers were at their best.
Preprocessing of data for experiments. Minimal processing of the data was performed as not to collapse exploitable signal within the raw data. The visits in the Train set were used to extract per-lab quantile distributions following the procedure outlined in Section 2.4 From these derived distributions, we encoded each lab-event in the Train, Dev. and Test sets as ([LAB), [VAL]) ), with [VAL] [[XLOW), [LOW], [MTD], [HIGH], [XHTGH], [UNK]] J. Finally, with visits rep-
12

---

Under review as a conference paper at ICLR 2022
resented sequences of token-value pairs, append special end-of-sequence token [EOS] to mark the 48 hour point.
Experiment 1 We chose the input sequence length to be 250 for each of the sequence models, because this sequence length choice captures the entirety 90% of the visits in the Train set (cf. T) For the value-aware transformer we found that setting the loss hyperparameter to r 0.9 was most effective, putting greater weight on the learning of the tokens and counteracting the value token's greater at-random probability of being correctly predicted. Since the number tokens is Re 750 its baseline probability of correct prediction is approximately 0.0013, while the number of quantilised values is 10, hence the baseline chance of correctly predicting [VAL] is orders of magnitude higher, at around 0.1. Learning rates across the order of magnitude le-4 to le-5 were consistently performant for our data for both transformers and value-aware transformers. We used gradient-clippin and small amount of dropout in both feedforward and attention layers. We trained each until it hit an early-stopping threshold of epochs without improvement on the Dev. set.
Experiment To determine a competitive setting for the FFNN baselines we ran hyperparameter search against the Dev. set. The all-round best- best-performing hyperparameters from the search were then chosen and 5 separate training runs were performed for each model, checkpointed at the epoch with the best Dev. set loss. For the transformers and value-aware transformers, fine-tuned the pre-trained models from Experiment (Section 4.2) five times each from scratch, checkpointing them the epoch with the lowest Dev. set loss. The loss for all models was the same: class-weighted binary cross-entropy. The 5 runs of each competitor were then loaded from their checkpoints and evaluated against Test set, with the mean + standard deviation of key metrics reported in Table 3 The Dev. set performance is detailed in Tables 7 & g Each model was trained until it hit an early stopping threshold of epochs without improvement on the Dev. set.
Experiment 3 Exactly the same procedure was followed as for Experiment 2, except that the MSE loss was used for all models. The Test results are reported Table 4 The Dev. performance is detailed in Table g
13

---

Under review as a conference paper at ICLR 2022
C
DEV. SET PERFORMANCE FOR MAIN PAPER EXPERIMENTS
Table 7: In-hospital mortality I days classification performance on the Dev. set
Model
Data dim.
Mortality S 3 days
Balanced-accuracy
AUROC
FFNN raw values (mean)
V
0,557 = 0,049
0.638 + 0.024
FFNN quants (mean, one-hot)
V
0,556 0.077
0.784 0.008
FFNN quants (mean, integer)
V
0.604 = 0.037
0.732 + 0.008
Transformer (depth 4)
h
0.698 0.020
0.800 0.012
Transformer (depth 6)
h
0.694 0.013
0.797 + 0.009
VA-Transformer (depth
1.5
0.737 0.049
0.845 0.019
VA-Transformer (depth 6)
1.5
0.726 0.020
0.831 + 0.014
Table 8: In-hospital mortality > 7 days classification performance on the Dev. set
Model
Data dim.
Mortality days
Balanced-accuracy
AUROC
FFNN values (mean)
1
0,616 0.041
0.649 + 0.055
FFNN quants (mean, one-hot)
V
0.696 = 0.007
0.784 0.015
FFNN quants (mean, integer)
V
0.691 0.008
0.779 + 0.003
Transformer (depth 4)
h
0.685 0.004
0.773 0.004
Transformer (depth 6)
1 h
0.687 0.008
0.772 + 0.003
VA- TA-Transformer (depth 4)
1.5
0.727 0.005
0.817 + 0.006
VA- VA-Transformer (depth
1.5
0.731 0.007
0.816 0.004
Table 9: Remaining length-of-staa regression performance the Dev.
Model
Data dim.
Remaining LOS at 48hrs MSE
R²
FFNN
raw values (mean)
V
158.9 + 3.7
0.106 + 0.021
FFNN
quants (mean, one-hot)
1
131.2 + 1.0
0.262 + 0.006
FFNN quants (mean, integer)
1 V
131.1 1.3
0,262 0.007
Transformer (depth 4)
1 h
130.1 1.2
0.268 0.007
Transformer (depth 6)
129.6 1.0
0.271 + 0.006
-N-Trrnsformer (depth 4)
1.5
124.3 + 1.9
0.300 0.011
VA- Transformer (depth 6)
1.5
123.3 + 2.2
0.306 + 0.012
14

---

Under review as a conference paper at ICLR 2022
D TEMPORAL EMBEDDINGS
Each of the events in our experiments come with a timestamp. The natural way to utilise this timestamp with a (value-aware) transformer is for the positional embedding be function of time. There are now number of variations on the absolute positional embeddings introduced in Vaswani et 2017), with two prominent variations being relative positional embeddings and rotary embeddings (see Wang (2021) for implementations of these, and more). The simplest approach is to use the version of absolute positional embedding detailed in Horn al. (2020). To add the temporal embedding indicating time / [0, 7' to vector U E Rd, we first the vector r E I2d with components
T2i sin(t/T2//4
72 cos(1//22/44
for indices satisfying < 2i, 2/+ d- and where T is the maximum time relevant for the prob- lem hand. This vector is then added component-wist to each concatenated token-value embed- ding, before being fed into the attentional section of the model. While we extracted this information during preprocessing, using it in the experiments would give further advantage to the transformer and value-aware transformers over the non-sequential models, and we wanted this comparison to be fair possible.
E STRONG-COUPLING EXPERIMENT
For the value-aware transformer chose to loose-coupling method to input the 1.5D data using an embedd-concalenate-embbee input structure. This scales favourably: with the number of variable tokens T' and the number of value tokens V, loose-coupling requires vocabulary of + V distinct input tokens, while strong-coupling requires vocabulary of TV. For large T and V this is important especially on smaller datasets consisting of sparse data, where fine quantilisation would result in considerable token sparsity issues. For our experimental set-up with V = 8 and T 755, strong-coupling does in fact provided competitive inductive bias, performing worse than loose- coupling on LOS (Table IT), comparably on Mortality < 3 days (Table [2]. but better on Mortality days (Table 13}. This could due to V being small in our experiment, so that loose-coupling does not have the advantage it would were V of larger size.
Table 10: Pre-training performance: transformer using the 1.5D data with strong-coupling Dev. set. freex-val] indicates the cross- cross-entropy loss, and Prrok-vAL) the corresponding average probability of correct next-token prediction.
Model
Input form
C CIOR-VALL
ProO--VAL]
Random
0.0002
Transformer (depth
[TOK-VAL]
2.252
0.1052
Transformer (depth 6)
[TOK-VAL]
2.198
0.1110
15

---

Under review as a conference paper at ICLR 2022
Table 11: Comparison with strong-coupling [TOK-VAL] 1.5D form: remaining length-of-styy on the Test set
Model
Data dim.
Input form
Remaining LOS at 48hrs MSE
R²
Dev. set
Transformer (depth 4)
1,5
[TOK-VAL]
127.6 + 1.2
0.281 + 0.007
Transformer (depth
1,5
[TOK-VAL]
125.7 = 1.6
0.292 + 0.009
VA- TA-Trassformer (depth 4)
1.5
([TOK], [VAL])
124.3 = 1.9
0.300 + 0.011
VA- VA-Transformer (depth 6)
1.5
([VAL], [VAL])
123.3 = 2.2
0.306 = 0.012
Test set
Transformer (depth 4)
1.5
[TOK-VAL]
128.8 + 0.9
0.261 + 0.005
Transformer (depth 6)
1,5
[TOK-VAL]
128.4 2.8
0.264 + 0.016
VA- TA-Transformer (depth 4)
1.5
([TOK), [VAL])
126.9 + 1.4
0.273 1 0.008
YA-Transformer (depth 6)
1,5
([VAL), [VAL])
125.2 3.0
0.282 + 0.017
Table 12: Comparison with strong-coupling [TOK-VAL] of 1.5D form: in-hospital mortality 3 days
Model
Data dim.
Input form
Mortality 3 days
Balunced-accuracy
AUROC
Dev. set
Transformer (depth 4)
1.5
[TOK-VAL]
0.767 + 0.019
0.839 = 0.006
Transformer (depth 6)
1.5
[TOK-VAL]
0.737 0.049
0.842 1 0.015
VA- MA-Transformer (depth 4)
1.5
([TOK), [VAL])
0.737 + 0.049
0,845 + 0.019
VA- WA-Tranfformer (depth 6)
1.5
([TOK), [VAL])
0.726 0.020
0.831 + 0.014
Test set
Transformer (depth 4)
1.5
[TOK-VAL]
0.749 + 0.041
0.841 + 0.019
Transformer (depth
1.5
[TOK-VAL]
0.734 0.030
0.831 + 0.016
VA- PA-Transformer (depth 4)
1.5
([TOK), [VAL])
0.725 + 0.039
0.825 1: 0.020
VA- TA-nnsnsforeer (depth 6)
1.5
([TOK), [VAL])
0.716 + 0.025
0.813 + 0.012
Table 13: Comparison with strong-coupling (TOK-VAL) of 1.5D form: in in-hospital mortality > days
Model
Data dim.
Input form
Mortality > 7 days
Balanced-accuracy
AUROC
Dev. set
Transformer (depth
1.5
[TOK-VAL]
0.734 + 0.006
0.820 + 0.004
Transformer (depth 6)
1.5
[TOK-VAL)
0.729 + 0.006
0.819 + 0.003
VA TA-Trssfformer (depth 4)
1.5
([TOK), [VAL])
0.727 0.005
0.817 + 0.006
VA- TA-Transformer (depth 6)
1.5
([TOK), [VAL])
0.731 + 0.007
0.816 + 0.004
Test set
Transformer (depth 4)
1.5
[TOK-VAL]
0.723 + 0.010
0.816 = 0.006
Transformer (depth 6)
1.5
[TOK-VAL)
0.727 + 0.007
0.807 0.003
VA-Trane Tr-Tnssfforrrr (depth 4)
1.5
([TOK), [VAL])
0.723 + 0.016
0.808 +: 0.003
VA-Transformer (depth 6)
1.5
([TOK], [VAL])
0.724 0.005
0.810 + 0.003
16


Reviews:
[{"review_id": "k5THyONf-15", "reviewer": "Reviewer_ph4X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles an important challenge with modeling sequential data in healthcare, sparsity of events-- labs are generally ordered based on the patient context and may be ordered multiple times and it is important to account for the sequential nature of the event as well as the value for each event. The concept that is termed as 1.5-dimensional setting proposes to account for the value when learning the embeddings for the token which are the lab events with their values, an interesting approach for using transformers. However, there are certain concerns with respect to the design choices regarding the architecture as well as the training process such as the decay rate, the dimensions of the hidden layer, $\\gamma$. Moreover, the performance that is achieved for say in-hospital mortality is only sub-par with respect to other approaches for predicting in-hospital mortality that are largely neglected in this work. This creates another problem to assess the merits of the proposed method. While there has been considerable work (see 1-5 below), for predicting in-hospital mortality using MIMIC data, comparison with such baselines needs to be performed. \n\nThe paper can also be improved to make it better for the reader. Adding why the value-aware representations would be beneficial for downstream tasks in the introduction would help to situate the need for this work. Similarly, adding context about the heterogeneity of the data for patients' individual health can help. The choice for the number of dimensions for the token and value representation needs to be supported. While that for the values can be justified based on the quantization, there are concerns with the generalizability of the quantization process as well. I would encourage the authors to think about the sensitivity of the quantization to the population incorporated in the study. The eICU dataset provides a richer ground to handle these generalizability issues and could be checked. \n\n\n1. Song, Huan, et al. \"Attend and diagnose: Clinical time series analysis using attention models.\" Thirty-second AAAI conference on artificial intelligence. 2018.\n2. Purushotham, Sanjay, et al. \"Benchmark of deep learning models on large healthcare mimic datasets.\" arXiv preprint arXiv:1710.08531 (2017).\n3. Che, Zhengping, et al. \"Recurrent neural networks for multivariate time series with missing values.\" Scientific reports 8.1 (2018): 1-12.\n4. Jarrett, Daniel, et al. \"Clairvoyance: A pipeline toolkit for medical time series.\" International Conference on Learning Representations. 2020.\n5. Li, Ke, et al. \"Predicting in-hospital mortality in ICU patients with sepsis using gradient boosting decision tree.\" Medicine 100.19 (2021).", "summary_of_the_paper": "The paper proposes an approach for learning representations of spare sequential data, commonly found in healthcare settings. The work challenges issues with sequential data in electronic health records such as sparsity of events, unequal lengths, which also present imputation challenges. A transformer-based approach is presented which incorporates information about the different sequential events (labs) along with their values which is termed as a 1.5-dimensional representation. The transformer is trained to obtain the representation of sequences of events, labs conducted for in-hospital patients, along with the values of the labs which are categorized based on the quantile range obtained from the training data. The representations from the transformer embeddings are used for predicting two outcomes, 1) in-hospital mortality, and 2) length of stay at 48 hours after admission. The value- aware transformer performs marginally better than a value-unaware transformer with respect to $\\mathcal{L_{\\text{\\[TOK\\]}}}$. There is also an improvement in the balanced-accuracy and ROC-AUC for in-hospital mortality when compared with feed-froward neural network and value-unaware transformer. Similar performance characteristics are also observed for length-of-stay regression. ", "summary_of_the_review": "In general, the paper suffers from clarity as well as comparison with existing baseline other than transformer-based approaches and feed-forward networks. Assessing the performance of the model on different subpopulations based on demographics would be helpful to identify how the method performs subgroups rather than on average since it is known that there are differences in lab-orderings based on demographic attributes such as gender and self-reported race. This could strengthen the contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ebVGSgpKIv2", "reviewer": "Reviewer_KQ5r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "> Good motivation: The motivation of solving the representing sparse sequential highly-multivariate data is novel. \n\n> Authors proposed the good idea in overall; however there are few heuristics involved: I agree that proposing the 1.5D representation and value-aware transformer are novel and quite good idea. However, when proposing the 1.5D representation, I could not find the mathematical grounds for few factors: For example, there is neither clear reason nor ablative studies for selecting 5 degreed values (ie. XLow to XHIGH). \n\n> Good performance while few ablative studies are missing: I could find that 1D representation have been explicitly experimented during the experiments; however I could not find the baseline that represents signals in the form of multivariate series data and applies corresponding models. I think this baseline is required as the evidence for authors' insist that the sparse sequential highly-multivariate data could be better represented in 1.5D compared to multivariate representation.\n", "summary_of_the_paper": "Authors proposed the method that could represent the sparse sequential highly-multivariate data, which could be trivially represented as neither 1D nor dense multivariate series data. Authors proposed the 1.5D representation which is composed of token-value pair and proposed the value-aware transformer that is able to use the representation. Experimental results on in-patient laboratory data showed the effectiveness of their data representation and algorithms (ie. value-aware transformer).", "summary_of_the_review": "Due to the reasons I stated in the main review, I am currently in the borderline accept; however effectively rebutting my comments will make me raise my voting towards higher score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HH4tzt7ANa", "reviewer": "Reviewer_qFRi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n1. The paper focuses on the task of modeling sparse and multivariate sequential data which is an important and practical problem.\n2. The paper is easy to follow.\n\nConcerns:\n1. The authors are missing several important related works and comparisons with recent and SOTA methods for sparse and multivariate sequential data [1, 2, 3].\n2. The main contribution of the paper is not clear. \n3. The authors focus on the data representation for the sparse and multivariate sequential data but fail to compare it with some of the similar data representations introduced in [3] and [4].\n4. The proposed framework has marginal novelty in terms of the model architecture as it is a fairly standard transformer architecture. \n5. The authors should provide experiments to show if the design principles hold in practice, in particular, provide a comparison with strong coupling.\n6. The authors should perform experiments with more datasets to show the effectiveness of the model.\n\nAdditional Comments:\n1. How did the authors choose the dimension of the token and value embedding?\n2. Why did the authors choose to not consider the temporal information such as the time of the lab events which is fairly common to consider in several works in this domain?\n3. What happens if multiple lab events are observed at the same time? How is the order decided in that case? \n4. Could the authors clarify more about the difference between CLS and EOS token and why it has to be added in the end? Based on my understanding of transformer architectures it doesn't really matter where the token is added. \n5. Is the padding used because of the variable length of sequence from different examples?\n\n\nReferences:\n1. Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural\nnetworks for multivariate time series with missing values. Scientific Reports, 8(1):6085, 2018.\n2. S. N. Shukla and B. Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021.\n3. Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for\ntime series. In Proceedings of the 25th International Conference on Machine Learning, 2020.\n4. S. N. Shukla and B. M. Marlin. A Survey on Principles, Models, and Methods for Learning from Irregularly Sampled Time Series. CoRR, abs/2012.00168, 2020.\n", "summary_of_the_paper": "This work proposes an approach for modeling sparse and multivariate sequential data which are commonly found in the healthcare domain. The paper presents and compares different data representations and introduces a 1.5d representation and outlines principles for modeling this type of data. The paper presents the value-aware transformer, a transformer decoder-only architecture for the 1.5d data representation. Based on the experiments on the MIMIC-III dataset, the proposed model and the data representation achieve better performance than several other standard data representations. ", "summary_of_the_review": "The contributions of the paper are not clear. The paper is also missing important related works and comparisons with recent approaches in this domain. The paper has only marginal novelty both in terms of the data representation and the proposed approach.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper propose a value-aware transformer for sparse multivariate time series data. While to approach is well motivated and the problem well-motivated from a clinical viewpoint, the comparison with related work brought up by reviewer qFRi and  reviewer ph4X would really make it clear where this paper stands. The authors attempt to diffuse this issue in their replies, but empirical comparisons in the paper would guide practitioners more. This is especially important as the paper is motivated by a real-world problem.
Decision: Reject