Compositionality in Time Series: A Proof of Concept using Symbolic Dynamics and Compositional Data Augmentation
Authors: Michael Hagmann
Michael Staniek
Stefan Riezler
Venue: Accepted by TMLR ()

Abstract: This work investigates whether time series of natural phenomena can be understood as being generated by sequences of latent states which are ordered in systematic and regular ways. We focus on clinical time series and ask whether clinical measurements can be interpreted as being generated by meaningful physiological states whose succession follows systematic principles. Uncovering the underlying compositional structure will allow us to create synthetic data to alleviate the notorious problem of sparse and low-resource data settings in clinical time series forecasting, and deepen our understanding of clinical data.
We start by conceptualizing compositionality for time series as a property of the data generation process, and then study data-driven procedures that can reconstruct the elementary states and composition rules of this process.
We evaluate the success of this methods using two empirical tests originating from a domain adaptation perspective.
Both tests infer the similarity of the original time series distribution and the synthetic time series distribution from the similarity of expected risk of time series forecasting models trained and tested on original and synthesized data in specific ways.
Our experimental results show that the test set performance achieved by training on compositionally synthesized data is comparable to training on original clinical time series data, and that evaluation of models on compositionally synthesized test data shows similar results to evaluating on original test data. 
In both experiments, performance based on compositionally synthesized data by far surpasses that based on synthetic data that were created by randomization-based data augmentation. 
An additional downstream evaluation of the prediction task of sequential organ failure assessment (SOFA) scores shows significant performance gains when model training is entirely based on compositionally synthesized data compared to training on original data, with improvements increasing with the size of the synthesized training set.

Published in Transactions on Machine Learning Research (03/2025)
Compositionality in Time Series: A Proof of Concept using
Symbolic Dynamics and Compositional Data Augmentation
Michael Hagmann
Sagmona@dd.nii-heideeeen
Deportment of Computational Linguistics Heidelbery Universily, Germany
Michael Stanick
stanick@oc.um--hccddleeeyy.
Department of Computational Linguistics
Heidelberg University, Germany
Stefan Riezler
riealer@dl.mii-heideee de
Department of Computational Linguistics and Interdiscilinaryy Center for Scientific Computing (TWR) Heidelbery University, Germany
Reviewed on OpenReview: https: // openreview. net/ forum? id= ms IO2LXVVJ
Abstract
This work investigates whether time series of natural phenomena can be understood as being generated by sequences of latent states which are ordered in systematic and regu- lar ways. We focus on clinical time series and ask whether clinical measurements can be interpreted being generated by meaningful physiological states whose succession follows systematic principles. Uncovering the underlying compositional structure will allow us to create synthetic data to alleviate the notorious problem of sparse and low-resource data settings in clinical time series forecasting, and deepen our understanding of clinical data. We start by conceptualizing compositionality for time series as a property of the data gen- cration process, and then study data-driven procedures that can reconstruct the elementary states and composition rules of this process. We evaluate the success of this methods ing two empirical tests originating from domain adaptation perspective. Both tests infer the similarity of the original time series distribution and the synthetic time series distri- bution from the similarity of expected risk of time series forecasting models trained and tested on original and synthesized data in specific ways. Our experimental results show that the test set performance achieved by training on compositionally synthesized data is comparable to training on original clinical time series data, and that evaluation of models on compositionally synthesized test data shows similar results evaluating on original test data. In both experiments, performance basod compositionally synthesized data by far surpasses that based on synthetic data that created by randomization--asse data aug- mentation. An additional downstream evaluation of the prediction task of sequential organ failure assessment (SOFA) scores shows significant performance gains when model training entirely based on compositionally synthesized data compared to training on original data, with improvements increasing with the size of the synthesized training set.
1 Introduction
Compositionality describes the systematic capacity a system to generate an unbounded mumber of valid outputs based on novel combinations from a finite set of clementary components. This capacity is considered one of the main pillars of the human ability generalize to new tasks and situations (Lake al., 2017). Standard examples for compositionality in human and machine intelligence natural language and vision.

---

Published in Transactions on Machine Learning Research (03/2025)
for example, the capacity to build an infinite number of sentences from finite vocabulary, the composition of images from elementary concepts of color, position, and shape. Formal accounts of compositionality arose independently in logic, natural language processing, and computer science, beginming in the 19th century (see
Partee (1984), Janssen 2012), or Szabó (2020) for historical overviews). The concept of compositionality currently experiences a renaissance in machine learning under the label of compositional generalization the ability to systematically generalize to test data that are composed from known components seen in novel combinations. The goal of this recent research strand is to understand and formalize the concept of compositionality, and to inject a compositional inductive bias into state-of-the art generative models, aiming at improved one-shot compositional generalization (Patel et al. 2022) from primitives seen during training to unseen novel combinations at test time.
Our goal is to investigate whether the concept of compositionality can be applied to the less obvious domain of time series, focusing on multivariate and sparse time series of clinical measurements of intensive care patients collected in electronic health records. We start from an algobraic formalization of compositionality as property of the data generation process which crucially depends on a homomorphism that maps physi- ological states to clinical observations preserving compositional structure. Compositionality of clinical time series then mcans that the clinical assessment of a sequence of physiological states is a function of the clinical assessments of its constituents and the way they are sequentially ordered. On the one hand, this interpreta- tion motivates the question of meaningful physiological states and the systematics of their ordering. On the other hand, it motivates a data-driven pipeline that deconstructs the data generation process to empirically detect elementary states and composition rules. An algorithm that discovers elementary components and composition rules will allow us to create synthetic data to alleviate the notorious problem of sparse and low-resource data scttings in clinical time scries forccasting. Furthermore, standing of the compositional data generation process underlying time series data from a domain adaptation perspective. The intuition is to infer that original time series data and compositionally synthesized data have the same distribution if domain adaptation from synthesized to original data is successful.
can gain a thcoretical under-
The experimental part of this work is built on an empirical pipeline that uses representation learning, in particular clustering of learned representations of time series subsequences (Ghaderi et al., 2023 Ma et al,, 2019), to operationalize a notion of latent classes representing meaningful physiological states, for example, healthy unhealthy states of certain organ systems. Mapping time series to symbolic representations makes them amenable to compositional methods developed for natural language processing (NTP). Here we apply a simple but entirely data-driven approach to inducing compositional structure from symbolic representations of time scries (Andreas] 2020). This algorithm implements the distributional principle (Firth 1957) to exchange subsequences that occur in the same context. to yield other valid sequences.
The theoretical contribution of this work is to motivate empirically testable criteria for the compositionality of the data generation process of time series in domain adaptation theory (Ben-Davit et al., 2006 2010b a Ben-David & Urner 2014). We exploit the assumption that successful domain adaptation requires small distance between source distribution (in our case, the distribution underlying compositionally synthesized data) and target distribution (in our case, original time series data), to infer a small distance between the underlying distributions of synthesized and original data from empirically testing the success of domain adaptation from synthetic to original data. Our first test evaluates compositionally synthesized time series by analyzing their utility for the training of a time scries forecasting (TSF) model. Our experiments demonstrate comparable test set performance of models trained on compositionally synthetic data to models trained on the original data for MIMIC-111 (Johnson et al., 2016) and eICU (Pollard et al. 2018) ) data sets. Our second test compares the use of compositionally synthesized data against original time scries data as test data in TSF tasks. Our empirical results show that both test sets yield a similar test set performance for a TSF model trained on original time series data. Furthermore, we find that compositionally synthesized data is much closer to the original data than synthetic data created by a non-compositional data augmentation algorithm Yuun et al. 2019).
In sum, the contribution of our work are as follows: We present analysis based on symbolic repre- sentations and compositional structure that allows an understanding of clinical time series as symbols that
1 |Code reproduce the experiments described in this paper is available at httpe://ggttuub..oo/Sttttttttt 2025_ compositional__ tiie__errieeeeee
2

---

Published in Transactions on Machine Learning Research (03/2025)
are emitted by an ordered sequence of physiological states. The underlying algorithm allows to create synthetic training and test data that are on par with the original data in real-world clinical time series forecasting task and show increasing improvements in the size of synthesized data in downstream tasks. This circumvents the notorious problem of sparse and low-resource data settings in clinical TSF. Furthermore, we present empirically lestable criteria for compositionaliyy rooted in domain adaptation theory, allowing broader claims on the compositionality of the data generation process underlying general time series data.
2 Related Work
Symbolic Compositional Structure in NLP. A connection of our work to NLP can be drawn by viewing natural language sentences as discrete time series of symbols, and by considering the task of next word prediction in language modeling as a TSF task. The central models developed in symbolic NLP the dominant paradigm in 20th century NLP were all explicit symbolic generalive processes allowing to construct an infinite number of sentences from a finite alphabet (vocabulary) and a finite recursive device (grammar ? Since NLP rescarch has long departed from the symbolic paradigm, the central question in current research on compositional generalization in NLP consequently an investigation of the compositional skills of non-symbolic neural network architectures. One research strand compositionality in non-symbolic NLP focuses on the creation of benchmark datasets to evaluate the compositional generalization abilities of neural networks (Lake & Baroni 2018);; Keysers et al. (2020); Kim & Linzen (2020). inter alia). Most such datasets are based on the NLP task of semantic parsing, where the components and composition rules are obvious and can be clearly defined. Compositionality is then quantified by measuring accuracy on test sets with a similar component distribution, but different compound distribution. Another rescarch strand focuses on directly injecting compositional inductive bias into neural sequence models (Russin et al. (2020): : Huang et al. 2002); ;; Sartran et al. (2022). inter alia). Our approach directly builds on work compositional data augmentation Andreas 2020); Akyürek et al. (2021) Qiu et al. (2022) that aims to provide a compositional inductive bias to otate-of-the-ars sequence learning models by adding compositionally generated data to their training sets.
Disentangled Representation Learning in Vision. The goal of disentangled representation learning in vision is to separate informative factors such that a change in a single latent factor lcads to change in single factor in the learned representation (Bengio et al. 2013). The state-of-the-art models in this area are auto- encoders that directly aim to reconstruct the generative factors of variation (Higgins et al. (2017): Montero let al. (2021): Xu et al. (2022), inter alia). The generalization abilities of these models are usually tested by the task of reconstructing compositionally generated test data that include combinations of generative factors that were not seen during training. However, it has been shown that unsupervised disentangled representation learning is impossible without inductive biases on both models and data (Locatello et al. 2019), and that increased disentanglement does not increase gencralization capabilitics, especially if one moves away from a simple artificial data set to real-world data (Schott et al. 2022 Montero et al.) 2022) Wiedemer et al. (2023) formalize compositionality as a property of the data generation process, assuming the composition function, as well as the latent description of the observations to be known, effectively reducing the learning task to a reconstruction of the component functions. In contrast to this work, we cannot formalize compositional generalization as a direct reconstruction problem since neither latent factors nor the composition function are known for time series. Instcad, we indirectly test compositional generalization by evaluating the utility of compositionally synthesized data as training and test data in real-world TSF tasks.
Domain Adaptation. Our work takes crucial inspiration from the work of Ben-David et al. (2006; 2010bla): Ben-David & Urner 2014) to motivate empirically testable criteria for compositionality in do- main adaptation theory. Starting from theoretical results that identify necessary and sufficent conditions for successful DA, we created two experimental setups that allow us to infer a small distance between syn- thetic and original data based on the distance between expected risks estimates. Similar evaluation setups of training and testing on original and synthetic data have been used by Esteban et al. (017)) however,
2yymholic approaches dominated the NLP fields of syntax and semantics, starting with Chomsky 1957 and Montague 1970 respectively, and are still relevant in formal language theory in computer science (starting with Chomsky (159
3

---

Published in Transactions on Machine Learning Research (03/2025)
without theoretical motivation. Furthermore, our work shows how to generate synthetic data distribution from an original sample with the properties of matching the original distribution on the level of elementary components, but differing from it at the level of compounds. Our results can be seen as strong hints at the potential of a formal study of compositional data synthesization, with the goal of a better understanding of its generalization properties.
Data Augmentation. Data augmentation has grown to an important subfield of machine learning on its own, with a substantial body of work already being devoted to data augmentation for time series (Yoon all 2019 Pei et al.] 2021 Wen et al., 2021). We would like to stress that the goal of our work is not present optimal data augmentation method for time series, but to analyze time scries from the perspective of compositional data generation, and to use empirically testable criteria on compositional data synthesization as a proof-of-concept for the validity of this perspective. Insights into the compositional nature of time series can only be gained systematic ablation studies where the contribution of compositionality can be isolated, removed and replaced by random sampling. This is what CutMix (Yun el al.) 2019) provides in our case a randomized variant of our compositional data augmentation method. Mixed-based data, augmentation approaches have been successfully applied to various machine learning tasks (Cao et al.] 2024), including the area of (physiological) time series data (Guo et al., 2023 Yang & Desell 2022). Our work provides empirical evidence for an ad advantage of augmentation techniques that mimic compositional data generation processes over randomization-boee based data augmentation.
Symbolic Dynamics. Research on transforming real-valued time series into symbolic representations has a long history (sce, for example, Williams| (2004) for an overview), where the central application is the analysis of dynamical systems (Lind & Marcus] 1995). Since the number of possible symbol assignments grows exponentially with the dimension of the time series, the traditional approach of partitioning the multidimenssonall phase space spanned by the input variables of a time series into finitely many pieces and then labeling each partition by specific symbol is only feasible for very low dimensional time series. This problem is overcome by representation learning approaches that first map time series into an embedding space (whose dimensionality can be controlled), where clustering methods are then applied to partition the space (Ma et. al. 2019: Ghaderi et al. 2023). We employ the latter techniques to learn a symbolic vocabulary that contains the elementary components of our compositional data synthesization process, and compare it to randomized version of traditional symbolic dynamics.
Pattern Recognition in Time Series Analysis. Work on time series motifs aims at the detection of elementary components of time series by identifying short segments that repeat themselves approximately the same within one larger time series (Patel et al.) 2002. Schäfer & Leser 2022). This is orthogonal to our problem of identifying elementary components as time series segments that appear in similar contexts across different time series. Furthermore, detection of motifs in time series is mostly applied to waveform data, e.g., quasi-continuous vital signals such as ECG recordings, whereas our framework is applied to vital measurements where at most one datapoint is taken per hour.
3
Compositionality in Time Series
3.1 Compositional Data Generation
We conceptualize compositionality as a property of the data generating process, following an algebraic formalization of compositionality (Montague) 1970 Partee 1984
Szabó, 2020).
Definition 1. (Compositional data generation.) Let (2.C2) be an algebraic structure of latent states where Cz is called the latent state ionposition function, and let (X' Cx) be an algebraic structure (of the same type) of observations where Cx is called the observation composition function. Furthermore, let yp: Z + X be a homomorphism that maps latent states to observations. We call data generating process f 2 + X that satisfies
f(a (e.....,.k)). 2k) = (CC zk)) Cx(pp()).....
(1)
compositional data generation process.
4

---

Published in Transactions on Machine Learning Research (03/2025)
state
lug
composition
cardiac fallure
MAP
Systemic
infection
PCT
Figure Conceptualization of compositional data generation process for time series of clinical variables: Physiological states are mapped observations of clinical measurements which are composed into a full time series. Healthy physiological states are realized by observations consisting of default measurements of vital signals lab measurements. Physiological states representing failure of organ systems like heart lung are realized by observation vectors where the measurements of mean arterial pressure (MAP) oxygen saturation (sO2) are out of range defined as healthy, a systemic infection causing an elevated procalcitonin (PCT) measurement. The clinical assessment, of this example can be interpreted as a clinical time series of a septic patient with multiple organ system failure, starting with healthy measurements, which are followed by indicators of lung failure and cardiac failure, consequent indicators of a systemic infection.
The data generating process of clinical time series can be understood as compositional by the following definition. Let latent states be physiological states of patients, Cz be the progression of a patient through these states, observations be clinical measurements, and Cx be the temporal regularities within time series. Then Equation D can be interpreted as follows:
The clinical assessment of a sequence of physiological states is a function of the clinical asscssments of its constituents and the way they are sequentially ordered.
The right-hand side form of Equation 1 can be illustrated by the generative process shown in Figure 1 This process starts from physiological states of patients, which emit observable components from the space of clinical measurements, which are ordered over time into a full sequence of clinical measurements. The left-hand side of Equation [ motivates a reconntruction of this process into elementary representations of latent states, which are composod to a sequence of latent states that is mapped sequence of obsorvations.
This deconstruction process builds the basis of the experimental work presented in this paper and is illustrated in Figure 2 Starting from input data of multivariate clinical time series, extract real-valued vector reprosentation of subsequences from the hidden states of TSF model, which is then fed into k-mcans clustering algorithm {Lloyd] 1982) that allows us to assign a symbolic representation to time series based on the cluster membership its subsequences. These symbolic representations are the input for an entirely data-driven compositional data augmentation algorithm (Andreas) (020) that results in an implicit set of rules allowing us synthesize novel time series in a compositional manner. Each component of this pipeline will be described in more detail below.

---

Published in Transactions on Machine Learning Research (03/2025)
C.
A
-
multiarritte time series
clustering
symbolization
composition
Figure 2: Empirical deconstruction of multivariate time series for compositional data synthesization: First, subsequences of time series are clustered and each cluster is assigned a symbolic label. Based on symbolic representations of time series sub-sequences, compositional data augmentation algorithms can be applied to synthesize full time series in a compositional manner.
3.2 A Domain Adaptation Perspective on Compositionality
Viewing the above sketched compositional data augmentation process from domain adaptation (DA) per- spective allows
arrive at two empirically testable conditions for compositionality. These tests exploit the fact that the training and test distribution of model are not identical in DA scenarios. For both of our empirical tests, one of these distributions will be synthetically created by a compositional algorithm, and the other distribution will be the original distribution whose compositional status is unknown. The compositionality of the data generation process underlying the original time series data can then be inferred from successful DA from compositionally syuthesized to original data.
Our first test basod on thcorctical results established by Ben-David et al. (2006 2010b a). which provide two jointly necessary and sufficient conditions for successful domain adaptation, one of which is a large enough similarity between training and target distributions. We leverage these results to motivate
empirically
testable critcrion based on the following rationale. We apply a learning algorithm to both the original training data and compositionally synthesized training data and evaluate the trained models on the original test data. Tf the estimated expected risks are identical, DA is considered successful and according to the results established by Ben-David et al. (2006) 2010b a), we can conclude that both distributions must identical. Thus, we can conclude that the original data generating process must be compositional.
Our second test is also grounded in the theory of DA and leverages a result that extends observation by David & Urner (2014). Our Theorem data distributions to the proximity of these distributions. We leverage this theorem to motivate second empirically testable criterion based on the following rationale. We evaluate a trained learner on both the original test set and a compositionally synthesized test set, and estimate the ratio of the corresponding expected risks. As established by Theorem [ the magnitude of this ratio allows drawing a conclusion about the proximity between the synthetic and original distributions. If this ratio is one, both distributions are identical, and hence the original data generating process must be compositional.
[ relates the proximity of the expected risks of model two
In the following, we briefly summarize the theoretical concepts of DA theory, especially the necessary and sufficient conditions for successful DA.
Lot X be a domain, y be a co-domain, and P and Q be distributions over X' x V. Furthormore, let Px and Ox denote the respective marginal distributions on X. Lei H C y be a hypothesis class of functions h, and let e (y. 9) be loss function on the target labels y and the predicted labels = h(x). We can now define the concepts of domain adaptation learner (Definition 2 and a concept quantifying domain adaptation learnability (Definition 3
Definition (Domain adaptation learner.) We call a learning algorithm A that receives training samples from Q but whose expected risk is evaluated P (conscrvative) domain adaptation learner.
Definition 3. ((c, Jeaaaaaraalllyyy Let P and Q be distributions on X y with common support. H. a hypothesis class, and A be domain adaptation learner. We say that (c, 6)-earns P from Q relative to H. if for chosen e, 8 0 there exists N such that when provided training sample T of size obtained
6

---

Published in Transactions on Machine Learning Research (03/2025)
from Q, with probability of at least 1 1-6 8 (over the sample space), the expected risk with respect to P of the obtained classifier hr == A (T) does not exceed the expected risk Ep [H] = classifier on P by more than
inf Ep [K(, k(x))] of the best
NER
P (Ep [K (r(x)) Ep [7]
(2)
Obviously DA works well if Ep [e(y, lr(()))] - Ep [H] can be bounded by small e, with high probability (small 8) for feasible sample sizes n. A further popular assumption in the context of DA is the covariate shift assumption:
Definition 4. (Covariate shift.) Let P and Q be distributions on X y with common support. Then P and Q satisfy the covariate shift assumption if the conditional distributions Pyjx and Ωylx are identical.
The covariate shift assumption is a rather weak requirement that only demands that the stochastic relation between r and be identical for Q and P. This assumption is only a necessary, but not a sufficient, condition for a successful DA. This fact is exemplified by the following thought experiment. Let us assume that we trained a model on 2, and that the difference between Q and P is such that Qx places a lot of mass on inputs that the learner prodicts well, and less on inputs where the learner performs poorly. In addition, assume that the situation is exactly the opposite for Px. Then the expected risk with respect to Q will be small, but the expected risk with respect to P will be large. This thought experiment illustrates the need to put constraint on the difference between Px and Qx. Since the seminal work of Ben-David et al. (2006). it is common to express this difference in terms of the so-called A-distance:
Definition 5. (A-distance.) Let Px and Qx be two distributions on X' and A S 2 such that each set in A is measurable with respect to both distributions. Then the A-distance between Px and Qx is
A (Px, 2x) = 2 sup (Px(A)) - 2x(44))
A'EA
It obviously is sufficient to consider only those domain subsets where the potential hypotheses prodict different outputs HAH == {{x E X Mxx) + 1'(x)} h1, E H}. Since we select hypothesis that should perform well with respect to P based solely on the information from Q, we need to assume that there exists a hypothesis h² that performs well on both distributions. This hypothesis has been named the low-error joint predictor by Ben--avvd et al. (2006):
Definition (Low-error joint predictor.) We call a hypothesis h" a low-error joint prodictor,
Ep [e(,, h" (x)) + Eg [f(y, h² (x)) R Ep [H] + Eg [H]
(4)
were Ep [H] == inf Ep [e(v, ((x)) and Eg [H] == inf Ee ([,, h(x))] NEW
NEW
It has been shown by Ben-David et al. 2006 2010b) that the combination of small distance duAR (Px: Qx) and the existence of ht" is necessary and sulficient. for A to be (c,5)-learner (making the covariate shift assumption redundant if both of the former conditions hold).
For our first proposed test, we train a model once on the original training data to get an estimate of Ep [70, and also on compositionally synthesized training data to get an estimate for Rp [e(y, hy (x))]- Calculating the difference between these estimates allows us to estimate an approximate magnitude for c. If this magnitude is small, DA was successful. This allows us to infer a small A-distance between the compositionally syuthesized and the original data distributions. Hence, we conclude that the original data generating process must be compositional as well.
If 1t" does not exist, we can still lest. the similarity of P and Q under the covariate shift assumption based on the expected risk ratio. The following theorem takes inspiration from Ben-David & Urner (2014) who observed that a bounded ratio between Px and Qx implies a bounded ratio of the expected risks calculated with respect to and Q for any learner. A proof is given in Appendix A.3
Theorem 1. (Bounded expected risk and density.) Let P and be two absolutely contimnous probability distributions on X' x y for which the covariate shift assumption holds such that fr(#,)) v) = f(y <r)((>)))))))))))))))))()))))<) and
7

---

Published in Transactions on Machine Learning Research (03/2025)
fz(2,)) fly f))((z) are the corresponding densities. Further let H be hypotheses class of learners, 82 0 be loss function, and C.C E IR+. Then,
Vx E X: Cfe(#) = fx(x) (f)(2 = vh E H: CE (z. h(x)) & Ep (((x, = (x,
Further C and 2 1.
Under the rather unproblematic assumption that Ee [e (x, h(x))) > O, and by defining fr()) in X whose densities are zero, we obtain bounds on the ratios of densities and expected risks:
for points
/e(w)
fr(2)
Ep (x, SC.
Vo E X: C
SC - vh H: C T0(
(6)
EØ e (()))
We utilize this relation to define a sccond empirically testable criterion. We evaluate trained learner on the original test set to estimate Ep [ (a, h(x))], and the compositionally synthesized test. set to estimate Ee [e (x, h(z))]. The ratio of these estimates is estimate of the corresponding expected risk ratio. This allows us to assess whether C and C are close to one, and by the above result, to infer whether fp (2(2,0) v) and fa( y) are close. If so, we conclude that the original data generating process is compositional as well.
4 Methods
4.1
Symbolic Dynamics
The empirical deconstruction process of multivariate time scries shown in Figure [ first needs to identify elementary components that can in second step be used to synthesize new time series in composi- tional manner. Since the composition technique used in the second step relics on discrete representations of sub-segments of time series, we need to transform sub-sequences of real-valued vectors into symbolic representations. The central concept in this context is the notion of a symbol space given in Definition
Definition 7. (Symbol space.) Let (X' d) be finite dimensional metric space with metric d and cj,..., X. Then the partition of X' given by:
S.
I IEXIC = arg argmin (d(z,,)))) j=.....k
for all i = ,,...,k k is called the symbol space of X' with symbols S, and centroids
In order to transform time scries into chain of symbols, break n-dimensional multivariate time series of length T into consecutive non-overlapping blocks of length A subject to T mod A 0. Next we map these blocks to their corresponding symbol in the symbol space and arrange them in the same order as the blocks. The domain of this mapping can be either the original input space MA.A, or the space of the learned neural block representations. In the first case, a straightforward application of traditional symbolic dynamics methods [Lind & Marcus, 1995; Williams) 2004) would require an alphabet size that grows exponentially with the number of input dinensions. This infeasible even for multivariate time scries of around 100 clinical variables. An approach to circunvent this problem is to randomly select feasible number of centroids in the input domain MA,A.
In the second case, we exploit the representations Iearned by neural network and apply k-mean clustering to create a symbolic representation of the time series with a computational learning cost that is linear in the embedding dimension. The learned representations consist of the hidden states of the encoder of the TSF Transformer described in Scction [.1 This Transformer model was trained to predict three future hours based on the current three hours on the training set, with hyperparameter settings as described in Appendix A.1 (except. the hidden size set to 50).
8

---

down Published in Transactions on Machine Learning Research (03/2025)
M
A
N
She
picks
the suitease
up
M
C
N
training data
She
puts
the suitcase
down
X
A
Y
B
He
picks
the box
up
X
C
Y
augmentation
He
puts
the box
Identify interchangeable fragments as (discontinuous) symbol soquences that appear in similar con- lexts (enwironments) in the training data, for example, A... B and C... in context M...N.
Find additional contexts in which interchangeable fragments appear, forming a template for substi- tuting in another fragment, for example, context X...Y Y for fragment A...B.
Augment the data with synthesized training example by exchanging fragments in a template, for example, substitute C...D D for A... ... B in context X X...Y.
Figure 3: Compositional data synthesization procedure for symbol sequences adapted from Andreas (2020) Symbol sequences A... B, C... D, ....N, and X... Y are generic and can be discontinuous. An illustration with natural language sentences if given in the respective second lines.
4.2 Data-Driven Compositional Data Synthesization
Andreas 2020) presented linguistically motivated data-driven approach induce compositional structure in symbol sequences in NLP. This algorithm can be readily applied to induce compositional structure from symbolic representations of time series. The key inspiration of the algorithm is the distributional strucbure of language (Harris 1954) according to which "You shall know word by the company it keeps!" [Fithh 1957). This principle grounded in the fact that words that are used and occur in the same contexts tend to purport similar meanings. The method of Anddeass (2020) exploits the distributional principle to generate novel valid sequences by exchanging subsoquences with similar meanings into new contexts An illustrative example of the distributional structure of language and how the method uses it is given in Figure 3
In the following, we will present formal account of the compositional data synthesization method illustrated in Figure 3 We will introduce the notions of fragment, template, and environment of a sequence, and give a formal definition of the synthesization process and provide pseudo-code for the algorithm used in our work (see Algorithm in Appendix A..4).
Definition 8. (Sequence.) A sequence of length ls is k-tuple whose elements are called symbols.
Definition (Fragment.) Given a sequence s and a of indices Ing 2(L....k), A fragment is the tuple of subsequences of given by consecutive indices in Ing-
Definition 10. (Template.) Given a sequence 8 with length k and a pair index sets Frg E 2(1....k) and 4p1 = [I....,k] k} Atrg, a template is defined as the tuple whose symbols are identical to those of for all the indices of Tept, and slot symbol * for all indices in Ing where consccutive slot symbols are reduced to one.
Definition 11. (Euvironment.) Given a template = 4x) and a window size w. Let Ienv == {i (L....... 1f} E (fmax(1,--m):-... "imasmm-)).....mttttttttt.t. tmin(k,i+==))) The corresponding subsequence of t given by Ienr is called the environment of (with respect to the window size w).
We used the scikit-learn (Pedregusa et al. 2011 version of k-means clustering, implementing Lloyd's standard algorithm Lloyd 1982 by default.
Note that this algorithm works historical data and but possible approximation of the original data generation
process assumed underlie (clinical) time series. We consider its purely data-driven nature and impartiality towards the underlying compositional process advantage for the current exposition.

---

Published in Transactions on Machine Learning Research (03/2025)
Definition 12. (Insert). Given a template = (f1...., tx) and a fragment f (fi...., fm) where card(f) card (4: E 4414 *), and let i i,,.....m f.m be the slot indices of I, then:
insert: (f.....,tt) x (fi...., fm) ($1,..., .n)
(f1....,tta) x (fi,...,, fm) + flatten ((===
J3 i
lis l o
combines t and f by replacing the slot symbols of t with the symbols of the corresponding subsequences of f to sequence s.
Definition 13. (Compositional data synthesization.) Let there be a fragment f and two non-identical sequences Sn and 8, such that 8a = insert(/a, and 8, = insert((e, [). Furthermore, let there be a sequence s = insert(ts, fo) such that fo # f and environment(fa. w) = environment(te, w). Then Sayn = insert(te. fb) is a compositionally created synthetic symbol sequence that is different to Sa: Sb and 8c-
In order to translate a sequence s = insert(t, f) into a time series, we choose a time series whose symbolization is idantical to the sequence 8. Because symbolization is index-preserving, we can directly map time series sections & and f by simply replacing their symbols with the time series blocks of the same index. To compose synthetic time scries Zayu == inscrt(te, fo) obtain time series T& and De with the corresponding symbolizations 8, and 8e and desymbolize le and So accordingly.
5 Experiments
5.1 Data, Preprocessing, and Models
In experiments, we focus two clinical time series datasets: MIMIC-III (Johnson et al. 2016 and eICU (Pollard et al. 2018). Both databascs contain anonymized information from ICU patients, including physiological measurements (e.g., heart rate, blood pressure), medications (e.g., dobutamine, epinephrine). interventions (e.g., intubation), lab test results (e.g., blood cultures,) and clinical notes. We extracted features that were frequently tracked during a patient's 's stay, namely 131 features for MIMIC-III, and 98 for eICU (a complete list can be found in Appendix A.2). For each patient stay we extracted the first 48 hours, but only took stays into account that had at least one feature measured per hour. Furthermore, we removed patients that did not stay long cnough in the ICU. The remaining clinical variables were 2-score standardized, and the resulting dalasets were partitioned into training, validation and test data (see Table in Appendix A for exact numbers). Notably, our data can be characterized as sparse and irregularly sampled Tipiinenii & Reddy 2022 Horn et al., 2020). This is because some fcatures like heart rate or blood pressure are recorded every minules, other Teatures like those only available through blood sampling, are available only every 24 hours. We store these sparse multivariate time series in a database of n quadruplets S = [(fe, ti, Ui, n:)) 1n where fi E F' is a clinical variable identifier, & E R>0 is time index, Uj R the observed value of f; at t,, and n, the unique stay identifier. Following SStanick et al. (2024) who showed advantage for compressing long input time series into 24 hourly bins that record the most important observations, we encode each quadruplet S into a dense representation æ where every timestep is a vector of feature values representing one hour. We construct this vector by choosing the first observed value during the represented hour for each feature. If no value was observed, we impute zero which corresponds to the mean value due to standardization of the data. Additionally, a mask indicating whether value was imputed is generated and appended to the vector. Based on this representation, we define the sparsity of feature as the relative frequency of imputations. For TSF, that generates output vector ộe E I (where [F] is the number of features used). The predicted output i. is a function of the history Jee of predicted timesteps until time t, the encoded input x, and the model parameters 0: fe = Io( To perform long-term TSF using the autoregressive setup, the out ÿ, from each time step = 1....,T T are concatenated. We employ a standard Transformer architecture (Vaswani ct al.) 2017p as our model of choice, the hyperparameters of which can be found Appendix A.1 The encoder takes as input the first 24 hours, the decoder then generates the next 24 hours. The complete model is trained with masked mean squared error (MSF) (see Appendix A.5
Transformer model with an autoregressive decoder
10

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: Estimated expected risk difference between model trained on synthesized data and model trained on original data, evaluated on original test data. Synthetic data are generated con-compositioally (CutMix) or compositionally (CDS). Symbolization is obtained by clustering in input or neural embedding space. Results averages of models trained from three different random sceds, standard error (SE) in brackets.
Synthetic distribution
Symbolization
MIMIC-III (SE)
eICU (SE)
CutMix
0.753 (0.015)
0.911 (0.012)
CDS
imput
0.070 (0.015)
0.064 (0.012)
CDS
embb
0.110 (0.019)
0.055 (0.015)
Table 2: Estimated ratio of expected risk on synthesized and original test data for model h with best result (MSE") obtained by training on the original training data.
Synthetic distribution
Symbolization
MIMIC-III
eICU
MSE
Ratio
MSE
Ratio
CutMix
10.201
1.375
7.648
1.445
CDS
input
7.398
0.997
5.279
0.998
CDS
embd
7.230
0.974
5.086
0.961
To generate a synthetic dataset, we first need to assign symbols to the training data, as described Section 4.1 by either choosing random centroids the input space (input), or by applying k-means clustering the learned representations (embd), with variable number of centroids (#Syms). In our experiments, the number of symbols varied between 40, 80, an 160, with best results on the validation set obtained for the largest symbol size. Furthermore, each symbol represents time series segment of hours, which we found to be a good compromise between very sparse 1-hour windows where over 90% of the measurements need to be imputed, and coarse grained segments with a better descriptive power of temporal patterns. Last, we apply the compositional data synthesization algorithm described in Section 4.2 to generate composit ional syuthetic data versions, or use the CutMix algorithm Yun et al. 2019) directly on the time series to produce non-compositional synthetic data.
5.2
Experimental Results
5.2.1 Test 1: Utility of Synthesized Data for TSF Training
The first empirically testable criterion described in Section 3.2 infers the compositionality of the original data generating process from the estimated expected risk difference é RJ Eorig [é(y, hr (x))] Eorig [H], where Eorig [e(y, (x)) is estimated by training a TSF model on synthetic data and evaluate on the original test data, and Eorig [H] estimated by training a TSF model on the original time series data and evaluate it on the original test data. The synthetic data can be generated either non-compositional (CutMix) or compositional (CDS) with symbolization obtained by clustering in input or neural embedding space. The estimate of é and its standard error is obtained by training a linear mixed effects model (Demidenko) 2013; Bates et al. 2015 Riezler & Hagmann 2024 the evaluation scores of models obtained from three different random seeds for optimization, and another three random seeds for symbolization. The main result is given in Table [ It shows values of that are close to zero for CDS training, and an order of magnitude larger for training on randomization- based synthetic data, on both datasets of clinical time series.
5.2.2
Test 2: Utility of Synthesized Data as TSF Test Data
Table 2 presents the experimental results for the second empirically testable criterion established Sec- This test basod on a model h* that is obtained by training on original time series data and its evaluation on original and synthetic test data. The goal infer uniform bound for the ratio Eyyn [e (sx, h² (x))))/ [e har, (xr)) of the synthetic and original data distribution based estimates of the
tion 3.2 2
11

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: Evaluation of downstream task of prediction of SOFA score on original test set for models trained on different sizes of synthesized data. Results are reported for best performing models (MSE") with 95% confidence intervals for the estimation of the respective evaluation score on the test in subscripts.
Training data
Dataset size
MIMIC-III
eICU
original
1x
3.96,43.32
2.518
CDS-imput
1x
4.066 4.254
2.536p 22.3892..882
5x
3.560 3.766,
2.397 (2.254,2.599
10x
3.332 3.332 33114833511)
2.347 22.4722.217,22222 22.217,2.470]
20x
3.194 3.020,33..3
2.322 (2.189.2444)
CutMix
20x
/..63/3.......om
2.583 p2.443.2.722
corresponding expected risks Esyn [e (az, h (z)) and Eorig [e (x))- We see that the obtained ratios are close to 1 for evaluations on original data and compositionally synthesized data, but not for test data synthe- sized via the randomization-base CutMix method. This again supports our reasoning of compositionality of the data generation process underlying the original time series data.
5.2.3 Evaluation on Downstream Task of SOFA Score Prediction
We performed an experiment where we evaluated training solcly on synthesizod data on the downstream task of prediction of the sequential organ failure assessment (SOFA) score (Vincent al. 1996). The SOFA
score is defined the sum of subscores for six organ systems, each ranging from 0-4 and depending for their part thresholded fundamental clinical variables observed during 24h window (see Appendix A.6 We added a regression head to our dense encoder, and trained this model on antomatically assigned SOFA scores (ranging from 0 to 24). The evaluation was done by computing MSE of the predicted scores. Table 3 shows the results of this downstream ovaluation. Assessing statistical significance by non-overlapping confidence intervals, we can altribute significant improvements for SOFA score prediction on MIMIC-III for models trained on compositionally synthesized data (CISS-input) that are 5, 10, or 20 times larger than the original training set. Similar, but smaller nominal improvements, are obtained by compositional data augmentation the larger eICU dataset. However, neither dataset, prediction for models trained on sundomized-basee data synthesization (CutMixx significantly improves over models trained on original data, despite the size of the synthetic dataset being 20 times larger.
5.2.4 Further Experimental Evaluation
In Appendix A.7 we present analysis of the distributional properties of compositionally created data. We find that the Hellinger distance of distributions of original and compositionally synthesized data show the desired properties for compositional data (Keysers et al. 2020): Closeness in unigram space, indicating similar distributions of symbols, and larger distance in higher n-gram space, indicated by dissimilar distri- butions of compounds. Furthermore, in Appendix [A.8] we present a qualitative assessment of the symbolic representations of clinical time scrics segments, showing that mcaningful physiological states can be Icarned by symbolic representation learning. Lastly, in Appendix [A.9 we present. quantitative evaluation with respect to the discriminative score, and qualitative evaluation according to PCA visualization, inspired
by works on GAN-based data synthesization (Yoon et al. 2019 Pei et al. 2021
6 Conclusion
We presented an investigation of the question whether non-linguistic time scries - here time scries of clini- cal measurements also exhibit the intriguing property of conpasitionalliy of natural language sequences, namely the characteristics of being generated by a process that combines elementary components following a compositional rule system. so, we could analyze clinical time series as a series of subsoquences that are emitted by an ordered sequence of physiological states, and create synthetic data for notoriously sparse and low-resource data situati in clinical TSF. We showed that method from NLP that is based entirely on
12

---

http///aaaaae//011///////999-000 Published in Transactions on Machine Learning Research (03/2025)
distributional properties of data allows us to synthesize novel combinations of clementary time series subse- quences with most interesting properties: Training and testing of TSF models on compositionally synthesized time series yields a similar utility than training and testing on original time series data, allowing us to con- clude that the data generation process underlying the original time series data can in fact be characterized as compositional. Our experiments are based on applying a pipeline of data-driven symbboli representation learning and compositional data augmentation to clinical time series data, and we find consistent improve- ments over randomizationn-assed data synthesization for TSF on two clinical time series datasets, and for the downstream task of SOFA score prediction. We show that our empirical tests can be motivated in domain adaptation theory, drawing a possible inference about the distributions of the data generation process of source and target data from the performance of training and testing models on these data. This theoretical motivation allows us to make broader claims on the compositionality of time series data beyond clinical time series, opening the doors to further research on compositional data augmentation and domain adaptation in general time scries modcling tasks.
A limitation of our work is the lack of embedding into a specific clinical problem, based on problem- specific clinical measurements, and an evidence- based clinical interpretation. This will be important task for future work.
Acknowledgments
This rescarch partially funded by Germany's Excellence Strategy EXC 2181/1 390900948 (STRUC-
TURES). We would like to thank Daniel Durstewitz and Filip Sadlo for fruitful discussions on early ideas leading to this work. Finally, we would like to thank the anonymous reviewers for pointers to related work.
References
Ekin Akyürek, Afra Feyza Akyürek, and Jacob Andreas. Learning to recombine and resample data for compositional generalization. In International Conference on Learning Representations, 2021. URL https:
Jacob Andreas. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), Online, 2020. URL 10. http:////////222aaammmaaa
676.
Douglas Bates, Martin Mächler, Benjamin M. Bolker, and Steven C. Walker. Fitting linear mixed-effects models using lmo4. Journal of Statistical Software, 67(1):1-48, 2015. URL https:///ol.orr//00..88
jss.v067. 07067.1011
Shai Ben-David and Ruth Urner Domain adaptation can quantity compensate for quality? An- nals of Mathematics and Artificial Intelligence, 70:185 202, 2014. URL /ttt::://doi.or/10..100
919472-013-9371-9
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Percira. Analysis of reprosentations for domain adaplation. In Proceedings of the 20th Annual Conference on Neural Information Processing Systems (NIPS06), Vancouver, Canada, 2006. URL http::///wroedddnnggerrr......op 2006 2D00ff10000000ccet00000444444110
pdf
Shai Bev-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151-175, 2010a. URL
fttps://link.springee
Shai Ben-David, Tyler Lu, Teresa Luu, and David Pal. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 129-136, Chia Laguna Resort, Sardinia, Italy, 2010b. URL hatrps://proceedings. mlr. press/v9/ david10a. html
http://ppoooddnggg.......s/////
13

---

Published in Transactions on Machine Learning Research (03/2025)
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. TEEE Trans. Paltern Anal. Mach. Intell. (5(8))1798-18888) 2013. httpss/////doii httttpp:///oi..og//1...o /ttg/10.109//
TPAMI 2013.50
Chengtai Cao, Fan Zhou, Yurou Dai, Jianping Wang, and Kumpeng Zhang. A survey of base- data augmentation: Taxonomy, methods, applications, and explainability. ACM Comput. Surv., 57(2), 2024.
URL https://dol: /tt../..rrr//0.11///66660000
Noam Chomsky. Syntactic Structures. De Gruyter Mouton, 1957.
Noam Chomsky, On certain formal properties of grammars. Information and Control, 2(2):137-183 167, 1959. (59)992-6666 90362-6
URL ftttp:://6ol..rr/11...
9958 (59)
Eugene Demidenko. Mixed Models: Theory and Applications with R. Wiley, 2013. URL https://doi.org/
10. 10.1002/0471728438
Cristóbal Esteban, Stephanic L. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional GANs. arXiv, (bs/1706.022 2017. URL 02633
John R. Firth. A synopsis of linguistic theory, 1930-55. In Studies in Linguistic Analysis, pp. 1-31. Blackwell, 1957.
Hamid Ghaderi, Brandon Foreman, Amin Nayebi, Sindhu Tipirneni, Chandan K. Reddy, and Vignesh Sub- bian. A self-supervised learning-based approach to clustering multivariate time-series data with missing values (SLAC-Time): An application to TBI phenotyping. Journal of Biomedical Informatics, 143:104401, 2023. URL attpps///oi. org/ 10. 1016/j01.jjj www.jjii........... 2023..042231 104401
Peikun Guo, Huiyuan Yang, and Akane Sano. Empirical study of mix-based data augmentation methods in physiological time series data. The 11th IEEE International Conference on Healthoure Informatics (IEEE ICHI), Houston, TX, USA, 2023. URL https:///oi.onn org 48550/arXi111 2309 09970
Zellig S. Harris. Distributional structure. WORD, 10(2-3):146 162, 1954. URL http::://a....og//11..00000 00437956. 11659520
I. Higgins, Loie Malthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In Pro- ceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France, 2017. URL attp::///ppaaiiiier..ee/r//rrrrs hef/
Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In Proceedings of the 37th International Conference on Machine Learning (ICML), Online, 2020.
URL https://procceeding. .mlr.mmrr... wttpp/gs.....p.p....e.emm wttppp////////////aaao..o
Chen Huang, Peixin Qin, Wenqiang Lei, and Jiancheng Lv. Towards equipping transformer with the ability systematic compositionality. Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
URL 10. 1et19660aaaa.. v38116. 29788
Theo M.V. Janssen. Compositionality; its historic context. In The Ozford handbook of compositionality, pp. wil.l http://www....//////////-1100000
19 46. Oxford University Press, 2012. URL pdf
Alistair E.W. Johnson, Tom Pollard, Lu Shen, Li wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC-III, a freely accessible
critical care database. Scientific Data, 3(1):160035, 2016. URL httpps::/aoi.orggeteee 1038//...11/1/00 ssata.daaaa...aaaa.aaaa 2016.
35
14

---

Published in Transactions on Machine Learning Research (03/2025)
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations (ICLR), virtual, 2020. URL https: //openreview.net///e wwt.:ee/tfffooee teet/forrmmidddSyCCCllrrrr
Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based on semantic inter- pretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 2020. URL 10. http:///ww/2220...mmmmmmaaa.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-ssequene recurrent networks. In Proceedings of the 35th International Conference on Ma- chine Larning (ICML), Stockholm, Sweden, 2018. URL http::/proceedd//n mlr press/
lakel8a. pdf
Brenden M. Lake, Tomer D. Ullman, Joshua B. Temenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017. URL https://doi. org/ 10. 10.10077801400255166000
Douglas Lind and Brian Marcus. An Introduction to Symbolic Dynamics and Coding. Cambridge University Press, 1995.
Stuart P. Lloyd. Least square quantization in PCM. IEFE Transactions on Information Theory, 28(2): 1982. 1056489
129 137, 1982. URL http::///bi..or//1...111
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. In Proceedings of the 36th International Conference on Machine Learning (ICML), Long Beach, California, USA, 2019. URL httpp///proceedings mlr
presss//97/loccteelll19 httpp//osssaaaaaa...././///////a
Qianli Ma, Jiawei Zheng, Sen Li, and Gary W Cottrell. Learning representations for time series clustering. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NewrIPS), 2019. URL httpps:/pproceddsss.ce paper
pdf
Richard Montague. Universal grammar. Theoria, 36:373-399 1970.
Milton Montero, Jeffrey Bowers, Rui Ponte Costa, Casimir Ludwig, and Gaurav Malhotra. Lost in latent space: Examining failures of disentangled models at combinatorial generalisation. In Advances in Neu- ral Information Processing Systems (NeurIPS), 2022. URL https://proocesdings http:/eeeddiiinnononnnioeeeeeeee neurips. coppepp.cccppppppprpr httpp////...2///////000000220000
pdf
Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations (ICLR), 2021. URL httppp//gppaeeeeeee..eee///rrrrp
Barbara H. Partee. Compositionalityy In Varieties of formal semantics, volume 3, pp. 281-311 1984. URL //web://web.. mit wtttteeel///////////cchnnng http:////.....eeeeaaaaaa///teeee 1ee__994.pppp. pdd]
Arkil Patel, Satwik Bhattamishra, Phil Blunsom, and Navin Goyal. Revisiting the compositional general- ization abilities of neural sequence models. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (ACL), Dublin, Treland, 2022. URI. https://aclanthhllle htw.llanntooooy.orogg//// wrt.g/22.r./02222... acl-short.45
P. Patel, E. Keogh, Lin, and S. Lonardi. Mining motifs massive time series databascs. In Proceedings of the 2002 IEEE International Conference on Data Mining (ICDM), Macbashi City, Japan, 2002. URL
http://bb....r//....///////oooo 118399
15

---

Published in Transactions on Machine Learning Research (03/2025)
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. Journal of machine learning research (JMLR), 12:2825 2830, 2011. URL https://wwww.....
org/
pdf
Hengzhi Pei, Kan Ren, Yuqing Yang, Chang Liu, Tao Qin, and Dongsheng Li. Towards Generating Real- World Time Series Data. In 2021 IEFE International Conference on Data Mining (TCDM), Los Alamitos, CA, USA, 2021. URL https:///oi.... htt.:/ooo..eeecmmtttterrrrceeeee httpp//cwwparrooolleoooo/o////// 11tp//0.1199/CCM5111..
Tom J. Pollard, Alistair E. W. Johnson, Jesse D. Raffa, Leo A. Celi, Roger G. Mark, and Omar Badawi. The eICU collaborative research database, freely available multi-center database for critical care research. Scientific Data, 5(180178); 2018. URL httpp:://oi..rrg// 1008/ sdata. 2018.178
Linlu Qiu, Peter Shaw, Panupong Pasupat, Pawel Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. Improving compositional generalization with latent structure and data augmentation. In Proceedings of the 2022 Conference of the North American Chapler of the Association for Compulational Linguisties: Human Language Technologies, Seattle, United States, 2022. URL 10. 21/865/vv/20// maacl-main 323,
Stefan Riezler and Michael Hagmann. Validity, Reliability, and Significance: Empirical Methods for NLP and Data Science. Springer, second edition, 2024. httap:///ooo.ogg//0000000/7-----
Jacob Russin, Jason Jo, Randall O'Reilly and Yoshua Bengio. Compositional generalization by factorizing aligmment and translation. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics: Student Research Workshop, Online, 2020. URL 10.
aww.
Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom, and Chris Dyer. Transformer grammars: Augmenting transformer language models with syntactic inductive biases at scale. Transactions of the Association for Computational Linguistics, 101442--433) 2022. URL 10. 1162/tacc.
a 3_00526
Patrick Schäfer and Ulf Leser. Motiflets: Simple and accurate detection of motifs in time series. Proc. VLDB Endow., 16(4):725-73)) 737, 2022. URL htttps://ooii wtt................... 14778/3574444 357457
Lukas Schott, Julius Von Kügelgen, Frederik Träuble, Peter Vincent Gehler, Chris Russell, Matthias Bethge, Bernhard Schölkopf, Francesco Locatello, and Wieland Brendel. Visual representation learning does not generalize strongly within the same domain. In International Conference on Learning Representalions (ICLR), 2022. URL httpp::/
Michael Staniek, Marius Fracarolli, Michael Hagmann, and Stefan Riezler. Early prediction of causes (not offects) in healtheare by long-term clinical time series forecasting. Proceedings of Machine Learning Re- search: Muchine Learning for Healtheure, 252:1 29, 2024. URL httts://arxiv.cr htrgg/ab//2400/ 03816
Zoltán Gendler Szabó. Compositionality, Stanford Encyclopedia of Philosophy, 2020. URL https://ppatoo.
stanford, wttp:///rrrrieee/ooppttttaaaat/t
Sindhu Tipirneni and Chandan K. Reddy. Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions Knowledge Discovery from Data, 16(6), 2022. doi: 10. 10.1145/3516367. URL httpps:/d// httttss://dii.or/1110 httg//...145/35111 1145/3516666
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advonces in Neural Information Processing Systems (NIPS), Long Beach, CA, 2017. CRL Bat.aa4aaaa-aaperpppe
filesslllppliee tttess//ppr//21177/
J.L. Vincent, R. Moreno, Takala, S. Willatts, A. De Mendonça, H. Bruining, C. Reinhart, P. Suter, and
L. Thijs. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ bysfunction/failuuee Intensive Care Medicine, 22( -700, 1996. URL htttp::///ot wttp///..0000////0007777000
16

---

Published in Transactions on Machine Learning Research (03/2025)
Qingsong Wen, Liang Sun, Fan Yang. Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey. In Proceediiggs of the Thirlieth International Joinl Conference on Artificial Intelligence, IJCAI-21, 2021. URL https://doi org/10. 24963/1cai23/2222 20211
631
Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Compositional gen- eralization from first principles. In Thirty-sevenlh Conference on Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, 2023. URL https://openreviii http::://peerrrieeeeeee/e/r
Susan Williams. Introduction to symbolic dynamics. In Proceedings of the Symposia in Applied Mathemalics, volume 60, 1-12 2004.
Zhenlin Xu, Marc Niethammer, and Colin Raffel. Compositional generalization in unsupervised com- positional representation learning: a study on disentanglement and emergent language. In Pro- ceedingg of the 36th International Conference on Newral Information Processing Systems (NeurIPS). New Orleans, LA, USA, 2022. URL https://papera. www.eeee.rrrrppppppepees http:://ppprr...nriis.cc.///pppp httppppprr.iilleeepprr//////////
Hong Yang and Travis Desell. Robust augmentation for multivariate time series classification. arXiu, abs/2201.11739, 2022. URL http::///bb..ooo///..6555/5///2.
Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial networks
Advances in Neural Information Processing Systems (NewrIPS), 2019. URL htttps://proceeednngg
neurips..cc/pper_fiili files/ http:////pe///ooeeeeoaaaaaaaaaan http://perrfleeeepppeeeeeeee
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of 1CCV, Seoul, Korea (South), 2019. URL 10.1109/000.2211000001
17

---

Published in Transactions on Machine Learning Research (03/2025)
A Appendix
A.1 Hyperparameters
Table 4: Hyperparameter settings for training. Best settings chosen on development data are shown bold
face.
Parameter
MIMIC-III
eICU
Embedding Size
128, 256, 512
256, 512, 1024 256, 512, 1024 256, 512, 1024
Hidden Size Encoder
128, 256, 512 128, 256, 512
Hidden Size DMS Decoder
Hidden Size TMS Decoder
Output Dimensionality
Output Dimensionality 1, 2,
# Encoder Layers
2
# Decoder Layers
2
Learning Rate
0.0005
0.0005
Finetune Learning Rate
0.0001
0.0001
Batch Size
32
32
Attention Heads Encoder
2, 4, 8
Attention Heads Decoder
1, 2, 4
1,2,4
Dropout
0.05, 0.1, 0.2
0.05
Epochs
100
600
Patience
6 Unixtime variation
Random Seed
Unixtime variation
18

---

Published in Transactions on Machine Learning Research (03/2025)
A.2 Data Statistics and Feature Lists
Table 5: Number of patient stays in clinical time series.
Data split
MIMIC-III
eICU
train
12,304
49,730
dev
3,169
12,433
test
3,803
3,008
Table 6: Feature list for MIMIC-III: Besides the following 131 dynamic variables, only age and gender were extracted. The 15 variables marked with an asterisk are directly used for calculating the SOFA score.
ALP
Epinephrine*
LDH
Packed RBC
ALT
Famotidinc
Lactate
Pantoprazole
AST
Fentanyl
Lactated Ringers
Phosphate
Albumin
FiO2*
Levofloxacin
Piggyback
Albumin 25%
Fiber
Lorazepam
Piperacillin
Albumin 5%
Free Water
Lymphocytes
Platelet Count" *
Amiodarone
Fresh Frozen Plasma
Lymphocytes (Absolute)
Potassium
Anion Gap
Furoscmide
MBP
Pro-admission Intake
BUN
GCS eye*
MCH
Pre-admission Output
Base Excess
GCS motor*
MCHC
Propofol
Basophils
GCS verbal*
MCV
RBC
Bicarbonate
GT Flush
Magnesium
RDW
Bilirubin (Direct)
Gastric
Magnesium Sulfate (Bolus)
RR
Bilirubin (Indirect)
Gastric Meds
Magnesium Sulphate
Residual
Bilirubin (Total)*
Glucose (Blood)
Mechanically ventilated
SBP*
CRR
Glucose (Serum)
Metoprolol
SG Urine
Calcium Free
Glucose (Whole Blood)
Midazolam
Sodium
Calcium Gluconate
HR
Milrinone
Solution
Calcium Total
Half Normal Saline
Monocytes
Sterile Water
Celazolin
Het
Morphine Sulfate
Stool
Chest Tube
Heparin
Neosynephrine
TPN
Chloride
Hgb
Neutrophils
Temperature
Colloid
Hydralazine
Nitroglycerine
Total CO2
Creatinine Blood
Hydromorphone
Nitroprusside
Ultrafittrate
Creatinine Urine
INR
Norepinephrine
Urine
D5W
Insulin Humalog
Normal Saline
Vancomycin
DВP*
Insulin NPH
02 Saturation
Vasopressin
Dextrose Other
Insulin Rogular
OR, OR/PACU Crystalloid
WBC
Dobutaming
Insulin largine
PCO2
Weight
Dopamine*
Intubated
PO intake
pH Blood
EBL
Jackson-Prais -Patt.
PO2*
pH Urinc
Emesis
KCI
PT
Foisinophils
KCI (Bolus)
PTT
19

---

Published in Transactions on Machine Learning Research (03/2025)
Table 7: Feature list for eICU: Besides the following 98 dynamic variables, there are 17 static variables covering age, gender, admission information, and ICU type. The 15 variables marked with an asterisk are directly used for calculating the SOFA score. On the right column, there are 35 drug-related variables. Some of them seem redundant due to different hospitals but not be merged because of different not standardized concentrations.
ALP
Lactate
Amiodarone Dobutamine dose Dobutamine ratio*
ALT
Lymphocytes
AST
MBP
Albumin
MCH
Dopamine dose
Anion Gap
MCHC
Dopamine ralio* Epinephrine dose Epinephrinc ratio*
BUN
MCV
Basc Deficit
MPV
Base Excess
Magnesium
Fentanyl
Basophils
Monocytes
Fentanyl 2
Bedside Glucose
Neutrophils
Fentanyl 3
Bicarbonate
02 L/%
Furosemide
Bilirubin (Direct)
02 Saturation
Heparin
Bilirubin (Total)*
PT
Heparin 2 Heparin Heparin vol
Bodyweight (kg)
PTT
CO2 (Total)
PaCO2
Calcium
PaO2*
Insulin 1
Chloride
Phosphate
Insulin 2
Creatinine (Blood)*
Platelets*
Insulin 3
Creatinine (Urine)
Potassium
Midazolam 1
DBP*
Protein (Total)
Midazolam 2
Eoisinophils
RBC
Milrinone 1
EtCO2
RDW
Milrinone 2
F1O2*
RR
Nitroglycerin 1
Fibrinogen
SBP*
Nitroglycerin 2
GCS eye*
Sodium
Nitroprusside
GCS motor*
Stool
Norepinephrine 1
GCS verbal*
Temperature
Norepinephrine 2
Glucose
Troponin
Norepinephrine rat
HR
Urine
Pantoprazole
Het
WBC
Propofol
Hgb
pH
Propofol 2
INR.
Propofol 3
Vasopressin 1
Vasopressin 2
Vasopressin 3
20

---

Published in Transactions on Machine Learning Research (03/2025)
A.3 Proof for Theorem 1
Theorem 1. (Bounded expected risk and density.) Let P and Q be two absolutely continuous probability distributions on X' x for which the covariate shift assumption holds such that fr(2, = fly ffff(() and fe y) fly fx) are the corresponding densitics. Further let H be a hypotheses class of learners, 220 0 be a loss function, and C.C RR. Then,
Væ E X: Cfo(z) & fx(2) & Cfo(z) = vh E H: CEa (e (x,)())) & Ep (2(((x((()) & CEa (x,
Further C and C 2 1.
Proof. To establish sufficiency, let assume that Vo X: Cfo(a) fp(x) & Cfe(2).
Then
((z,)) x CSw v))o()) fl ())(()) C fly fo(x))
=Jo(ay)
=fr(a,))
ffllay)
Because & is non-negative for all h, we can conclude by the monotonicity and linearity of the integral that wh E H: CEe le (x, h(x)) Ep (x,
To prove necessity, first assume that Ah H: CT [e h())) Ep [e (x, h(x))] holds. Let assume for the moment that Vr E X: Cfo(x) > fr(r). Repeating the argument made establish sufficiency, the momentary assumption made above implies that vh E H: CE2 |e (z, h(x)) > Ep [e (z, h(x)). Obviously this conclusion contradicts our first assumption. Therefore we have to conclude that Vo E X: Cfe(x) S fr(x).
Repeating this argument for the second incquality finishes the proof.
To demonstrate that C S 1, we recoguize that C/o(2) S fr(x) = (fSo(a) S S fr(x)) which directly establishes this fact. The same argument can be repeated show that C2 1.
21

---

Published in Transactions on Machine Learning Research (03/2025)
A.4 Compositional Data Synthesization Algorithm
Algorithm Compositional data synthesization (CDS) algorithm.
frg_to_tph tpp (ifaultdictt(isst
tpl_to_frr defaultdicttiistt
env_to_tpl tpt = defaultdict for seq in dataset:
#frg, tpl and env are indez sets
for frg in frgs(seq):
tpl template (seq, frg)
env euvironment (tpl, vindow_size,
# #fetch symbols
frg_s = get_ get_syms (seq, frg, value_ Salue_tpp==ffgg )
tpl_s = get get_syms (seq, tpl, value aalue_type='tp)
env_s = get_syms (seq, (seq, env, value_type='or env!)
#append maps
[frg_a] appen_ (tpl_s)
apl_to_frg[tp___ append (frg_s)
[e_to_tpl[eev____ append(tpl_s) (tpl_s)
frg. list list (fsg(ftgtoo)ppp))
while True:
shuffle(frg_11s) list)
for frg in frg_list:
tpl_c_list
(buffle(tp____list))
for tpl_c in tpl_c_list:
#get all tpl for frg without tpl_c
tpl_a_list = [tpl for tpl in tpl_c_liss list if tpl != tpl_c] shuffle(tpp_a_listt)
for tpl_a in tpl_a_list:
#retrieve templates with same environment as tpl_a a for tpl_b in env_to_tee (environment(tpll. (tplaa (tpl_a, window_sizee #retrieve all fragments for tpl_b
for frg_b in ppl_to_frg[tpl) if frg_b != frg:
ts_tpl_c, ta ts_frg_c = get ta ta_seemeets (tpl_c, frg) ts ts_tpl_b, ts ls_frg_b- = get ts segments (tpl_t _b frg.b)
yield insert (ts_ttt__ppppp ts ts_ [s_frg_b)
22

---

Published in Transactions on Machine Learning Research (03/2025)
A.5 Evaluation Metrics
Given N lime series in our datasel, with a prediction window of T hours for TSF, the masked mean squared error (MSE) over hourly prediction vectors oe is defined as follows:
N 11 7 mmmfl
MSE NT
(8)
where m² E {0, 131f is a mask indicating if the variables in yi were observed or not, and O is a component- wise product. In our experiments, T is set to 24 hours.
For each synthetically generated dataset and for the original data, we train three differently seeded models. In our experiments, report the average MSE of the three training runs (MSE) and the corresponding standard deviation (SD). In addition, we report the best performing model (MSE") and the 95% confidence interval for the estimation of the evaluation score of MSE" on the test set. This calculated from the sample mean a and standard deviation s of MSE" scores on test of size N as
3
KT T 1.96
TO + 1.96 VN
23

---

Published in Transactions on Machine Learning Research (03/2025)
A.6 SOFA Score Definition
The Sepsis-related Organ Failure Assessment (SOFA) is calculated by summing six subscores ranging from 0 to 4. In our setting, we had to recalculate MAP (mean arterial pressure) by SBP and DBP (systolic and diastolic blood pressure), the Horowitz cocfficient PaO2/Fi00 by PaO2 and FiO2, but ignored the kind of mechanical ventilation. If no value for calculation in a SOFA subsystem was available, we took a value of
Table 8: SOFA score Vinneen et al. 1996p. Abbreviations: CNS Central nervous system; GCS Glasgow
Coma Scale; MV mechanically ventilated including CPAP; MAP mean arterial pressure, UO = Urine output.
CNS
Cardiovascular
Respiratory
Coagu- lation
Liver
Renal
Score
GCS
MAP or vasopressors
PaO2/FiO2 (mmHg)
Platelets (x10%/x4)
Bilirubin (mg/dl)
Creatinine (mg/dl) or UO
+0
15
MAP 70 mmHgg
2 400
150
< 1.2
1.2
+1
13-14
MAP <, 70 mmHg
< 400
< 150
1.2-1.9
1.2-1.9
+2
10
dopamine S /g/kg//min OR
< 300
100
2.0-5.9
2.0 3.4
dobutamine (any dose)
+3
6-9
dopamine ug/kg/min OR epinephrine S 0.1 ug/ /g/kg/min OR uorepinephrinee Σ pg/ ///////min min
200 AND MV
6.0-11.9
3.5-4.9 OR
500
ml/day
+4
< 6
dopamine > 15 pg/ lg/km/ii /min OR epinephrine > 0.1 μg/ kg/ min OR norepinephrine 0.1 pg/ /g/min
< 100 AND MV
< 20
> 12.0
5.0 OR
200
ml/day
24

---

Published in Transactions on Machine Learning Research (03/2025)
A.7 Distributional Properties of Compositional Data in Symbolic Space
Table 9 shows the distributional properties of compositionally synthesized data by computing the pairwise Hellinger distance between symbolic representations of original training data (Tr), original test data (Te), and synthesized data (S). Given two discrete probability distributions P (Pr....,PPl Pk) and Q = (q...... 9k);
the Hellinger distance computed by the Euclidean norm of the difference of the square root vectors:
- lIVP- VQl2.
H(P,Q)
(9)
V2
Our experiments show that the distance between original train and original test distributions an order of magnitude smaller for unigrams than for bigrams or trigrams, and the same relations hold for the dis- tributional distance between synthesized data and original test data. These distributional properties similar distributions between train and test data the level of elementary components, here unigrams, and different distributions the level of compounds, here higher order n-grams data for compositional generalization [Keysers et al. 2020 and are recreated by our compositional data synthesization method.
desired for benchmark
Table 9: Hellinger distance between distributions of symbolic representations of compositionally synthesizod data based MIMIC-III time series. Symbolization was performed by random assignment of centroids in input space.
#Syms
Unigram H(S, Te)
Bigram H(S, Te)
Trigram H(S, Te)
H(Tr, Te)
H(S, Tr)
H(Tr, Te)
H(S, Tr)
H(T,, Te)
H(S, Tr)
40
0.0908
0.0895
0.0764
0.1612
0.1436
0.3438
0.4007
0.2482
0.2482
0.0237
0.0692
0.0597
0.1488
0,1886
0.1180
0.5231
0.5401
0.2845
160
0.0348
0.0616
0.0435
0.2712
0.2820
0.1185
0.7271
0.7142
0.3145
25

---

Published in Transactions on Machine Learning Research (03/2025)
A.8 Qualitative Interpretation of Symbolic Time Series Representations
The goal of this qualitative evaluation is to investigate whether the learned clusters of subsequences of clinical time series can be interpreted as meaningful physiological states. We created dataset versions MIMIC-III and cICU that are restricted six featurcs that are mcasured with high frequency (HR, SBP, DBP, MBP, RR and 02 Saturation). Based on these low-sparsity features, we performed k- k-means clustering on 3-hour blocks of time series subsequences, and assigned symbols to the clusters.
Figure 4 presents the results for k-means clustering of neural representations MIMIC-III data, with k set to 10. Cluster SO is characterized by elevated heart rate values and can be interpreted as representing the physiological state of tachycardia. Cluster $4 can he interpreted to represent the physiological state of hypertension due to elevated systolic, diastolic, and mean blood pressure. Similar results are obtained by computing clusters based on random centroids in the input domain of MIMIC-III. Figure | shows clusters representing tachycardia (S7), hypertension (S0), and tachypnea (S8) due to elevated respiratory rate.
1.9
0.78
-0.52
0.06
-0.23
-0.26
S1
0.85
1.1
-0.76
-0.33
-0.69
-0.29
S2
0.81
-0.47
00.51
-0.15
00.99
0.077
S3
-0.54
-0.28
-0.59
-0.8
-0.76
0.057
Symbol
$4
0.34
0.25
1.4
1.7
1.6
0.0033
S5
-0.49
-0.48
0.16
0.31
0.27
0.13
S6
0.53
0.04
0.53
0.27
-0.21
0.65
0.75
-0.1
-0.17
0.54
1.2
0.09
-0.083
S9
-0.36
1.2
-0.34
-0.5
-0.5
-0.24
SBP
MBP
s02
Physiological Variable
Figure 4: Physiological states learned by time scries symbolization basod on clustering neural representa- tions. For example, cluster SO is characterized by elevated heart rate (HR), representing the physiological state of tachycardia. Cluster S4 he interpreted to represent the physiological state of hypertension due to elevated systolic, diastolic, and mean blood pressure (SBP, DBP, MBP).
26

---

Published in Transactions on Machine Learning Research (03/2025)
-0.99
-0.22
1.3
0.82
1.1
0.31
-0.96
0.02
-0.88
-0.41
-0.56
-0.11
-0.33
-0.0033
0.24
-0.077
0.45
0.68
0.59
-0.023
Symbol
S4
0.33
0.49
-0.41
-0.43
-0.47
-0.71
S5
0.52
-0.03
-0.81
-0.53
-0.69
0.4
96
-0.55
-0.92
-0.47
-0.75
-0.66
0.35
S7
1.4
-0.027
0.073
0.23
0.14
0.017
0.68
1.8
-0.31
-0.35
-0.38
0.06
0.12
0.08
-0.55
-0.31
0.15
HR
RR
SBP
DBP
MBP
s02
Physiological Variable
Figure 5: Physiological states learned by time series symbolization based clustering in input space. Tachycardia is represented by cluster S7, showing elevated HR values. Hypertension is represented by cluster SO, showing elevated BP values. Cluster S8 represents the physiological state of tachypnca due to elevated respiratory rate (RR).
27

---

Published in Transactions on Machine Learning Research (03/2025)
A.9 Discriminative Score and PCA visualization
For further quantitative evaluation of our data synthesization algorithm, we lake inspiration from work based on generative adversarial networks (GANs) (Yonn et al. 2019 Pei et al., (022)), and calculate a discriminative score as |0.5 accuracy for the classification accuracy of distinguishing synthotic examples from original examples. Similar to Yoon et al. (2019), we train an RNN and evaluate the trained classifiers on a held-out testset. The discriminative score averaged over runs and can be seen in Table 10 Tower discriminative scores are obtained for CDS variants compared to CutMix, indicating higher similarity of original data to compositionally synthesized data than to data synthesized by CutMix.
Table 10: Discriminative scores CutMix compared to our methods CDS embd and CDS input.
Model
discriminative score
CutMix
0.384=0.004
CDS embd
0.301 0.000
CDS input
0.306=0.008
28

---

Published in Transactions on Machine Learning Research (03/2025)
Following Yoon al. (2019), we furthermore present qualitative evaluation where perform PCA on the dense representations of the original data and each of the generated synthetic data. The visualizations are shown in Figures and 7
We see that CDS covers outlier regions of the original data better than CutMix.
Syrthetir in Faregmound
Sold in Foreground
0old
CDS embd
-10
Cold
-10
-20
-10
20
-20
-10
70
30
sptu
sppca
Figure PCA visualisation for MIMIC-III dataset. PCA is applied the input portion of the dense representation. CutMix covers the main region of the gold data, CDS embd and CDS input however to capture the outlier region better.
29

---

Published in Transactions on Machine Learning Research (03/2025)
Syr thetic Foreground
Sold in Foreground
cold Cuttftx
-10-
10
-5
imput
10
xpca
X:pc3
Figure 7: PCA visualisation masked portions of dense representations for MIMIC-III dataset. Here, CDS embed and CDS input both show the same distinct representation gap on the right side of the scatterplot. This gap is blurred by CutMix.
30


Reviews:
[{"review_id": "iicgWUsnoA", "reviewer": "Reviewer_PVFn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The proposed method is interesting and intuitive.\n- The paper is well written.\n\nWeakness:\n- The experimental results are less convincing.\n- The metrics are not enough.\n- The baselines are not enough.\n- Need more qualitative results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors propose an interesting approach for synthetic time-series data generation via compositionality.\n- The proposed method is identifying symbolic representation sequences from the original time-series data to generate synthetic time-series data (based on the learned symbolic representation distributions).\n- The experimental results show that the proposed method performs better than the simple baseline.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**1. Experiments**\n- What is \"x\" here? How is this \"x\" related to \"S\"? How do you encode input to make \"x\"?\n- Do you have some \"qualitative results\"? Like showing the synthetically generated samples and compared with the original samples?\n\n**2. Baseline**\n- As we know, there are various time-series generation methods (including TimeGAN and the papers which cite this paper).\n- However, the authors only compare with CutMix which is actually not developed for time-series synthetic data generation models.\n- In order to provide the advantages of the proposed method, the authors should utilize more advanced time-series synthetic data generation methods than the current baselines.\n\n**3. Metrics**\n- For synthetic data generation, there are various other metrics including \"discriminator approach\" (train another classifier that tries to distinguish between generated data and original data.\n- Also, we can do some feature analysis as well.\n- Currently, I think the ideas are interesting and promising but the (empirical) supports are not enough.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4GYhQMBWT4", "reviewer": "Reviewer_Xak5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Proposes a novel framework for testing compositionality in clinical time series data\n- Introduces symbolic representation through clustering to enable compositional data synthesis\n- Develops two domain adaptation-based tests to assess compositionality empirically\n- Uses compositional synthetic data to train time series forecasting models, showing comparable performance to models trained on original data\n- Suggests generalisability of the approach to other time series data types\n\n## Weaknesses \n- The framework relies on fixed-length (three-hour) segments, which may overlook variable-duration events that are common in clinical data. This segmentation approach risks losing important physiological nuances that variable-length segments might capture.\n- While symbolic states are generated through clustering the clinical relevance of these states is not validated. Without expert interpretation, it is unclear whether these symbols represent meaningful physiological conditions or are artifacts of the clustering process.\n- The synthetic data and compositional tests are not evaluated in the context of a specific clinical application. This limits the practical relevance of the findings, as the symbolic representations and synthetic sequences are not tested for direct utility in clinical decision-making or prediction. The authors acknowledge this, yet it remains something that would add significant weight to the paper and should be feasible given the current experiments and datasets used.\n- The experiments are fairly basic. There are a number of quick add-on experiments that can be done – the authors could include a more diverse set of tasks, such as classification (e.g., distinguishing between patient states or detecting specific conditions like sepsis), or including additional baseline methods, such as data generated through more traditional augmentation techniques (e.g., random time shifts or scaling of real data). That would provide a stronger context for assessing the uniqueness of the compositional approach\n\n## Minor things\n- Only the first equation numbered, not subsequent ones\n- Figure 1 caption: “hearth and lung” -> “heart and lung”\n- Abstract: “framework work” -> “framework”\n- “Transformer” is inconsistently capitalised, see in particular section 5.1.2 which uses both. Recommend sticking to one or the other throughout\n- dobutamin -> dobutamine\n- In the Szabó reference: Stanford Encylopedia of Philosophy -> Stanford Encyclopedia of Philosophy\n- Figure 3: “with with” -> “with”", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a framework for assessing compositionality in clinical time series by decomposing data into symbolic representations of latent physiological states. The authors propose two tests, based on domain adaptation theory, to evaluate whether clinical time series data can be accurately represented and generated through compositional principles. The effectiveness of this synthetic data is evaluated by comparing model performance on compositional versus random synthetic data. Results indicate that models trained on compositional synthetic data perform comparably to those trained on the original data, suggesting that the synthetic data retains relevant structural patterns. The authors suggest that this approach may be generalisable to other types of time series data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Apart from fixing the above minor things, here are the changes I recommend:\n\n- Critical: The authors should discuss the impact of using fixed-length (three-hour) segments, including how this choice might limit the ability to capture variable-duration clinical events. Highlighting trade-offs in this segmentation approach would provide a clearer understanding of the compositional assumptions.\n- Strengthening: Additional experiments/tasks such as classification (e.g., detecting sepsis or other available label) alongside time series forecasting would provide more comprehensive evidence for the versatility of the synthetic data in clinical applications.\n- Strengthening: Including comparisons with baseline methods like traditional data augmentations (e.g., random time shifts or scaling) would provide stronger evidence for the uniqueness and benefits of the compositional approach.\n- Strengthening: A discussion on the interpretability and potential clinical significance of the symbolic states generated through clustering would improve the paper.\n- Strengthening: Conducting a preliminary analysis to see if certain symbolic clusters correlate with known clinical markers or conditions\n\nI've listed the additional experiments as *strengthening* here, yet want to reiterate that the experiments are marginal and the paper would significantly benefit from more experiments. It is finely balanced between that and the fact the paper is very interesting and well-written.", "broader_impact_concerns": "This is a medical/clinical paper yet is very fundamental. There is nothing that I see that would warrant a specific impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CDq5OFTcc0", "reviewer": "Reviewer_hx28", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) Innovative creation of synthetic data: The approach utilizes compositinality to synthesize clinical time series data to improve time series forecasting. \n\n2) Comarison of input and embedding states performance: The method properly evaluate two different and interesting states methodologies, e.g. clusters from the input space and clusters from the embedding space.\n\n3) Potential for clinical decision-making: The findings indicate that synthetic data can improve model performance, which may ultimately support better clinical decision-making.\n\n\nWeaknesses:\n\nSee requested changes", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate the utility of synthesized clinical time series data for training time series forecasting (TSF) models using two prominent datasets: MIMIC-III and eICU. They focus on the challenges posed by sparse and irregularly sampled data in intensive care units, from which they extract a comprehensive set of clinical features from patient stays. The authors employ a Transformer-based model with an autoregressive decoder for long-term TSF, evaluating its performance based on various synthesized data versions. A key component of their approach is the compositional data synthesis (CDS) technique, which clusters data in either input or embedding space to capture the natural structure and temporal relationships of clinical measurements before generating synthetic data. This clustering ensures that synthesized data better reflects the inherent patterns of the original dataset. Their experimental results provide insights into the effectiveness of synthesized data for improving model performance, with comparisons between models trained on original and synthetic datasets. The study demonstrates that while non-compositional methods (CutMix) may yield higher error rates, CDS retains useful information from the original data. Overall, the findings highlight the potential of synthetic data in enhancing the robustness of TSF models, ultimately aiming to support better clinical decision-making in healthcare settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) The introduction leans heavily toward describing methods, particularly after the opening paragraph. To strengthen the introduction part, the authors could expand on the importance of compositionality for real-world applications, specifically for time series. A more comprehensive background should address broader contextual points, such as why compositionality is critical in practical scenarios, specifically for time series modeling. Similarly outlining relevant global challenges, existing limitations, and the current literature’s attempts to address these limitations. Highlighting gaps and limitations within the literature would also provide a clearer rationale for the approach taken in this study, setting up a strong foundation for introducing how this work specifically aims to improve upon those areas.  A short introduction of experimental setup could work in the last part of the introduction, but not heavily described. \n\n\n2) The related work section would benefit from a more direct comparison with prior methods, particularly regarding the obtention of states/clusters which is a significant aspect of this work. For example [1] is a novel alternative approach for states detection, ideally, one could highlight the limitations and advantages of their proposed approach against such work [1] Schäfer P, Leser U. Motiflets: Simple and Accurate Detection of Motifs in Time Series. Proceedings of the VLDB Endowment. 2022 Dec 1;16(4):725-37.  Similarly, the authors tend to over discuss NLP when certainly time series have a large community which is better tailored to their methodology. \n\n3) I would strongly suggest the authors to do not include heavy mathematical notation in the tables caption. Similarly, I would suggest they keep most of the notation within section 3 or methods. \n\n4) The authors did not provide a clear explanation of the methodology or selection criteria for the units of measurement used for each variable. For example, a single laboratory value in the MIMIC dataset may be recorded in various units, which can lead to significant differences in interpretation (e.g., 3.5 mmol/L is not equivalent to 3500 µmol/L). This lack of clarity on unit standardization could impact the reliability and comparability of the results.\n\n5) The authors explain their methodology for handling irregular sampling rates; however, the use of diverse variables with varying frequencies may introduce limitations, such as patient or hospital visit bias, since some features may be recorded more frequently based on specific clinical conditions. These variations could impact the effectiveness of their state detection approach. I recommend that the authors discuss the implications of their methodology regarding sampling rates. Ideally, a brief evaluation stratifying patients or conditions would help quantify these variances, though I do not request extensive experiments due to time constraints.\n\n6) The wording definitions can be improved in terms of unification, for example call ‘states’ or ‘clusters’ across the whole manuscript in a single way when possible. Similarly, ensure that terms like \"synthetic data,\" \"compositional data synthesis,\" and others are used consistently throughout your thesis to avoid confusion.\n\n7) I would strongly suggest to be more explicit with the title. Compositionality in clinical time series is a very broad title. I would suggest the inclusion of synthetic/generation, or time series forecasting. \n\n8) The authors present two clustering methods, the input space clustering and embedding space clustering but do not adequately compare the strengths and weaknesses of each approach. A more detailed discussion on how these methods impact the compositionality of the generated data would be beneficial. Specifically, how does clustering in the input space compare to embedding space in terms of interpretability and capturing underlying physiological states?\n\n9) The authors did not mention the specifics on the thresholds for the definition of healthy variables. At least, a brief discussion of these would be ideal, for example, ALP is higher for children and adolescents due to bone growth. Albumin lower levels are more common in elderly patients. Hemoglobin normal levels are typically higher in males than in females. Creatinine is generally higher in males than in females due to differences in muscle mass. \n\n10) Section 5.1.1 Data contains the storage approach and notation for sparse and irregularly sample time series. From my perspective, this would fit better into the methods section. \n\n11) The authors did not focus on incorporating classification downstream tasks in their analysis of synthesized clinical time series data. Ideally, the authors should have either integrated classification methods into their study or discussed their rationale for omitting this approach. \n\n12) The authors did not test nor properly discuss their synthesis approach for the enhancement of performance on out-of-distribution (OOD), where one could evaluate a model trained on MIMIC, on the eICU dataset. Ideally, the learned compositional representation should be able to generalize towards distinct dataset which are typically the real-world clinical settings.\n\n13) The authors define time series as clinical variables, technically, this is correct, however, by time series, in the medical comunity we would firstly refer to waveforms such as ECG, PPG, EEG. Therefore, it would be great to distinguish between the used variables and waveforms, at least a discussion part into how these would (if would) be integrated into the authors pipeline.\n\n\nOverall: I would suggest at least discussion of the mentioned topics, and an improved text structure to improve redability (specific text in sections + better handling of math notations). Similarly, additional experiments are a good plus.", "broader_impact_concerns": "Potentially including some statement regarding synthetic data usage. For example, if representation are obtained from MIMIC dataset, there might be some bias towards high density variables or patient metadata. Which might not be aplicable towards other datasets.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have updated the paper sufficiently in response to the reviewers concerns and suggestions. I feel that it is ready as is for publication. I will caution the authors however against some of the broader concerns raised in the reviews about adequate positioning of the contributions of this work. There are some considerations of more thoroughly discussing prior work before submitting the camera ready version of the work. Additionally, I recommend that the authors follow through with their promise of making their code available, as they have promised.
Decision: Accept (poster)