Sortability of Time Series Data
Authors: Christopher Lohse
Jonas Wahl
Venue: Accepted by TMLR ()

Abstract: Evaluating the performance of causal discovery algorithms that aim to find causal relationships between time-dependent processes remains a challenging topic. In this paper, we show that certain characteristics of datasets, such as varsortability (Reisach et al. 2021) and R2-sortability (Reisach et al. 2023), also occur in datasets for autocorrelated stationary time series. We illustrate this empirically using four types of data: simulated data based on SVAR models and Erdős-Rényi graphs, the data used in the 2019 causality-for-climate challenge (Runge et al. 2019), real-world river stream datasets, and real-world data generated by the Causal Chamber of (Gamella et al. 2024). To do this, we adapt var- and R2-sortability to time series data. We also investigate the extent to which the performance of  continuous score-based causal discovery methods goes hand in hand with high sortability. Arguably, our most surprising finding is that the investigated real-world datasets exhibit high varsortability and low R2-sortability indicating that scales may carry a significant amount of causal information.

Published in Transactions on Machine Learning Research (07 (07/202) /025)
Sortability of Time Series Data
Christopher Lohse
lohsec@tod.
School of Computer Science and Statistics University of Dublin Trinity College
IBM Research Europe, Dublin
Jonas Wahl
jonas. wahl wwwhladfii.com
Deutsches Forschangszentrum für künstliche Intelligenz (DFKI)
Reviewed on OpenReview: https: openreview. net/ formm? OGumCpcHHV
Abstract
Evaluating the performance of causal discovery algorithms that aim to find causal relation- ships between time-dependent processes remains challenging topic. In this paper, we show that certain characteristic ics datasets, such as varsortability Reisach et al. (2021) and R²- sortability Reisach et al. (2023), also occur in datasets for autocorrelated stationary time series. We illustrate this empirically using four types of data: simulated data based on SVAR models and Erdös-Rees Rényii graphs, the data used in the 2019 causality-for-climast challenge (Runge et al., 2019). real-world river stream datasets, and real-world data generated by the Causal Chamber of Gamella et al. (2024). To do this, we adapt var- and R²-sortability to lime series data. We also investigale the extent to which the performance of continuous ccsre-based causal discovery methods goes hand in hand with high sortability. Arguably, our most surprising finding is that the investigated real-world datascts exhibit high var- sortability and low RE-sortability indicating that scales may carry significant amount of causal information.
1 Introduction
Inferring causal relationships between variables in a multivariate time series setting is an ongoing topic of research in many fields, including ecomomies/cconomeetiic (Varian 2016), climate science (Runge et al.
2019 Runge 2020), medicine (Yazdani Boerwinkle, 20155 and neuroscience (Bergmann & Hartwigsen 2021). The process of inferring causal structure from data is often referred to as causal discovery or causal structure lcarning. Most of the literature causal discovery is devoted to the following two types of methods:
Constraint-based causal structure learning algorithms, such the PC-algorithm (Spirtes & Glymour 1991 and PCMCT (Rungel 2020) for time series data, use tests of conditional independence to iteratively learn causal graph. Score- based method such as GES (Chickering) 2002) fit directed acyelic graph (DAG) to the data by optimising a score function, but need to search a large discrete space of DAGs which is an NP-hard problem.
More recently, Zheng al. (2018) have proposed embed the discrete DAG-search space into a continuous one through differentiable acyclicity constraint to make the problem amenable to continuous (gradient based) optimisation. The method introduced in Zheng et al. (2018) NOTEARS, showed impressive per- formance on data simulated with lincar additive noise models (LANMs). However, as shown by Reisach et al. (2021) the strong performance of NOTEARS and similar moth on LANM data vanished after the data was normalized. Reisach et al. (2021) noticed that, before normalization, additive noise model data highly varsortable, meaning that, on average, the causal order of the system can be recovered well by sorting the variables by the amplitude of their estimated variances. Since high varsortability and good NOTEARS performance were highly correlated, they therefore conjectured that NOTEARS implicitly made

---

Published in Transactions on Machine Learning Research (07/2025)
use of variance sorting in its optimisation. The issue has recently been revisited by Ng et al. (2024) who pointed out that (a) NOTFARS does not necessarily perform well in the presence of high varsortability and that (b) normalisation of data generated by LANMs moves the data far away from the assumption of equal noise variances that underlies the NOTEARS methodology. Thus, according to Ng et al. (2024) NOTEARS should not have been expected to perform well on normalized LANM data in the first place as one of its fundamental assumptions is not satisfied.
Rather than weighing in on this discussion, in this contribution explore how much var- and R -Froorbbbylyty (the coefficient of determination which acts proxy for the fraction of the variance a variable that is explained by its causal parents introduced in Reisach et al. (2023)) can be seen in data commonly used in method validation of time series causal discovery methods. We evaluate the degree of var-/ /trt-sotaabbiltty in different datasets as well as the performance of different algorithms for time series causal discovery in the presence/absene of var-/R sorttabbyyy and before/ /after normalisation. One of the main questions under- lying the discussions on sortability is this: how much -orttabbbyyy do we expect to see in real-world data? The answer to this question is likely highly context-dependent, and the question can be notoriously difficult to settle even for single dataset, since
needs to know the causal ground truth to compute sortability. We investigate sortability scores in two of the rare cases where cansal ground-truth is available, the river flow dataset used in Tran et al. (2024) and the Causal Chamber datasets recently published by Gamella et al. (2024). In both cases, we find sortability values quite far from 0.5 (the case where there is no information in the sortability criterion), meaning that there is information in the variance or R2-values of the data. In the river stream datasets, marginal variances tend to decrease along the causal order (varsorta- bility close zero) while in the Causal Chamber case, marginal variances increase along the causal order (varsortability close to onc). For the R²-score find the exact opposite: values close to one for some rivers in the river dataset, and values close to zero for some of the causal chamber datasets.
These observations illustrate that discarding scales in causal discovery as arbitrary may be premature, as scales may encode significant causal information. We hypothesise plausible physical explanations for the observed varsortability in the investigated datasets which also call into question the validity of an equal noise variance assumption in these cases, For the river data we know that the the width of the rivers decrease from the source to the mouth which has potentially an inffuence on the variance of extreme flows.
In more detail, our main contributions are:
1. We extend var- and R Rossorbability and the simple benchmarking algorithm of 2023) to the time series setting.
Reisach et al. 2021
2. We show empirically that simulated data typically used to evaluate causal discovery algorithms for lime series data is varsortable, meaning the amplitude of the marginal variance increases the lower the variable in the causal ordering of the ground truth summary graph. We also show that, in this case, varsortability is largely driven by contemporanous dependencies.
3. We demonstrate that there is positive correlation between the performance of continuous score- based causal discovery algorithms for time series and the varsortability of data generated by struc- tural antoregressive processos.
4. We investigate sortability of the data used in the 2019 causal discovery challenge by Runge et al. (2019) and show that some data sets are highly varsortable and simple benchmark perform well on these.
5. We calculate var- and R² sootoribbyyy of the river flow dataset used in Tran et al. [202]), and of the recently published datasets generated by the Causal Chamber Gamella et al. 2024 We find that in these rial-live datasets, the scale plays an important role for potential causal discovery algorithms.
Chamber(Ganella
https://causeme: www.seeeeeeuveeeeeeee www.twwwweeeeeeppe211///////////
2

---

Published in Transactions on Machine Learning Research (07/ 2025)
2 Preliminaries
2.1 Causal discovery for time series
A stationary time series graph (ts-graph) is a directed graph 9 = (V x Z, V= (1....,,] d), whose edges (i,t-k) + (J.t) are assumed invariant under translation of the time component. In addition, it is typically required that there a finite maximal lag Tmar = max. ( h). ()) E Dj < oo and that the
contemporaneous component of is acyclic, where & N. Any stationary ts-graph induces a directed, potentially cyclic summary graph Geum over V that contains directed edge (i,5) if (i,t- - k) (j.t) E D. The adjacency matrices of 9sum will be denoted by W, W respectively.
The aim of most causal discovery methods for time series is to recover either G or Gaum from observa- tional or interventional data. Often the data is assumed to be generated by a discrete multivariate process (X,), X, (X, ...,, x(()) compatible with G. The process (Xe)tez is typically modelled as structural vector autoregressive (SVAR) process process(Hyvárine (Hyvärinen et al. 2010 in which case, compatibility with 9 means that (Xt)tzz follows the evolution rule
Tmaz
X Σ
x X v/
(1)
sEpagaum
(i)
k=0
for j E {1, d), where n are Gaussian white noise processes, pag.um (i) denotes the parents of node j in the summary graph, and di only allowed to be non-zero if (i,t-k) + (j.t) is an edge in G. SVAR-processes can be considered the time series analogue of additive lincar noise models for which sortability
discussed by RReiachh et al. 2021 2023 When generating data from such a model, coefficients are lypically randomly drawn, sometimes with a pre-specified proportion of contemporaneous links, and the process is being run until it has converged to a stationary distribution (or is discarded if the distribution is non-stationary).
2.2 NOTEARS and Derivatives
Zheng et al. 2018) propose the continuous score-based causal discovery method NOTEARS, which em- beds the discrete search space of DAGs into a continuous one by using the differentiable function h(W)- = tr e WWo d. This function is 0 if and only if W is the adjacency matrix an acyclic graph and hence mcasures the DAGness of W (Zheng et al. 2018). By combining this function with a score evaluating how well the estimated weight matrix W fits the data, Zheng et al. (2018) formulate the constrained optimisation problem to find
X XWI
n
s.t. W is acyelic, which is modelled by h. 11.113 is the Frobenius norm.
The DYNOTEARS algorithm Pamfil et al. 2020 modifies the NOTEARS algorithm to work with time-lagged and auto-correlated dependencies by redefining the optimisation problem to minwi ((W., Wi) s.t. W. is acyclic, where W1 E IRdxdxTmaz is the lagged adjacency matrix and W. is the contemporaneous adjacency of the underlying time series process. As the underlying time series graph to estimate is acyclic if and only if W. is acyelic, it suffices to enforce the acyelicity constraint only on W. (Pamfil et al. 2020). To ensure sparsity of W, Parnfl et al. (2020) also add an e penalty term, leading to the constraint optimisation problem
min 1 W 2n
1 - XW. X,W. W/ll + Aa (
(3)
s.t. We is acyclic. Here a and X are two regularisation parameters and W, is the lagged adjacency matrix.
The continuous optimisation problems of NOTEARS and DYNOTEARS can be solved efficiently by rewrit- ing the problem using the augmented Lagrangian method and using a numorical solver such as L-BFGS (Zheng et al. 2018 Pamfil et al. 2020). After applying the numerical optimisation algorithm in both of the algorithms, threshold t is applied to remove weights close to zero (Zheng et al. 2018 Pamfil al. 2020
3

---

b Published in Transactions on Machine Learning Research (07 2025)
2.3 Sortability Criteria
In the following, we briefly reiterate the two sorting criteria. varsortability and R²=sortability introduced by Reisach et al. (2021) and [Reisahh et al. (2023) respectively. In essence, both of these approaches calculate a score s of sortability in a comparable manner; s is the measurement of the degree agreement between the true causal order and the increasing order of a sortability criterion cri.
For any causal model containing the variables {_x(1)... ,x.,,, x( with (non-degenerate) adjacency matrix W, the sortability score is the fraction of directed paths that start from node with strictly lower sortability criterion than the node they end in. Thus the sortability for one selected criterion eri can be calculated as
E E. sew increasingleri((").oom cri(())))
[0, 1].
(4)
2 EE Eissew
where
1 a a<
increasing(a,b b)
1/2 0
(5)
The matrix W in Eqqution is the power k since the (i,5) entry of the k-th power of an adjacency matrix of DAG exactly counts the number of directed paths from i to j. In the case of varsortability Reisach et al. (2021). the criterion cri(X') Var(X*) is the marginal variance, whereas the case of R²² --sortabilliyy
the coefiicien
of determination cri(X R²(X') which acts as a proxy for the fraction of the variance of X; that is explained by its cansal parents, (see RReisach et al. (2023) for details). Varsortalility is defined by using the marginal variance as cri. H2 -orttabbiyly is defined by using the obtained RZ- coefffiicss as the sorting criterion cri (Reisach et al. 2021 2023).
Reisach et al. (2021 showed that varsortability is usually high in data generated by LANMs (see the analytical and empirical proof in their paper). Based on their sortability criteria, Reisach et al. (2021) and Reisach et al. 22223 introduced the baseline methods varsorturegress and =f*-sotaregrress respectively. These algorithms sort the
system variables based on their variances or R which are estimated by fitting regression model; these simple benchmark methods are then shown to have similar performance to some state of the art causal discovery algorithms
Reisach et al. 2021 2023 on LANM data.
3 Modified Sortability Criteria for Summary Graphs
In time series cansal discovery setting, the summary graph Goum describing the relationship between the different time-evolving processes may contain cycles. Since the marginal variances Var(X:) do not depend on the time index due to the assumed stationarity of the processes, compute varsortability in this situation, in principle, one could still use Equation as is.
However, any pair of processes X and X' that belong to the same strongly connected component of the summary graph (i.e. that are connected by a cycle) would always contribute a to the mumerator and a 2 to the denominator of [Equation 4
Therefore the presence of cycles would dilute the sortability signal and would naturally push it closer to 1/2. Tn other words, R²- and varsorting of eyclicly connected processes meaningless, and we only interested in whether nodes that are not cyclicly connected can be sorted. Our sortability criterion thus becomes
8 = EOSSIEAP(GOm bncreasing(er((x)..rr E [0, 1].
(6)
E E(sdEEAP(aam)
where AP(g') = ((t.1) E V X ii j. j # 1) is the set of admissible node pairs (the long double arrow indicates the existence of a directed path).
For the example in Figure the sortability score is 1+1+1+0 0.75. The contribution of the pairs (B, C) and (C,B) is ignored since the two nodes belong to the black cycle. Cycle-free directed paths that connect admissible node pairs are depicted in colour. If we would calculate it including cyclicly connected pairs, it would result in 0.67.
4

---

Published in Transactions on Machine Learning Research (07 (07/2025)
B Cri(B) =2
C
Cri(C) =
A Cri(A) =1
D Cri(D) = 2.5
Figure 1: Example of the calculation of with cycles.
High varsortability and the performance of DYNOTEARS Based on heuristics and empirical find- ings, Reisach et al. (202I) argue that high variability causes gradient- gradient-based optimisation algorithms such as NOTEARS
(Zhengg Zheng et al. 2018) to favour graphs whose edges point the direction of increasing marginal variances during their first optimisation step. Since DYNOTEARS is lime-series adaptation of NOTEARS, that essentially coincides with NOTEARS on contemporaneous components of ls-graph, Reisach et be applicable to (the contemporaneous part of) DYNOTEARS as well.
2021 arguments might
However, in the recent work Ng et al. (2024) provide examples which continuous optimization- methods can not perform well in the presence of high varsortability. [g et al. (2024) also give alternative explanation for why continuous structure learning performs significantly worse after standardisation of LANM data. Continuous structure learning assumes equal noise variances for all variables in the system, which is violated in the standardised model leading to poor performance.
4 Sortnregress for Time Series Graphs
In order have simple algorithms that exploit high R²- and varsorlability, present time series adapled sort- nregress algorithu based on sor sortnregress from Reisach et al. [202] 2023). To estimate contemporaneous dependencies, use the standard sortnregress algorithm, which consists two steps:
1. Sort nodes by increasing marginal variance Rs-oce
2. Each node is regressed its predecessor, determined by order, using penalised regression technique. As described in Reisach et al. (2021), LASSO regression is used, using the Bayesian Information Criterion (BIC) for hyperparameter selection.
This gives estimated contemporaneous adjacency matrix W.. A random sortnregress algorithm is also used, where we determine the order of the variables randomly using i..dd Bernoulli trials. To estimate lagged dependencies between variables, predecessors Pi,t, where E [1, Tmaz, indicates the time lag. After this step we have an estimated lagged adjacency matrix W..
the same first step and change the second step: We now regress each node each of its
5 Numerical Experiments & Results
In the following section, we first give an overview of the evaluation metrics which are used for all the considered datasets. After that we outline the setup and the results for each dataset. We conduct our experiments Python, using the TIGRAMITE library for simulating data with SVAR models. When assessing the performance of different algorithms across a range of sortability values, the hyperparameters of the DYNOTEARS algorithm are set X1 A2 0.05. The weight threshold is set to 0.1. As a constraint-based comparison algorithm, PCMCT* (Runge 2020) is run with a = 0.01 and the ParCorr conditional independence test.. We further use the varsortnregress, Rd -orttrreeeessss and random regress algorithms described in Section
We assess the Fl-score using the formula:
TP FI TP F.5 FN)
httpp:://gtthub.cmmm///// http:///hthbbcc..kkkkooorgtttttt
5

---

Published in Transactions on Machine Learning Research (07/2025)
to gauge the performance of the selected algorithums concerning the comparison between the estimated binary time series adjacency matrices W and the ground truth W.
Here, TP represents the number of true positives, FP represents the number of false positives, and FN represents the number of false negatives for edge detection.
We refer this metric the overall Fl-score. Additionally, we calculate the Fl-scores comparing the estimated contemporaneous adjacency matrix W. with the true contemporaneous adjacency matrix We, and the Fi-scores comparing the estimated lagged adjacency matrix W. with Wt These metrics are denoled Fl-contemp and Fl-lagged, respectively. In cases where only information about the summary graph is available, we calculate the F1-score between W sum and the estimated summary adjacency matrix W Ssm.-
5.1 NEURIPS Competition Data
httpii/--fo---imattttoomppttottt (Runge et al. 2019) data. This dataset: is relevant not only because was used in the competition, but also because follows same structure as the CauseMe platform (Munzz-arr et al.] 2020). which is widely used to evaluate causal discovery algorithms (Bussmann et al. 2021 Runge (2020). The dataset includes simulated and partially simulated datasets of varying complexity, including high-dimeensional datasets and non-linear dependencies, with 100, 150, 600 1000 realisations for each dataset specification. We excluded the datasets with missing values.
Sctup We also assess var- and R'-sortability on the 2019
We set the maximal time-lag in methods Toue following the description of the respective dataset (ranging from to 5).
we observe a varsortability ahove 0.5 for all data except the logistic model, which has a varsortability of 0, meaning that each causal child has a lower marginal variance than its parent. In particular, the realistic climate and weather models have high varsortability, with a mean of over 0.86 for all of them. For R² -F-sortability, we observe a value around 0.5 most of the realistic models, with exception of FinaICLIM models with values of 0.25 and 0.16 for the 5 and 40 variable datasets respectively. The linear and logistic models have scores between and 0.6. For the data sets with high varsortability (var varsortnnegress out purpfoforms or is en par with PCMCI*.
Results As illustrated in Table 1
Table 1: Mean Sortability criteria and Fi-scores for different benchmark algorithms over different realisations on the NeurIps competition data. We set Tmaz = 3.
Dataset
var
R²
PCMCT+
varsortaregress
R sortnregress
random
Testlinear-
T-150
0.59 0.17
0.61 0.19
Testlinear- VAR N-100 T-150
0.65
0.12
Destnonlinear-AARR N-20 T-600
0.11
0.11
0.25 0.07
0.10 = 0.07
0.10 = 0.06
0.15 # 0.06
Finallinea--AAt V- N-10-TT-100
0.12
0.12
0.15
0.17 = 0.09
0.16 0.09
0.20 + 0.09
Finallinear- Veallieer-VVRR.... BOIRR_NNN00___ SS-100_T-1500
0.16
0.63
FinaICLIMI N-5 T-100
0.11
0.29
0.40
0.13
0.51 = 0.23
0.23 0.18
0.31 4 0.19
FinaCCLIM N-40 '--00
0.9
0.19
FinaICLIMnoise N-5 T-100
0.13
0.3
0
0.43 0.22
0.21 + 0.18
0.28 + 0.21
FinaICLIMnoise N-40 T-100
0.99
0.13
Finallggistic-lagguoisss N-5 T-150 medium
0.312
0.58
0.22
0.02 0.10
0.04
0.04 0.13
FinaWWEATHuoise N-5 T-1000
0.27
0.5
0.31
0.25 0.20
0.20 1 0.20
0.26 1 0.18
FinaWWAATHnois N-10 T-1000
0.11
0.23
0.21 0.12
FinalWEATH N-10 T-1100
0.16
0.31 0.22
0.32 0.14
0.22 0.12
0.27 - 0.12
FinaWEATHH 1-0_-N-55TTOT00 1-1000
0.2
0.47 0.31
0.36
0.34 0.18
0.22 0.17
0.31 1 0.17
5.2 Data Generation with Reris-Rényi Graphs and SVAR Models
Setup In order to investigate var- and Rt-sortability for datasets used to evaluate continuous score- based causal discovery methods, replicate of the two data generation methods used by Pamfil al. (2020).
Following Pamnil et al. (020) Zheng el al. 2018 when generating random lime series graphs, we use Erdös Rényi Graphs (ER Graphs) (Newman 2018) to draw the contemioranneou edges with i.dd Bernoulli trials. Sampling only lower triangle entries of the contemporaneous adjacency matrix and then permuting the node order ensures that the
contemporaneous adjacency matrix W. is acyelic. Pamfil al. (020 sample the graph ensure a pre-specified

---

Published in Transactions on Machine Learning Research (07/2025)
mean degree de for the contemporaaneous dimension and di for the lagged dependencies for total number of variables d.
In order to ensure that the expected mean degree is d. the probability of each Bernoulli trial is set to d./(d- where d. The edge coefficients are sampled uniformly at random from [-2, [0.5, 2.0] (Pamfil et al. 2020). Again following sampled depending on & from [-0.5a, 0.3aa]][0.330 0.5a]. The weight decay > reduces the influence (weights) of variables further back in time. We randomly sample Erdös-Rénp graphs with degree de for the contemporaneous dimension, and for each lag we set di = 1; 8 is set to 1.1.
Pamfil et al. 2020 the edge weights for lagged variables
where a = 1/6"
We examine sortability values for different numbers of nodes d E {10, 20,50, 100). For each number of nodes we randomly generate 500 different graphs and generate 500 samples per graph. The samples are taken after a burn-in period to ensure stationarity. We then calculate the overall varsortability, the varsortability of only the contemporanous dependencies and the varsortability of all all the lagged dependencies.
We also want to investigate whether vansortability driven by contemporaneous or lagged dependencies. This is why compule var- and R² sortoiiabyyy over a grid of de, di E [0, 1, 2,2,,,4,,,88 6,8. We do this for d = 10 and d = 20 nodes.
We further investigate the influence of the two sortability criteria on the performance of score and constraint-based algorithms. In order to do s0, we generate data for d 10 variables, which has varying sortability values. We then randomly draw 30 samples per sortability interval, which we se to 00.0.2). [0.2, 0.4], [..1 [0.6, 0.8). [0.8, 1]. We report the performance the following algorithms for varying var- and sortabibiil DYNOTEARS, DYNOTEARS standardised (run after standardising the data), PCMCI*, varsortnregress/ s*ssortneeesess and randomregress. This means that for DYNOTEARS, the data has a varsortability as defined by the respective bin before standardising (after standardisation the varsortability is always 0.5).
Results We observe that the overall mean varsortability for the ER-SVAR data used by Pamfil et al. (2020) ranges from 0.58 for ten nodes 0.54 for 100 nodes. We do not see trend in varsortability for different numbers of nodes. The varsortability of the contemporaneous dimension is around 0.7 for all numbers of nodes. The lagged varsortability is around 0.54 for all numbers of nodes except 10 nodes where 0.56. The R² sortability is around 0.5 for all numbers of nodes. Varsortability of the contemporancous component of G is always above 0.7 and higher than the overall varsortabillty. Consequently, lagged varsortabilityy always lower than contemporanous and overall varsortability. The detailed results can be found in Appendix A
Sprikes
DWWortANS ahe standrriigg
Plgres
03
02
B 1.
026144
OR
Vanorabltt
(a) Performance of different algorithms for varying var- sortability.
(b) Performance of different algorithms for varying R²- sortability.
Figure 2: Comparison of algorithm performance under different sortability conditions: (a) Varying varsorta-
bility, (b) Varying -t--ttrabillity.
Figure 2a shows that the higher the varsortability, the higher the Fi-score of DYNOTEARS. The varsortnregress benchmark model also improves with higher vassotaability and seemss to outperform the DYNOTEARS algorithm for varsortability values higher than 0.6. The constraint-based PCMI* algorithm does not seem to be affected by varsortability as the randomregress algorithm. The DYNOTEARS algorithms perform better on standardised data for low varsortability values before standardisation.

---

Published in Transactions on Machine Learning Research (07/2025)
For high if'-ootabilityy values, we observe different behaviour: Both DYNOTEARS and PCMCI* seem unafeccted by varying values. Again the -frsoorareegress algorithm seems to oulperform DYNOTEARS for 1t2- -f--sorabiliity values of 0.6 higher (see Figure 2b).
We also investigate whether high vansortability leads to a higher Fl-score of DYNOTEARS due to better identifica- tion of contemporaneous edges or lagged dependencies. As we can Figure 3 the effect of increasing F1-scores with increasing varsortabilityy is even higher for contemporaneons dependencies. The Fi-score for lagged dependen- cies seems to be unaffected. Moreover, we observe our experiments that the contemporaneous F1-score has a Pearson correlation of more than 0.6 with the varsortability score. Furthermore, higher weighting thresholds of the DYNOTEARS algorithu seem to increase the influence varsortability on the performance of DYNOTEARS as measured by the Fl-score.
Dynesure Conteme
Vessmbiity
Figure 3: Fl-score of Contemporaneous and lagged Dependencies DYNOTEARS for different varsortability values.
We also want determine if different degrees of contemporaneous and lagged dependencies influence the overall varsortability of the generated data. In general we observed that a higher value for de increases the varsortability and higher values of seem to decrease it. If there are more nodes the varsortability seems to be higher. there are contemporaneous edges we observe varsortability values of around 0.5. For -i*ssortbbllity, the mean score is not affected by the degree or number nodes. The detailed results can be found in Appendix B
5.3 Extremal River Flow Problem
Setup Next, we investigate the two sortability criteria on four real-world datasets modelling extremal river flow previously used in (Taan et al. 2024). The aim of the problem is to recover the direction and connections of river network with only extreme flow measurements at certain stations, without knowing the location of these stations. This is time-dependent process, as an extreme measurement recorded at one station at time t may lead to extreme measurement at another station at time (Tran et al.) (2024). Having the ground truth river network allows to report R and varsortability on this real-world dataset and investigate whether high values can also occur real time series data. We also include the Fl-scores of selected benchmark algorithuss for a Tiae value of 3 as this gave the best results. Following Tran al. (2024), we treat the downstream river direction the causal ground truth Gaom; which perhaps may be controversial due to the fact that downstream extreme events might have damming effects further upstream. Nevertheless, even if one is not comfortable calling the flow direction causal, the results below show that varsorting is able to cover the 'flow graph'.
As we only know the ground truth/flor graph on the summary level, i.e. we know Goum, we calculate the Fl-score between the ground truth for W.um and the estimated adjacency matrix W Wamm for the following algorithms: varsort- nregress, R sossnrrnss,sss randomregress, PCMCI* and reversed varsortnregress. We use reversed varsortnregress as observed very low varsortability values and wanted to investigate if an algorithm that orders by decreasing variance could exploit this fact.
varsortability for the entire river network is below 0.5, with varsortability value for the Danube close to 0. This indicates that as the river network follows a tree structure, the variance of the extremal river velocities for nodes decreases with increasing distance from the river source. In other words, the real-life dynamics of the river flow systems entail sortable marginal variances. It can be observed that for high n²2 sorsootabillty, the R² -r-ssnnrrerees algorithm performs as well as PCMCT+. Conversely, for low varsortability values, the reversed
Results As illustrated in Table 2
8

---

Published in Transactions on Machine Learning Research (07/2025)
varsortability algorithm, while being superior random, is unable to match the performance of PCMCI*. Notably, the varsorturegress ulgorithm oulperforms the reverse varsor ansorlaregress algorithm on the upper Colorado dataset even though the varsortability is below 0.5. We also tried to DYNOTEARS this dataset. However, the algorithm
Table 2: Varsortability(var) and Ro-sortability (Rº)and Fl-scores of different algorithm reported the extremal river flow datascts. We set Tmax 3 as this leads to the best results.
Dataset
R2
PCMCI+
varsortnregr.
varsortnregr. rev.
-o-sortnregr
randomregr.
danube
0.07
0.82
0.56
0.08
0.20
0.18
0.11
lower-colorado
0.34
0.61
0.36
0.08
0.08
0.08
0.10
middle-colorado
0.29
0.94
0,26
0.09
0.23
0.26
0.22
upper-colorado
0.43
0.95
0.41
0.26
0.11
0.30
0.22
did not converge after a couple hours run time as it did not manage to satisfy the acyclity constraint, at least for our choices of hyperparameters.
5.4 Causal Chamber Data
Setup We now investigate the values of the two sortability criteria for data generated by the recently introduced Causal Chamber (Gamella et al. ) which provides toolbox consisting of real physical systems systems that can be used to evaluate causal discovery or other Al algorithms on real data.
We use each dataset contained in their PYTHON library and calculate the var- and R -o-sorrability for each dataset and each experiment in the datase 3 one value experiment performed the dataset. We again benchmarks varsortnregress, sor-sortnsegress, randomregress, PCMCT* and evaluate the F1-score between W acm and W smmn-
We then calculate the mean and standard deviation of the different dalasets, where
Table 3: Mean and standard deviation of var- and H² sortabbbllyty as well as Fl-score of benchmark algorithms obtained on each datasct with different experiments. The standard deviation given after We set Tmax 2.
Dataset
π22
PCMCI
varsortnregress
-orttnunes
random
It camera test___ vl
0.94 1 0.01
0.25 1 0.24
0.05
0.30
0.17
0.14
It camera validate_ camera_validatee vl
0.99 + 0.02
0.01 = 0.03
0.08
0.37
0.05
0.25
It camera walks vl
0.95 + 0.0
0.23 = 0.07
0.31
0.34
0.13
0.22
It color cillo__rggeesioon... vl
0.94 + 0.02
0.15 + 0.06
0.19
0.25
0.18
0.19
It interventions standard staeendadnnssssaaddddaa vl
0.94 + 0.03
0.46 + 0.04
0.26
0.18
0.12
0.08
It malus_av
0.98 + 0.02
0.02 + 0.01
0.20
0.24
0.25
0.19
It test vl
0.96 + 0.03
0.01 = 0.02
It validate_ vl walks
0.99 + 0.02
0.02 = 0.02
0.34
0.30
0.29
0.13
0.9 0.08
0.24 = 0.04
0.16
0.38
0.22
0.33
wt bernoulli vl
0.97 + 0.06
0.06 = 0.05
changepoints
0.94 + 0.01
0.14 = 0.02
0.12
0.14
0.09
0.06
wt intake impulse__y vl
1.00 1 0.00
0.32 1 0.08
pe laepppllllllllllllllpppeeel validate_ vl
0.78 1 0.00
0.14 1 0.0
wt pressure_ control vl
0.92 1 0.0
0.36 1 0.0
0.21
0.29
0.26
0.25
test
0.94 1 0.08
0.14 = 0.13
wt validate_vy _v
0.97 1 0.05
0.03 = 0.04
wt walks vl
0.92 + 0.03
0.28 + 0.06
Results We observe very high varsortability values for all datasets; almost all datasets have values over 0.9 with the wt wt__e_____a____ee validate v1 dataset beeing the only exception at 0.78 shown in Table 3 All R²-sortability values are below 0.25. Most of them are between 0.2 and 0.32. The It camera validate vl, It malus vl, test bernoulli and wt validate have values very close 0. The values for each individual experiment con- tained in one dataset can be found in Appendix C. As shown in Table Table 3 varsortnregress outperforms other
test _v,
9

---

Published in Transactions on Machine Learning Research (07/2025)
causal discovery algorithms 9 out of 10 of the selected Causal Chamber datasets. We ouly report Fl-scores for 10 of the data sets since for For wt_pe_ wt_pe__validate validate_ .1 there are not enough samples to select the hyperparanutter for sortnregress and the other 6 data sets have over 50, 000 samples resulting in very long execution times.
6 Discussion and Conclusion
In general, both varsortability and sosoorbbily are present in both simulated and real datasets for benchmarking causal discovery algorithms. In line with Reisach et al. (2021), we observe that DYNOTEARS, which is a NOTEARS based algorithm, seems to perform better when varsortability is higher, which is in line with our hypothesis in section 3
Furthermore, we observe that the data used in the NEURIPS competition (Runge et al. 2019) are highly varsortable, especially the realistic datasets, by our defined time series varsortability metric. This is also reflected by the good performance of sortnregress, which should be too simple to perform as good as the more sophisticated PCMCI algorithm.
The low varsortability in the simulated var models could potentially be due the fact that these datasets do not have contemporaneous dependencies, and as we showed earlier, contemporaneous dependencies seem to be the driver for high varsortability simulated time series data. This hardly surprising given that a team exploiting this effect won the Coupetition(Weicwwaa performance of simple benchmark algorithms. However, R'-sortability values observed on the NEURIPS and ER Datasets seem to be too low to be exploited by the sorting algorithm.
et al, 2020). As for sf-sortabilliy, we see that high values can lead to better
The low varsortability value observed for the river data, particularly for the Danube, may be attributed to the fact that the width and catchment area of a river increase from the source to the mouth, resulting in reduction in the impact of extreme flows on river velocity closer to the river's mouth. Consequently, the marginal variance decreases. This serves to illustrate the importance of scales in real dataset. The data for the other rivers only covers parts of the river, which probably results in a less intense effect and higher values for varsortability.
In addition, can see that the Causal Chamber data is highly varsortable overall while having low R--value. This could be due to the fact that the variables controlled by the user are high in the causal order. We conjecture that deeper in the system and further from the user-controlled variables, dynamic turbulence and other sources of noise start having a larger and larger influence. Thus, mnxxlained noise is higher the lower we are in the causal order. This is in line with the low R"ssortability values as the causal parents explain less and less down the causal order. The increase in noise variance also drives the increase in total marginal variance and affects varsortability in this way.
Limitations This paper is an empirical study and we do not determine analytically why high varsortability leads to better performance of score based causal discovery algorithms for time series data. While we believe that our explanations for varsortability in the considered real-world datasets are plausible, they should be treated cautiously as hypotheses only. The only thing that we can say with certainty that sortability is highly context-dependent and therefore discarding scales as arbitrary for causal discovery seems premature.
Conclusion In conclusion, paper represents empirical extension of the work of Reisach et al. 22021 2023
We demonstrate that high var-sortability occurs in SVAR-simmlated time series data, resulting in enhanced per- formance of continuous score-based causal discovery algorithms assuming equal noise variance. Moreover, in some settings our simple benchmark algorithms outperformed or were en par with more sophisticated algorithms in the presence of high sortability. Consequently, we advise cantion when assuming equal noise variance for time series cansal discovery algorithms. Furthermore, it may be advisable to examine simulated data for high varsortability before using them as benchmark data, as was done in the 2019 NEURIPS competition we observe high and low varsortabilityy, as well as so-sortabillyy, in two different types of real-life datasets: the Causal Chamber data is generated in a controlled environment while the river flow dataset is measured in an uncontrolled environment. This indicates that var- and sortobbbbyy in auto-correlated time series data is not solely phe- nomenon observed in simulated data. For this reason, we believe that marginal variances may contain relevant causal information and exploiting variance or inverse variance sorting may be justified in some situations, if one combine it with physical reasons for one or the other (even though these physical reasons might already eliminate the need for causal discovery in the first place). Furthermore, the observed sRostrbbbiy scores indicate that the assumption of equal noise variance is equally tricky as the relative fraction of unexplained variance may change Uhroughout the graph and unequal noise variances are one possible reason for this.
et al. 2019). Finally,
10

---

Published in Transactions on Machine Learning Research (07/ 2025)
Acknowledgments
JW was supported by the European Research Council (ERC) Starting Grant CausalEarth under the European Union's Horizon 2020 research and innovation program (Grant AgreementNo. 948112, PI Prof. Jakob Runge) as well as by the European Regional Development Fund (ERDF) and the German Federal State of Saarland as part of the project (To)CERTAIN. During the initial phase of this project, JW was employed at the Technical University of Berlin and affiliated with the DLR Institute of Data Science as a guest researcher.
References
Til Ole Bergmann and Gesa Hartwigsen. Inferring causality from noninvasive brain stimulation in coguitive neuro- science. Journal of cognilive neuroscience, (3):11195 225, 2021.
Bart Bussmann, Jannes Nys, and Steven Latré. Neural additive vector autoregression models for causal discovery in time series. In Discovery Science: 24th International Conference, DS 2021, Halifaz, NS, Canada, October 11-13, 2021, Proceedings 24, pp. 446-460. Springer, 2021.
David Maxwell Chickering, Optimal structure identification with greedy search. Journal of machine learning research, 3(Nor):007-55, 2002.
Juan Gamella, Peter Bühlmann, and Jonas Peters. The causal chambers: Real physical systems as a testbed for ai methodology, 2024.
Aapo Hyvärinen, Kun Zhang, Shohei Shimizu, and Patrik O Hoyer. Estimation of a structural vector antoregression model using con-gaussianity. Journal of Machine Learning Research, 11(5), 2010.
Munoz-Mari, G Mateo, J Runge, and G Camps- Valls. Causeme: An online system for benchmarking causal discovery methods. In Preparation, 2020.
Mark Newman. Networks. Oxford university press, 2018.
Ignavier Ng, Biwei Huang, and Kun Zhang. Structure Learning with Continuous Optimization: A Sober Look and Beyond. In Proceedings of the Third Conference Causal Learning and Reasoning, pp- 71-105. PMLR, March 2024. URL. hettp::/pproceeding. mlr. mar.pree greass/236/ng244.. html
Roxana Pamfil, Nisara Srivattanaworachai, Shaan Desai, Philip Pilgerstorfer, Konstantinos Georgatzis, Paul Beau- mont, and Bryon Aragam. Dynotears: Structure learning from time-series data. In International Conference Artificial Intelligence and Slatisfics, pp. 1595 1605. PMLR, 2020.
Alexander Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated dag! causal discovery bench- marks may be easy to game. Advances in Newral Information Processing Systems, 34:27772-27784, 2021.
Alexander Gilbert Reisach, Myriam Tami, Christof Seiler, Antoine Chambaz, and Sebastian Weichwald. scale- invariant sorting criterion find causal order in additive noise models. In Thirty-seventh Conference on Newral Information Processing Systems, 2023.
Jakob Runge Discovering contemporaneous and lagged causal relations in autocorrelated nonlinear time series datasets. In Conference on Uncerlainly in Artificial Intelligence, pp. 1388 1397. PMLR, 2020.
Jakob Runge, Sebastian Bathiany, Erik Bollt, Gustau Camps- Dim Coumou, Ethan Deyle, Clark Glymour, Marlene Kretschmer, Miguel D Mahecha, Jordi Muñoz-Mann et al. Inferring causation from time series in earth system sciences. Nature communications, 10(1):2553, 2019.
Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse cansal graphs. Social science computer review, 9(1):62-72, 1991.
Ngoc Mai Tran, Johannes Buck, and Claudia Klüppelberg. Estimating a directed tree for extremes. Journal of the Royal Statistical Suciely Series B: Statistical Methodology, pp. qkad165, 2024.
Hal R Varian. Cansal inference in economics and marketing. Proceedings of the National Academy of Sciences, 113 (27):7310-7315, 2016.
Sebastian Weichwald, Martin E Jakobsen, Phillip B Mogensen, Lasse Petersen, Nikolaj Thams, and Gherardo Varando. Causal structure learning from time series: Large regression coefficients may predict causal links better in practice than small p-values. In NeurIPS 2019 Competition and Demonstration Track, pp. 36. PMLR, 2020.
11

---

Published in Transactions on Machine Learning Research (07/2025)
Afsanch Yazdani and Eric Boerwinkle. Causal inference in the age of decision medicine. Journal of data miining in genomics & proteomics, 6(1), 2015.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances neural information processing systems, 31, 2018.
12

---

Published in Transactions on Machine Learning Research (07/2025)
A Var and R2-sortability for different sizes of ER Graphs
Table 4: Mean of and standard deviation denoted by of varsortabilityvvar and -frsortabiliity (R²) for ER Graphs with different number of nodes. For each mumber of nodes 500 random graphs have been created and 500 samples have been generaled.
10 Nodes
20 Nodes
50 Nodes
100 Nodes
var
R
var
R
var
R
var
R
Contemp
0.71 + 0.16
0.54 + 0.22
0.72 + 0.12
0.52 + 0.15
0.71 + 0.08
0.50 + 0.12
0.72 + 0.06
0.50 + 0.08
Lagged
0.56 + 0.18
0.49 4 0.18
0.54 + 0.13
0.49
+ 0.13
0.54 0.11
0.50 0.11
0.54 0.10
0.49 1 0.09
Overall
0.58 1 0.09
0.51 0.10
0.56 + 0.06
0.51 + 0.06
0.54 1 0.03
0.50 1 0.03
0.54 1 0.02
0.50 1 0.02
13

---

Published in Transactions on Machine Learning Research (07/2025)
B Investigation on Influence of Different Degrees
B.1 On Varsortability
10
0.5
0.0
0.5
0.0
1.0
0
0.5
300
dc
0.6
0.0
1.0
0.6
00
080
0
0.6
0.0
di
Figure 4: Varsortability for d 10 nodes for different contemporaneous degrees de and lagged degrees d;
14

---

Published in Transactions on Machine Learning Research (07/202) 2025)
v.d
40 00
000 4B
op [] aD
v. d.d=1
<ue [] cm
9-0-
dd
D-a
T-
-
de -6
00 -0
dc
8
00
a5
1.0
20
30
40
60
8.0
d:
Figure 5: Varsortability for d = 20 nodes for different contemporaneous degrees de and lagged degrees dj
15

---

Published in Transactions on Machine Learning Research (07/202) 2025)
B.2 On R2-sortability
v.d==1 v. v.d.0..5
v.d.= dds
8
0
Figure 6: Itssoorabilityy for d 10 nodes for different contemporaneous degrees de and lagged degrees
16

---

Published in Transactions on Machine Learning Research (07/202) 2025)
v.
v. v.d0033.
11
v. v.d
V. -
v.
60-1-00
600
0.5
000
00P
T
0.0
0.5
1.0
2.0
3.0
4.0
80
8.0
Figure 7: RR-sortability for d = 20 nodes for different contemporaneous degrees de and lagged degrees d;
17

---

Published in Transactions on Machine Learning Research (07 (07/2023 /025)
C
Results Causalchamber Data
Dataset
Experiment
varsortability
R"-sortability
It camera_ walks__ _v1
color mix
0.95
0.28
It. camera walks_ v1
actuator mix
0.95
0.18
It color_ rengrrrerresionn__. v1
pol_1_90 __90
0.89
0.21
It. color regressionn
aperture 11.0
0.95
0.16
It regllorrrgreesoon__. vl
1000000.0
0.95
0.18
color cellorrrrrrssion...
bright_
0.95
0.00
It color regression v1
shutter bautter_speedd_e 0.002
0.95
0.19
___oloor____ recborrsngsesion.... vl
aperture 5.0
0.95
0.16
It "__color__a regierssooosssio____ _l
pol 1_4 45
0.95
0.16
color ceolor_rggressoo__11
reference
0.95
0.18
It regierrsegsssion__... v1
iso 500.0
0.95
0.16
color regoorrsggsssson__.. vl
shutter speed 0.001 uniform ir mid uniform diode vis mid uniform ost angle mid uniform __11__1______ 1__mid uniform diode ir 3 m_d uniform t ir 1 strong
0.95
0.14
interventions _saaddardddddddraaa vl
0.95
0.44
It anttervantios_sstaddaar vl
0.95
0.47
It. intervention_sstaddee
0.95
0.51
It cattttiionssttadrrdd vl
0.95
0.49
interventions sttaedddoosssttaddaddd vl
0.95
0.49
It interventions_staee _taaddast_ddd____a vl
0.95
0.40
interventions standard
uniform diode ir 3 strong uniform_ __te____________ _ed_mid mid
0.95
0.47
1t interventions_stanie
vl
0.95
0.47
interventions sntervemiions_sstaddaddddd vl
uniform_diode ir 2 mid
0.95
0.46
It interventions_ste sttasssttaaadd__._aa vl interventions strerventions_sstnddadd vl
uniform_ _orr__ss___________ angle 2 strong uniform_t f_ vis strong uniform pol 1 strong
0.95
0.47
0.82
0.46
snterventions_staadaaddd vl It interventions
0.86
0.46
It sttaandddnnnssssddddddaaa. vl
uniform osr angle weak
0.95
0.46
vl
uniform pol 2 mid
0.95
0.46
It interventions_staadee vl 1t snterventionsssttaddae vl It iitervantions_standadddd vl
uniform t ir 3 weak
0.95
0.49
uniform ir 2 weak
0.95
0.47
standard
uniform diode vis 2 mid uniform t vis weak
0.95
0.47
interventions tttervantions_ssaddaaddd
0.95
0.46
It interventions_stie Lactoonss_taaad____a_e vl
uniform 1 11 _iid
0.95
0.53
interventions staaddar vl
uniform osr angle mid uniform_v _______a_u_ v_aagl__________ _ssron uniform osr _er_k weak uniform_t t _att______r__________ mid uniform osr c strong uniform diode ir _strog
0.95
0.44
It interventions sttpenttinnstttaaadddaa v1
0.91
0.53
interventions stteadddrioosssttdaaddda vl
0.95
0.47
It interventions__taan _tt_stttaddd___ vl
0.95
0.44
interventions standdrddsaddadddaaaaaaaa vl
0.95
0.42
It ditervantions_standadddd vl
0.95
0.47
It interventions_staam
vl
uniform [ [_ir___________ ir ______iddd______________________
0.95
0.47
1t interventions__taanee standard vl
uniform ir strong
0.91
0.42
It. pitrveettons_standdrtt vl
uniform_ _polpp 1__ [ictorm__iid_________ vis 3 mid
0.91
0.47
1t seateettionssstandde
uniform diode
0.86
0.44
It interventions_ sttasssttaaddd_aa vl
uniform diode ir mid uniform blue mid
0.95
0.46
interventions staanddr vl
0.95
0.37
It interventions_staae standdarsddddd__aaa vl
uniform osr mid
0.95
0.47
uniform_t [_ vis vis_1__ssooog__ strong uniform _geeen___eeee__ee srre
0.95
0.47
1t Interventions_staane standarddd_d__________ vl
0.95
0.49
interventions stanoaadd_aaddddd_aaaaaaa vl
uniform diode ir 2_____sstr strong uniform_ _or___________________ las__angle__1 weak uniform v angle 2 mid uniform ir 1 weak
0.95
0.47
It interventions_ stt.sstttaddd____ee v1
0.95
0.46
It interventions standardtddddddaaa vl
0.95
0.53
It cetpuutiiossttaaaattee vl
0.95
0.46
It interventions standard stanndionnnttttarada_aaaaaa vl
uniform t vis 2 2_strong uniform bluu__stronggggg_ee___ee__a_e___ uniform t_ vis mid mid uniform mid
0.91
0.46
lt interventions snterventions_staadaadd vl
0.95
0.32
It interventions staniiard___adddd______ vl
0.95
0.49
interventions standard
0.95
0.47
18

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
It interventions __tandardd_dd_____ v1
uniform 1_22 _2_
0.95
0.49
It Interventions _ttanddda vl
uniform =_d
0.95
0.47
It interventions standard _ vl
uniform green _srong
0.95
0.12
uniform v_ -_angle__ 2 strong It interventions seteveenions_sttaddaddd vl
0.91
0.51
It ditervantions_staadaadd vl
uniform t_ vis 3 mid
0.86
0.46
It. interventions s
uniform osr angle I_ _srong
0.95
0.47
uniform t ir 2 strong It interventions_ate _ttaddarddddd___ vl
0.95
0.42
uniform angle mid It interventions_ste standard vl interventions_staniee
interventions staanddda vl
0.95
0.46
uniform pol 2 strong
0.86
0.49
uniform red strong vl
0.95
0.51
It. interventions __ttssssttdd_____a vl
uniform_ c s_strong
1.00
0.46
interventions _staddasttddd____a vl
uniform t vis mid
0.95
0.42
It. interventions iitevvantioss_standaaddd vl
uniform t vis weak
0.95
0.42
1b interventions_
vl
uniform ( vis 3 __eeak
0.91
0.47
II interventions_staan standdad vl
uniform 1 31 _iid
0.95
0.47
It interventions _ttnddardddddddd__aae vl
uniform 1 21 mid
0.95
0.49
It. interventions_sstneee vl
uniform reference
0.95
0.49
It [_waaks__vl _1 walks
actnators_whiin
0.96
0.26
color color_mix
0.84
0.21
wt walks_ vl
actuators random__ 9
0.93
0.26
wt walks vl
actuators _as___radoom_.mm pandom_walkk
0.93
0.19
wt walks_ vl
hatch mix s____slo__rrr www_ru___.
0.93
0.24
wt walks vl
actnatoor_randdee wwwrrrnndd_wa_a walk
0.93
0.36
wt walks vl
actuators rand_arndoommmmm Lanndm_wwlkk__..
0.88
0.24
wt walks vl
loads hatch mix slow_ run 3
1.00
0.36
wt walks v1
loads hatch mix slow run actuators random Baaddom__wkk____ actuators satiss_andommmm_ Laaddom_wll_____ loads loach_ mix sals__ssww________________ slow_run_ _ actuators random _aa___aakkk_
0.93
0.29
wt walks v1
0.95
0.29
wt walks vl
0.90
0.21
wt walks_ vl
0.93
0.29
wt walks
0.86
0.14
wt walks_ vl
loads_haach_______ mix ssows_lwww_____________ run
0.93
0.26
wt walks vl
actuators celuuatorsrrandonn.
0.95
0.24
wt walks vl
actuators sar__rrddd_weeeee 2
0.88
0.24
wt walks vl
actuators random wall actuators cetraatos_raaddmm raadoaannddaaaaaaaaaa 10
0.93
0.31
wt walks vl
0.86
0.21
wt walks vl
actuators random Saciloom_alll____ 12
0.93
0.31
wt walks v1
loads hatch mix fast run 5 loads hatch mix fast run actuators random sacttoommwallk___ 13 loads Load__hathh_____ actuators random walk actuators satiss_rnddmmmmm handom__alkk walk 14
0.93
0.33
wt walks vl
0.93
0.33
wt walks vl
0.90
0.29
wt walks_v vl
0.93
0.33
wt walks
0.88
0.24
wt walks_ vl
0.93
0.24
wt walks
loads hatch Biatch___i___________ fast 3 loads hatch mix fast run actuators random walk red _55
0.93
0.31
wt walks vl
1.00
0.36
wt walks vl
0.93
0.31
It malus_ vl
1.00
0.04
malus_v vl
white _128
1.00
0.02
It malus vl
green 64
0.96
0.02
malus vl It malus v1
blue_ 255
1.00
0.02
green 128
0.96
0.00
It malus_ve v1
white 64
1.00
0.02
It malus vl
blue 64
0.96
0.02
It. malus_ vl
green 255
1.00
0.00
It. malus v1
blue 128
1.00
0.01
It malus vl
white 255
1.00
0.02
It. malus vl
red 64
0.96
0.02
It malus__v vl
red 128
0.96
0.04
wt bernoulli v1
random loads both
1.00
0.00
19

---

Published in Transactions on Machine Learning Research (07 (025)
wt bernoulli vl
fans off
0.90
0.10
wt bernoulli vl
random loads intake
1.00
0.07
wt changepoints v1
load in ____eed___ 8
0.94
0.13
wt changepoints vl
load in _eed_sed____ 9
0.98
0.10
Wt changepoints vl
load in seed 4
0.94
0.13
changepoints vl
load in seed 5
0.94
0.15
wt changepoints vl
load in _sed__e____ 7
0.94
0.13
wt changepoints
load in seed 6
0.94
0.15
wt changepoints vl
load in _sed__e_____ 2
0.94
0.15
wt changepoints_
load in seed 3
0.94
0.15
wt changepoints_ vl
load in seed
0.94
0.15
wt changepoints vl
load in seed 0
0.94
0.15
wt intake impulse vl
load out 0.5 osr _or__ddwiind_
1.00
0.29
wt intake_1a jickiiee_pp________ vl
load_out_ _od__ou__0.5_1 0_5__.5______ _05_ __r__s____________ _as__dowwwidd_
1.00
0.29
wt intake impulse vl
load out
osr downwind 4
1.00
0.26
wt intake_impuull impulse
load out __t__u______ 0.5 oorr downwind
1.00
0.29
wt intake_imppnn immplle_imuuse___.
load out 0.01 osr dwwnwwnd
1.00
0.45
wt pressure_control control vl
hatch_0
0.92
0.36
test vl It test vl test vl
current_sennorr sensor
1.00
0.04
angle_senssor sensors
0.96
0.00
[iaalogccalbration
0.93
0.00
11 lest v1
ir ir_sensors
0.96
0.00
wt test vl
zero_lose eife_effects potis_coare coarse tach resolutinn
1.00
0.21
wt test vl
0.93
0.24
wt test vl
0.83
0.12
wt test
1.00
0.12
wt test vl
osr mic load potis _ fine osr barometers analog steps
1.00
0.00
wt test vl
1.00
0.10
wt test vl
0.86
0.12
wt
0.81
0.10
wt test v1
analog_calibration calibration
1.00
0.00
wt lest_v
0.93
0.43
It camera test v1
polarizer effect bright
0.95
0.00
camera. test
pure__cloore colors bright
pure
0.92
0.50
It. camera test v1
polarizer _effecttfttt. dark
0.95
0.07
camera test _est__l
pure pure_colors, colors_oodaaeeeee palette
0.95
0.50
It camera test vl
0.95
0.18
wt validate vl
validate 2
1.00
0.00
wt validate vl
validate in
1.00
0.02
w[ validate_vi validate wt va validate_ vl
validate_ _oaell___dd_____ee _ut
intake
1.00
0.00
validate
1.00
0.00
validate load nutt_curreett in
1.00
0.02
wt palidate_..
validate_v.
1.00
0.00
wt validate vl
validate_ood osr validate pot
1.00
0.00
wt validate
1.00
0.00
wt validate v1
validate osr -orwwiddd
0.88
0.12
wt validate
validate res out validate osr
1.00
0.00
wt va lidate vl wt va lidate vl
1.00
0.02
validate pot pot_
1.00
0.00
wt validate vl
validate load out mic validate load in mic validate load in current out validate_ Salidate_hathh____ validate_ost osreee__s_ottt
1.00
0.00
wt validate vl
1.00
0.02
wt validate vl
1.00
0.02
wt validate_v vl
0.90
0.02
wt validate vl
1.00
0.02
wt validate vl
validate osr usw__p
0.88
0.12
validate
validate
1.00
0.02
wt validate_ vl
validate load in validate mic
1.00
0.00
wt validate
1.00
0.00
20

---

Published in Transactions on Machine Learning Research (07/202) 2025)
wt validate v1
validate _sr intake validate _oadd_oaa______ out validate_oo _attaatte________e__ _as__________ asr_anmbent-
0.88
0.12
wt validate vl
1.00
0.02
wt validate vl
0.90
0.12
wt validate vl
validate_oo osr_in in validate__or _ortitt__sss_.. osr mic
1.00
0.00
wt va validate_1 vl
1.00
0.00
validate
validate hatch mic
0.93
0.07
wt validate vl
validate _atchhhhhhhhhhhhhhhhhh pressures
0.90
0.02
wt pe validate vl
validate pressure downwind loads
0.78
0.14
It validate__ vl
validate 1_1
0.96
0.00
validate_vv vl
validate ost
0.98
0.02
It. validate_ v1
validate 12
0.96
0.02
validate vl
validate v_c
1.00
0.00
It validate vl
validate aagle 2
0.98
0.02
11 validate vl
validate
0.96
0.09
11 validate
validate_oo cacee__ss____a_ sar_a
0.98
0.02
1t validate vl
Salidate_ppl___
0.94
0.06
validate
validate liee_ddodee_v___ vis
1.00
0.02
It validate_ vl
vis
1.00
0.02
validate
validate pol
1.00
0.02
It validate vl
validate _dode vis validate diode ir validate_ angle validate diode ir
1.00
0.02
validate
1.00
0.02
11 validate vl
1.00
0.02
validate
1.00
0.02
It validate vl
validate diode ir
1.00
0.02
validate
validate v v_angle__
1.00
0.03
It validate
validate green
0.98
0.00
validate vl
validate blue
0.98
0.00
It validate
validate 1 31
0.96
0.02
It validate vl
validate 32
0.96
0.00
validate
validate ir
1.00
0.00
It validate_ vl
validate t vis
1.00
0.00
validate
validate vis
1.00
0.00
It validate_ vl
validate 1 22
0. 0.96
0,02
validate
validate
1.00
0.02
It. validate_ve vl
validate vis
1.00
0.02
validate_vi
validate ir 3
1.00
0.02
It validate_ v1
validate 21
0.98
0.02
camera validate vl
validate shutter_speet
1.00
0.00
1t. camera validate v1
validate iso
1.00
0.00
1t. cameraa_validaeee validate_ v1
validate_green
0.98
0.00
camera validate v1
validate pol__
1.00
0.00
It. camera validate v1
malidtt__blee
0.98
0.00
lt. camera validate v1
validate caattt__ppttuueeee
1.00
0.00
It camera validate_ camera__vliidtte vl
validate_ _eed__eed___
0.96
0.09
camera validate vl
validate pol_
1.00
0.00
21


Reviews:
[{"review_id": "oqq3B0pHVA", "reviewer": "Reviewer_uH77", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong aspects:\n\n\t- Their main contributon is the Modified Sortability Criteria for Summary Graphs and all the conducted experiments. \n\n\t- Time-series variable of previously known varsortability and R^2-sortability introduced, implemented and comprehensively tested for time-series domain.\n\n\t- The main take away message \"Consequently, we advise caution when assuming equal noise variance for time series causal discovery algorithms.\" is important for the community to learn about and keep in mind.\n\nWeaknesses:\n\n\t- Their realisation that \"the presence of cycles would dilute the sortability signal and would naturally push it closer to 1/2.\" because they \"would always contribute a 1 to the denominator and a 2 to the numerator\" is clear to me. However I don't see a clear reason to exclude those and only those edges that were excluded in the current for of equation 6. It's also not clear that the computation of AP(G′) is efficient. Later on it's stated that \"If there are more nodes the varsortability seems to be higher.\" Is it surely not an intrinsic charachteristic of the introduced metric?\n\n\t- The chosen set of conducted experiments seems comprehensive, but the discussion/interpretation of the results is mostly left for the reader. Showing how certain datasets' varsortability and R^2-sortability relate to each other and that this realation can be vastly different accross datasets, and causal discovery methods, could be valuable. However, what does these indicate about the applicability of the causal discovery methods in general?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "As it is claimed by the authors, the paper presents empirical extension to existing works of Reisach et al., on varsortability (2021) and R/2-sortability (2023).\n\tThey make an attempt to extend those results to the time series setting and conduct comprehensive experiments in this field on several datasets (both (partially) simulated and real-world datasets) and causal discovery algorithms.\n\tThe authors' observations illustrate that scale plays an important role for potential causal discovery algorithms, so discarding scales in causal discovery as arbitrary may be premature, as they may encode significant causal information.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Provide more reasoning or background behind the introduced metric.\n\n\t- Discuss the \\textbf{meaning} of the descriptive observations made in the 6th chapter, mainly with respect to the applicability of causal discovery algorithms on the real world dataset(s).", "broader_impact_concerns": "- Listing the empirical results can be a valuable baseline for causal discovery researchers.\n\n\t- Pointing caution to the potential harmfullnes  of normalisation is also valuable for causal discovery research and maybe time-series modeling in general.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9j4LFTpkFI", "reviewer": "Reviewer_5XRN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n- The extension of sortability concepts to time series is interesting.\n- The empirical observation that DYNOTEARS performance correlates with high varsortability is insightful and aligns with previous work on score-based causal discovery methods.\n\n**Weaknesess**\n- The novelty is limited, as the main contributions are direct extensions of existing ideas from non-temporal causal discovery to the time series setting.\n- The work is purely empirical and some theoretical analysis would strengthen the paper. Specifically, it would be very highly valuable to see under which assumptions sortnregress would approximate the true causal ordering.\n- Var/$R^2$-sortability requires access to groundtruth, which is rarely given in real-world data. This limitation is acknowledged by the authors.\n- The proposed sortnregress is consistently outperformed by baselines, except for one instance.\n- The manuscript lacks clarity in several key explanations (e.g. introduction, notation, ...) (see Requested Changes).\n- There is not enough convincing evidence supporting the importance of sortability in time series data (see Requested Changes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends the notion of varsortability and $R^2$-sortability in the time series setting, and introduces a benchmark method, sortnregress, which is based on the proposed methodology. The authors empirically explore the varsortability of synthetic, competition, and real-world datasets; commonly used for causal discovery in time series data. The findings suggest that high varsortability is common in practice and that it correlates with improved performance of certain score-based causal discovery algorithms, namely DYNOTEARS.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Clarity**\n\n- The clarity in the introduction could be improved in the 3rd and 4th paragraph.\n  - R2 sortability is not adequately described.\n  - The motivation for exploring varsortability in real-world data is unclear, especially given that ground truth is rarely available.\n  - The sentence “we find sortability values quite far from 0.5” is vague. What do these values imply?\n  - The following sentences are not very clear: \n    - “discarding scales in causal discovery…” (4th paragraph) is confusing. What do the authors refer to here by scales?\n    - “We hypothesise plausible physical explanations…” It would strengthen the clarity to briefly specify the proposed hypothesis, and explain some intuition behind it and how these mechanisms relate to violations of the equal noise variance assumption.\n- Section 3 clarity could be improved:\n  - Referring to cycles and Eq. (4): “would always contribute a 1 to the denominator and a 2 to the numerator of Equation 4.” but if the value tends toward 1/2, shouldn’t it be the other way around? Please clarify.\n  - The colours in Figure 1 are confusing. Consider using one colour for admissible pairs, one for summary graph edges, and one (e.g. black) for bi-directed (ignored) edges.\n  - The explanation of admissible node pairs would improve if the explanation in the caption could be moved to the main text (while referencing the figure).\n  - The paragraph on DYNOTEARS performance makes strong claims that should be tied to the experimental results section. Please include specific references.\n\n- Experiments. Section 5.2: \n  - The use of d, d_l, d_c, and \\delta is confusing. Could you introduce d_l together with d_c?\n  - If the dataset generation follows the setup in DYNOTEARS, please mention that samples are taken after a burn-in period to ensure stationarity.\n\n- The notation could be improved. \n  - The edges are presented as a tuple ((i, t-k), (j,t)), but shortly after $(i, t-k) \\to (j,t)$ is used. \n  - To define the maximal lag, I believe the set should indicate $k\\in\\mathbb{N}$.\n  - For the summary graph, it should be indicated that k>0.\n  - Pa_G_sum is not necessary in Eq. (1), as a=0 if there is no edge in G. Therefore a simpler $\\sum_{i=1}^d$ could be used.\n  - In Eq. (4), the matrix $W^k$ uses a superscript k, what does this mean?\n  - The denominator in Eq. (6) introduces a sum in k, which is not used. Is this a typo?\n\n**Missing citations**\n\nThere might be some citations missing:\n- When introducing SVARs, consider citing some previous work (e.g. Hyvärynen et al. (2010)).\n\nHyvärinen, Aapo, et al. \"Estimation of a structural vector autoregression model using non-gaussianity.\" Journal of Machine Learning Research 11.5 (2010).\n\n\n**Experiments**\n\nGeneral: The paper could benefit from stronger empirical analysis demonstrating why sortability matters in time series causal discovery. It currently lacks insight into how high/low sortability influences practical outcomes.\n\n- Neurips data:\n  - The experiment just reports sortability scores. What does high/low sortability imply? Some interesting questions to consider here: \n    - Would they be easier to estimate using dynotears, or other baselines? \n    - How would the sortnregress algorithm perform in each case?\n- ER-SVAR data:\n  - The experiment also reports varsortability scores, but what do they imply? The figure is insightful in terms of whether sortability translates to higher performance, but I am not sure whether analysing sortability alone is significant unless some implications could be derived for those cases.\n  - In the original DYNOTEARS paper, they use a similar setup with n=500 and for d=10 the reported F1 score is close to 1.0. Why is the performance so low in your setting? The differences don’t seem to be clear.\n- Causal Chamber:\n  - The experiment reports very high sortability scores, but the sortnregress does not perform well, which seems contradictory given its design. Could you clarify what could be the problem?\n  - Table 4 only shows results for 2 out of the 17 datasets presented in Table 3. Does sortnregress also outperform PCMCI in the other datasets?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7KcHPAm1Jn", "reviewer": "Reviewer_4Bv7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well written and well formatted.\n\nThe experiments include one competition dataset and two real-world datasets with ground truth.\n\nThe contribution of the paper is clearly stated.\n\nThe limitations section is clearly discussed.\n\n**Weaknesses**\n\nThe discussion of related work on causal discovery in time series could be more comprehensive.\nIt would be beneficial to include a broader range of related methods, beyond just the most popular or classic approaches.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends var- and $R^2$-sortability concepts to the time series setting, evaluating both simulated and real-world datasets. The authors show that simulation data commonly used to assess causal discovery algorithms for time series is typically varsortable. They also demonstrate a positive correlation between the performance of continuous score-based causal discovery methods and the varsortability of data generated by autoregressive processes. Additionally, the paper provides valuable recommendations for designing benchmark datasets in the field of time series causal discovery.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the Weaknesses.", "broader_impact_concerns": "No ethical concerns to report.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)