CausalRivers - Scaling up benchmarking of causal discovery for real-world time-series
Authors: Gideon Stein
Maha Shadaydeh
Jan Blunk
Niklas Penzel
Joachim Denzler
Venue: ICLR 2025 Spotlight (2025)
Keywords: Causal Discovery
Benchmarking
Time-series
Primary area: datasets and benchmarks
TL;DR: The largest in-the-wild causal discovery benchmark kit to this date, including high-resolution ts and ground truth causal graphs with over 1000 nodes.

Abstract: Causal discovery, or identifying causal relationships from observational data, is a notoriously challenging task, with numerous methods proposed to tackle it.
Despite this, in-the-wild evaluation of these methods is still lacking, as works frequently rely on synthetic data evaluation and sparse real-world examples under critical theoretical assumptions.
Real-world causal structures, however, are often complex, evolving over time, non-linear, and influenced by unobserved factors, making
it hard to decide on a proper causal discovery strategy.
To bridge this gap, we introduce CausalRivers, the largest in-the-wild causal discovery benchmarking kit for time-series data to date.
CausalRivers features an extensive dataset on river discharge that covers the eastern German territory (666 measurement stations) and the state of Bavaria (494  measurement stations).
It spans the years 2019 to 2023 with a 15-minute temporal resolution.
Further, we provide additional data from a flood around the Elbe River, as an event with a pronounced distributional shift.
Leveraging multiple sources of information and time-series meta-data, we constructed two distinct causal ground truth graphs (Bavaria and eastern Germany).
These graphs can be sampled to generate thousands of subgraphs to benchmark causal discovery across diverse and challenging settings.
To demonstrate the utility of CausalRivers, we evaluate several causal discovery approaches through a set of experiments to identify areas for improvement.
CausalRivers has the potential to facilitate robust evaluations and comparisons of causal discovery methods.
Besides this primary purpose, we also expect that this dataset will be relevant for connected areas of research, such as time-series forecasting and anomaly detection.
Based on this, we hope to push benchmark-driven method development that fosters advanced techniques for causal discovery, as is the case for many other areas of machine learning.

Published as a conference paper at ICLR 2025
CAUSALRIVERS SCALING UP BENCHMARKING OF CAUSAL DISCOVERY FOR WOALDWORLD TIME-SERIES
Gideon Stein, Maha Shadaydeh, Jan Blunk, Niklas Penzel, Joachim Denzler Computer Vision Group Jena Friedrich Schiller University Jena Jena, Thuringia 07743, Germany gi gideon. stein@un unni-nni--- wniuunnnnnnnneeeee
ABSTRACT
Causal discovery, or identifying causal relationships from observational data, is a notoriously challenging task, with numerous methods proposed to tackle it. De- spite this, in-the-wild evaluation of these methods is still lacking, as works fre- quently rely synthetic data evaluation and sparse real-world examples under critical theoretical assumptions. Real-world causal structures, however, are often complex, evolving over time, non-linear, and influenced by unobserved factors, making it hard to decide on proper causal discovery strategy. To bridge this gap. we introduce CausalRivers[] the largest in-the-wild causal discovery benchmark- ing kit for time-series data to date. CausalRivers features extensive dataset on river discharge that covers the eastern German territory (666 measurement sta- tions) and the state of Bavaria (494 measurement stations). It spans the years 2019 to 2023 with 15-minute temporal resolution. Further, we provide addi- tional data from flood around the Elbe River, distributional shift. Leveraging multiple sources of information and time-series meta-data, constructed two distinct causal ground truth graphs (Bavaria and eastern Germany). These graphs can be sampled to generate thousands of sub- graphs benchmark causal discovery across diverse and challenging settings. To demonstrate the utility of CausalRivers, we evaluate several causal discovery ap- proaches through a set of experiments to identify areas for improvement. Causal- Rivers has the potential to facilitate robust evaluations and comparisons of causal discovery methods. Besides this primary purpose, we also expect that this dataset will be relevant for connected areas of research, such as time-series forecasting and anomaly detection. Based on this, we hope to push benchmark-driven method development that fosters advanced techniques for causal discovery, as is the case for many other areas of machine learning.
event with pronounced
INTRODUCTION
Causal discovery, the process of identifying causal relationships from observational data, has made significant theoretical progress over the past decade (Pearl 2009). (Peters et al.
2017). This has 2022) that espe-
led to the development of various methods (Vowels et al. 2022). Assaad et al. cially bear potential for fields where randomized controlled trials are impractical due to restrictions concerning interventions, such earth sciences, neuroscience, and economies. However, despite this progress, causal discovery remains a predominantly theoretically motivated area research. We argue that of the primary reasons for this is the challenge practitioners face in selecting appropriate causal discovery strategies, especially given the strong assumptions these methods are often required make about the underlying data, e.g. causal sufficiency, linearity, or the absence of hidden confounders. As an example, methods based on additive noise models (ANMs, (Peters et al., approaches like PC (Spirtes al.
2011h) assume specific noise distributions, while constraint- 2001 and FCI (Spirtes) (001)) assume that causal relationships underlying observational data are of a faithful nature, an assumption that was criticized by Andersen (2013).
https//caasslliivrrsiiitbbbbbiiu

---

Published as a conference paper at ICLR 2025
(a) Eastern Germany
(b) Bavaria
Figure 1: The causal ground truth graphs for river discharge measurement stations are provided with this benchmarking kit. Jointly, these two graphs hold over 1000 nodes. Different colors represent different data origins that specify in appendix A.1
Violations of these assumptions are particularly very common in fields like neuroscience or climate science, where the data-genette bata-generating process complex, often unknown, and typically influenced by unobserved confounding factors. This, in turn, also limits the reliability of synthetic benchmarking, as data-ged generatine processes fail meet the complexity of real-worle scenarios, leading inflated assessments of method performance, discussed in Reisach et al. (202T). Additionally, even exten- sive survey papers like Vowels al.) (2022) can provide limited guidance for practitioners, they cannot directly address which methods might provide meaningful insights when assumptions violated. Furthermore, a large part of the causal discovery literature relies on either purely synthetic experiments (Pamfil et al. (220) and simple real-workd examples with few nodes (Mooij al. 2016 Runge et al. 2019). This situation seems to be especially pronounced for time-series data, as even fewer datasets available. Instead, the focus of many works lies proving theoretical guaran- tees under assumptions as proof of their validity. While these insights are by no means unnecessary and provide an essential foundation for methods evaluation, they provide, again, limited help when faced with the complexity and unpredictability of the real world. Here we feel it necessary to re- call the iron rule of explanation the cornerstone of modern science (Strevens 2020): "scientists [...] resolve their differences of opinion by conducting empirical tests" In machine learning, this implemented through benchmark datasets, which provide standardized environments for rigorous evaluation of the performance of competing methods. These benchmarks not only facilitate fair comparisons but also reveal systematic weaknesses and, thus, actively contribute method devel- opment. For instance, computer vision reshaped by the ImageNet challenge that brought the surprising performance of the AlexNet architecture to the field's attention (Alom et al. 2018). similar vein, we believe that large-scale and realistic benchmark dataset for causal discovery could have a profound impact on the field. We also find that no such benchmark has been established for causal discovery from time-series for which provide evidence in the next section.
To bridge this gap, and inspired by a single five-node example in Muñoz Mari et al. (202)), we in- troduce Causal the by far largest in-the-wild causal discovery benchmarking kit, specifically
2

---

Published as a conference paper at ICLR 2025
for time-series data, to date. CausalRivers features extensive dataset river discharge, spanning from the year 2019 to the end of 2023, with a 15- 15-minute resolution. It covers the entirety of the eastern German territory (666 measurement stations) and the state of Bavaria (494 measurement stations). Further, we include an additional dataset from a subset of stations (A.3). which exhibits pronounced distributional shift through very recent extreme precipitation event (Figure 2). To complement this dataset, we constructed two causal ground truth graphs (Figure 1). that include all measurement stations. For this, we leveraged multiple informational sources such as Wikipedia crawls and remote sensing. Further information on the data origins is included in appendix A Importantly, as the full ground truth graphs hold over 1000 nodes, a direct application of causal discovery approaches to these time-series is unfeasible. Instead, we provide sampling strategies to generate thousands of subgraphs with a flexible amount of nodes and unique graph characteristics such as single-sink nodes, root causes, hidden-confoundingg
simply connected graphs. Along with the general characteristics of river discharge, which we discuss later, the dataset allows us to assess the impact of conditions such as e.g., high-dimensionallty, non-linearity, non non-stationarity, sea- sonal patterns, the presence of hidden confounding (through weather), misalignment of causal lag and sampling rate, and generally distributional shifts method performance.
To demonstrate our benchmarking kit, we conducted three sets of experiments, providing an overview of potential benchmarking use cases. First, we provide experiments on multiple sets of subgraphs. For this, we report performances of well-known causal discovery approaches, provide naive yet effective baselines, and evaluate some recent deep learning approaches. Here, we find that simple strategies can be robust, where many causal discovery methods struggle. Second, we evaluate how the selection of specifically informative subsections of observational data can affect the performance of different methods, something that could prove helpful in real-world applications. Here we find mixed results, as a proper selection depends on the specific causal discovery approach. Finally, we provide some examples of how domain adaption might be an interesting tool to cope with the complex nature of the provided data distribution. To make usage as accessible as possible, provide ready-to-use benchmark package with many features"] With this benchmarking kit, hope to pave the way for more work for closing the gap between causal discovery research and its potential applications. Finally, we are looking forward to seeing whether the provided data, as the amount of time-series data is extensive, might also be interesting to related disciplines such time-series forecasting, anomaly detection, or regime and change point identification (Aminikhanghahi & Cook 2017 Ahmad et al. 20246). To summarize, this work provides the following contributions:
menchmark-focussee method development and provide the ground-
The largest real-world benchmark for causal discovery from time-series to date.
An evaluation of established causal discovery methods on high amounts of in-the-wild data.
An introduction and ready-to-use implementation of the complete benchmarking kit.
2 BACKGROUND
The impact of benchmarking becomes evident in various fields where large-scale and realistic datasets have driven significant advances. As already mentioned, computer vision reshaped by the ImageNet challenge that brought the surprising performance of the AlexNet architecture to the field's attention (Alom al., (2018). Other examples GLUE (Wang al. 2018). which has become a standard for eva evaluating natural language processing models. Next to this, the SQuAD Further, WMT-2014 (Bojar et al. 2014) helped with establishing Transformers (Vaswani et al. (017) as the dominant architecture in natural language processing. Similarly, the LAION-5B dataset (Schuh- mann et al. (222)) has driven the development of vision foundation models. Moreover, RESISC45 (Cheng et al. (017)) helped cement deep learning for remote-sensing scene classification. Finally, the Cityscapes benchmark (Cordts et al. 2016) has accelerated research in autonomous driving, while the CASP13 benchmark has revolutionized protein folding via AlphaFold (AIQuraishi) 2019).
benchmark (Pranav et al. 2016) has pushed the state-of-the-art in question-answeriin
In similar vein, we believe that large-scale and realistic benchmark dataset for causal discovery could have a profound impact on the field. To date, however, such a benchmark is lacking. To visualize this absence, we provide overview of existing datasets that either cover real-worke data
3

---

Published as a conference paper at ICLR 2025
An extensive list of works that are used evaluate causal discovery approaches. A for "TTmm" denotes that the data source is time-series. A / for "Real world" denotes that both observational data and ground truth causal graphs are not synthetic. Further, denotes theoretical limit on the number of variables as datasets have synthetic components. We emphasize that there is no comparable-sine benchmark for time-series data to date.
Table 1:
Real world
Number of variables
Topic
Origin
Time
Semi-synthetic generation ATE
Neal et al. (2023)
X
Semi-synthetic generation" ATE
Shimoni al. (2018
X
Gen expressions Production line
D
Dibaeinia & Sinha 2020
X
X
Göbler al. 2024)
X
Gen expressions
Van den Bulcke
2006
X
Gen networks
Pratapa ct al.
2020
X
Visual understanding
McDulf et al.
2222
X
0
Mixed challenge 4TE
Dorie et al.
2019
X
Mixed challenge ATE
Hahn ct al.
2019
X
Ø
Benchmark kit (LLL))
Zhou et al.
20224
X x
109
Single-cell perturbation
Chevalley et al. (2022)
622
Mixed challenge
Guyon al.
(2008
X
132
Cause-effect pairs
Mooy et al.
2016
X
100X2
Congenital heart disease
Spiegelhalter al. (1993)
X
20
Lung cancer
Lauritzen & Spiegelhaltee (1988)
8
Food manufacturing
Menegozzo al. (2022)
34
Protein signaling
Sachs et al. (2005)
11
Bridges
Yoram Rerch
(1989
12
Abalons
Warwick Nash
(1994]
X
8
Arrow of time
Bauer et al.
(2016)
X
Ø
Pain diagnosis
Tu et al.
2019)
X
14
Aerosols
Jesson et af.
(202)
14
Industrial systems
Mogensen et al.
2024
233
Semi-synthetic generation ODE
Cheng et al.
(2023)
Ø
X
Kuramoto
1975]
Gen networks
Greenheld et al. (2010)
Ø
FMRI
Smith et (2011)
50
Benchmark kit (CauseMe)
Muñoz-Mari et al. (2020)
VIX
510
Benchmark kit (OCBD)
Zhou et al. 2024a)
VIX
11/0
Multi-Bes Balti-Bencchark
CausalRivers
1000
or attempt to imitate specific characteristics of real-world domains (semi-synthett data) in Table I For completeness' 's sake, also include datasets that only provide sample data (no temporal dimen- sion) as well as some datasets that are considered for average treatment effect estimation since is possible to repurpose them for causal discovery. As can be observed from our summary, while found almost 30 distinct datasets, few of them provide time-series data. Further, many datasets that provide authentic, real-world data have limited number of nodes included, making hard rely on them for benchmarking as they become susceptible to overfitting. Of course, we are not the first to recognize the difficulty benchmarking and comparisons in the causal discovery literature. Often, this situation is attributed to the fact that causal ground truth, along with proper observational data, notoriously hard to find (Mogensen al., 2024| Niu et al. 2024). Noteworthy, some works that attempt to improve on this situation through other means are Montagna et al. (2023), which tries to assess the robustness causal discovery methods towards violations of their assumptions, or Faller et al. (2024), which attempts to score methods based on their consistency
multiple subsets of data. Further, some approaches such as MMñoz-Marr et al. (2020). Niu et al. (2024) or Zhou et al. 2024b), aim to provide benchmarking through collection of varying synthetic and semi-synthettic data sources. While these approaches are, of course, a step in the right direction and should be considered along real-world benchmarking, they are not sufficient to fully dissect performance dif-
4

---

Published as a conference paper at ICLR 2025
ferences of varying causal discovery methods for in-the-wild applications. Finally, recent and promising attempt to benchmark causal discovery performance, we want to mention Mogensen et al. (2024) as complementing work. Here, the ground truth graph is of sufficient size (Table 1) to prop- erly benchmark performance. Importantly, as the domain is completely distinct from ours, we see this work promising additional benchmarking approach.
BENCHMARK DESCRIPTION
Table 2: Overview of the three provided datasets in CausalRivers.
Name
Nodes
Edges
Start date
End date
Resolution
RiversEastGermany 666
651
1.1.2019
31.12.2023
15min
RiversBavaria
494
490
1.1.2019
31.12.2023
15min
RiversElbeFlood
42
42
09.09.2024
10.10.2024
15min
Here, provide information on the origin of the data included in benchmark kit and the construction of causal ground truths. Further, we discuss unique challenges for causal discovery on in-the-wild datasets and some specific features that native to data domain: Hydrology- Finally, to provide comprehensive overview, we also include a list of features that we provide next to the data in our benchmarking kit, such sampling strategies and naive baselines.
3.1 BENCIMARK CONSTRUCTION
This benchmarking kit is concerned with river discharge, so the amount of water that flows through river. is measured in m² m²/s. As the amount water measured an upstream station directly influ- ences the amount of water measured by all downstream stations at a later point in time, we consider them as causal. Through causal discovery, these causal relationships are potentially recoverable from observational data, in this case, time-series data, alone. To produce the datasets provided in our benchmarking kit, we began by collecting information on available measurement stations in our selected geographical area. Through cooperation with eight different German state agencies (A.I. each state has its own network of measurement stations that serve primarily for flood prevention), were provided with raw time-series data along with some measurement station metadata. After some initial filtering (mostly removing duplicates and malfunctioning measurement stations), we ended up with around 666 and 494 valid time-series for the selected time intervals. Notably, no further preprocessing was performed, as we consider it interesting to challenge researchers to come up with preprocessing steps that specifically benefit their causal discovery approach. To construct the causal ground truth for these measurement stations, we leveraged mixture of meta-information provided by the state agencies, remote-sensing (Wickel et al. 2007). Wikipedia information crawls and handcrafting for a semi-automatic construction of the graph. Further, all edges were double- checked by hand in the final stage to correct for potential matching errors. For documentary pur- poses, we provide the full construction pipeline 3 Note that it specifically construeted to allow additional nodes to be added in the future. With this, and especially as there was recently a call for less static benchmarks (Shirali al.] (2022), we leave room extend the provided data in the future. In summary, we provide three distinct sets of time-series as displayed in Table 2] along with two ground truth causal graphs (Figure I). as the RiversElbeFlood causal ground truth is a sub- set of the RiversEastGermant benchmarking source as it more diverse in terms of geography and data origin than RiversBavaria. Alternatively, we suggest Bavaria as a tool for the exploration of domain adaptation.
graph. Importantly, we envision RiversEastGerman as the primary
3.2 BENCHMARKING KIT FEATURES
To maximize the usability of this benchmarking kit, we provide additional tools and resources along with the time-series and causal ground truth graphs. These tools and resources should allow searchers to tailor the dataset to their specific needs and evaluate the performance of methods in a more targeted and streamlined manner. Specifically, we provide:
5

---

Published as a conference paper at ICLR 2025
m?s
10
576
m2/s
10-
M
577
578
m7s
Jo
2019
2020
2021
2022
2023
2024
Year
x10->
Distributional shift the Elbe River
Data
Eibe in livooEd
Density
Elbe Rivers estGrrmany
578
200
1000
Discharge in mP/s
Figure 2: Left/Top: A single sampled causal relationship along with time-series data from Ri- versEastGermany A massive precipitation event is marked in red. Right bottom: The pronounced distributional shift between the same nodes of the Elbe in RiversEastGermany and RiversElbeFFood.
Tools sample from ground truth causal graphs access subgraphs with any number of nodes. Further, subgraphs can be restricted through graph characteristics such as connec- tivity or, e.g., geographical reality. An example of such sample can be found in Figure
Strategies to assess climatic conditions, especially precipitation, around any node by build- ing on the German weather service (DWD). These tools might be helpful for dissecting confounding effects and selecting specifically interesting time-series windows.
General preprocessing, data loaders, and display tools for all included datasets.
Three naive baseline strategies that can be used to assess performance properly.
Tutorials on all provided tools and on how reproduce the results reported in section 4
3.3
BASELINE STRATEGIES
With our benchmarking kit, we provide three baseline causal discovery strategies. First, we de- termine the causal direction between two time-series, here denoted 21 and I2, purely based on cross-correlation between 21 and lagged versions of 22. For this, we look for the lag at which the cross-correlation is maximized. If this lag is negative, meaning the highest correlation is between I2 is inferred. If the lag is positive, 2 <- inferred. We call this strategy simply CC for Cross-Correlation. Second, we rely on the actual magnitude of the time-series, featuring principle of causality that can be found in physics, where the mass of an object determines the causal direction (e.g., gravity). While in Physics, the arrow of causation typically points at the object with the lower mass, for rivers, this is reversed, as it is technically impossible for a very big river flows in smaller river (at least without river splits). To I2 if the mean of is bigger than the mean of J1. We call this strategy Reverse Physical, in short, RP. Notably, both RP and CC decide on one direction for each potential edge. However, as it is typically the case that rivers only flow in single location, we additionally restrict these strategies to select a maximum of one child node per parent node. This is done either by selecting the next larger river (+N) or the biggest river (+B) the only link or the river with the highest gross-correlation as the successor (+C). Finally, we evaluate the union between RP and CC, which denote Combo and where also test each restriction.
the present of .i and the future of æ2, 21 -
leverage this principle, we simply assume

---

Published as a conference paper at ICLR 2025
3.4 UNIQUE CHARACTERISTICS
Because our benchmark dataset covers a large area of Germany and is combined from multiple data sources, it exhibits number of interesting and unique features. Further, the domain of Hydrology brings in, of course, additional unique characteristics. In the following, we will discuss these at- tributes to help understand the complexity of the dataset. With this, we also hope shine a light on the specific challenges and opportunities it provides for causal discovery. Further, we include some additional dataset statisties in A.1
With over 1000 nodes, the datasets cover a wide range of geographical conditions, such mountainous, coastal, and urban areas, and a wide variety of distances between stations. With this, it also covers a wide range of causal structures, lags, and strengths. Additionally, while the geographic closeness of nodes influences the difficulty of detecting a causal relationship. other factors such as effect strength and elevation (and with this flow speed) also play a major role. The datasets include a range of interesting geographical anomalies, such dams, pump water storages, artificial canals, and tide effects, which can affect the causal relationships between nodes by altering the flow rate, water level, and consistency of relationships.
Geographical Realitics:
Weather Confounding: Weather confounding plays important role in the innovation of all time-series in the dataset. Rainfall can occur in a single node, across all nodes, or in a subset of nodes. Therefore, the impact of weather might be beneficial determine causal direction or be detrimental. Further, as rainfall appears suddenly. the dataset is characterized by non-stationarity, non- non-linearity, and seasonal patterns. To visualize, Figure displays the effect of a massive precipi- tation event at the end of the time-series that affects all nodes in the sample.
Causal Lag: Due to the varying distance and elevation between nodes, the speed of the rivers, and, in turn, the lag at which the causal effect occurs varies greatly throughout the dataset. Moreover, the causal lag of a specific relationship differs throughout the years as it depends on the amount of water that is present at a given time (the more water, the higher the velocity of the river.) We estimate this, along with weather confounding, to be core challenge of the benchmark, as many causal discovery methods assume a static causal structure with a fixed lag.
The sampling rate which data is collected directly impacts the accuracy of inferred causal relationships (Gong et al.] 2017 2015). If the sampling rate is too low, critical causal interactions between variables are missed. Moreover, sigh-frequency sampling may increase the computational burden and result in models that overfit transient fluctuations rather than true causal interactions. As the dataset is provided in a 15-minute resolution, it allows to explore the impact of different sampling and aggregation rates on causal discovery performance in real-world applications.
Sampling Rate:
Domain Biases: Besides occasionally allowing for the provision of a skeleton graph (e.g. (Runge) jet al. 2019),, causal discovery methods typically integrate little domain knowledge. Here, we want to note that depending on the domain, this might be unnecessarily agnostic as such information, if leveraged, could be beneficial. For CiveslRivers, an example of such information might be that rivers typically have single endpoint, which makes nodes with multiple children highly unlikely. Further, the magnitude of the time-series can be beneficial as the amount of water is unlikely to reduce along the causal direction. While these biases are quite specific to Hydrology, we expect that other biases in a similar manner exist in other domains and could also be utilized there.
4 EXPERIMENTAL RESULTS AND DISCUSSION
To demonstrate our benchmark kit, we conducted three experiments to demonstrate use cases and to gain interesting insights into the performance of various causal discovery strategies. During these experiments, we deployed the following well-established methods from the literature: PCMCI with
a linear conditional independence test (Runge et al. 2019), Varlingam (Hyvärinen et al. 2010), Dynotears (Pamfil et al. (022)) and a simple linear Granger causal approach ( (VAR), aiming at cov- ering all common archetypes (Assaad et al. 2022). Further, we evaluated two recent deep-learning techniques. First, CDMI (Ahmad et al. 2024a), nonlinear Granger causal approach that analyzes residuals of deep networks under knockoff interventions. Second, Causal Pretraining (CP, (Stein et al. 2024),, which learns direct mapping from multivariate time-series to a causal graph from synthetic data. Notably, we specifically chose to include CP as it directly allows for domain adap- tion via finetuning. Additionally, we evaluated the performance of proposed naive baselines
7

---

Published as a conference paper at ICLR 2025
during all experiments, As causal discovery methods typically come with at least some Hyperpa- rameters, we performed a rudimentary Hyperparameter search per method which we document in appendix A We, however, also note that methods that require fewer Hyperparameter configu- rations are more likely to be successful in practice, which should be considered when comparing methods. Therefore, while evaluate few method-specific parameters, we often selected default parameters. For all experiments, we test different resolutions (15min - 24H). To omit the complica- tion of finding an individual proper decision threshold, for all experiments, we chose report the mean (over all samples) of the AUROC scores of the best best-performing Hyperparameter combination as the final performance measure for method. During this, we ignored autoregressive links as these are always present and could potentially skew results. Finally, we want to emphasize that our benchmarking strategy serves as an example of what benchmarking with CausalRivers can look like. However, benchmarking procedures should be adjusted to the aspect of method performance in fo- cus. For example, our approach potentially overestimates performance when either a high variance of performance between different Hyperparameter combinations exists (as they cannot be properly selected without labels) or it is hard to determine decision thresholds. These are both complica- tions that should be taken into account for real-world applications and could also be integrated into benchmarking procedures in the future.
4.1 EXPERIMENT SET VARYING GRAPH STRUCTURES
As the first and most extensive experiment set, we performed causal discovery on subgraphs with varying graph characteristics and with the full-time-series available. We took Riverayat as the base graph for this experiment. For each set except the last one, we report results for graphs with three or five nodes. The following graph sets were evaluated:
Random: We sampled all possible connected subgraphs with three five nodes. Notably, this covers the entire dataset and the complete diversity of conditions that the benchmarking kit offers.
Close: We sampled all possible connected sub-graphs with edges that have maximum geo- graphic distance of 0.3 (euclidean distance of the nodes). By excluding long distances, the causal effect should be more pronounced. Notably, all subgraphs of this selection are included in "Ran- dom".
Random + 1: We samples all possible connected sub-graphs that have two or four nodes. We then added another disconnected node to the graph. To prevent confounding, we sampled the random nodes from the coast and border area where we have several completely disconnected nodes.
Root cause: We sampled all possible connected sub-graphs that have three or five nodes and where each has a maximum of one parent. With this, graphs are connected in the form of a single chain. We consider this useful for works on root-cause analysis (Ikram et al. 2022). Notably, all subgraphs of this selection are also included in "Random".
Probably, the most interesting set, we here selected sub-graphs with four or six nodes and where a single node has multiple children (while rare, these examples exist when rivers are naturally or artificially splitting). We then removed the node that has multiple children from the sample simulate permanent hidden confounding scenarios.
Confounder:
We sampled all possible connected sub- bob-graphs that have five nodes and combine two of them into single disjoint graph. To prevent connectivity, we chose to combine sampled with the largest possible geographical distance between them. With this, we aim to evaluate how causal discovery methods perform under larger number of potential non-related variables.
Disjoint:
The largest set, Random-5, holds more than 7500 subgraphs. The smallest set, Confounder-3, holds only 24 subgraphs. We report the results of this experiment in Table Further, we report full list of set sizes and alternative performance metrics A.3 With some exceptions, we found that our naive baselines often achieve scores similar to actual causal discovery approaches. Concerning established causal discovery approaches, we found the linear Granger causal approach (VAR) to be the most reliant. For the "Root Cause" graph sets, we found that ordering the graphs according to their size (RP+N) can outperform all other causal discovery methods. Notably, while both CP and CDMI allow for non- lineariyy and, to some extent, seasonality, we found no evidence for their superiority over linear approaches CausalRivers. Finally, while we found these graph characteristics to be a great start for comparison, many other characteristics could be explored (c.g.. single-sink nodes, empty graphs, or causal pairs) to further unravel differences between causal discovery strategies.
8

---

Published as a conference paper at ICLR 2025
Table 3: AUROC scores for Experiment Set 1. We mark the Top performances in green. Null model refers to predicting no causal links, which achieves the smallest possible AUROC. f: CP net- works are not able to process more than five variables. With some exceptions, Granger-bar based causal discovery (CD) approaches (VAR, Varlingam, and CDMI) achieve the most robust performance.
Structures:
Close
Root cause
Random +1
Confounder
Random
Disjoint
Method
3
5
3
5
3
5
3
5
3
5
10
RP
.80
.76
.76
.70
.74
.73
.62
.66
.79
.75
.75
RP+N
.72
.62
.81
.77
.71
.65
.58
.58
.72
.64
.65
Naive Baselines
RP+B
.78
.71
.61
.53
.81
.71
.61
.62
.76
.68
.58
CC
.68
.63
.70
.67
.64
.66
.65
.59
.71
.71
.66
CC+C
.69
.64
.72
.70
.69
.67
.64
.60
.71
.67
.75
RPCC
.70
.66
.70
.65
.68
.68
.63
.60
.72
.71
.71
RPCC+N
.68
.60
.73
.70
.67
.64
.62
.57
.69
.64
.66
RPCC+B
.68
.63
.60
.55
.72
.66
.63
.58
.69
.65
.57
RPCC+C
.71
.66
.71
.68
.71
.67
.65
.61
.71
.67
.75
Null model
.50
.50
.50
.50
.50
.50
.50
.50
.50
.50
.50
VAR
.81
.81
.79
.75
.80
.79
.71
.72
.82
.80
.82
CD Strategies
Varlingam
.79
.77
-77
.77
.84
.79
.68
.70
.79
.75
.83
Dynotears
.50
.50
.50
.56
.52
.61
.53
.53
.50
.61
.61
PCMCI
.64
.62
.70
.74
.83
.74
.66
.64
.65
.65
.80
CDMI
.81
.81
.72
.65
.82
.80
.63
.71
.80
.78
.75
CP (Transf)
.60
.65
.62
.68
.80
.72
.56
.56
.60
.65
CP (Gru)
.66
.58
.68
.56
.81
.65
.56
.56
.64
.60
4.2
EXPERIMENT SET TIME-SERIES SUBSAMPLING
Given that the full time-series is very long (roughly 175k time steps for the original resolution), were interested in whether selecting specific shorter, and hopefully informative, subsections might influence the performance of causal discovery algorithms. As motivation, one might imagine that the complete time-series most likely holds sections with little innovation, displays annual patterns, and includes nonstationary windows with high amounts of change (such RiversElleeFlody To test whether providing only a subselection can improve in-the-wild causal discovery, we restricted the causal ground truth graph to the 42 nodes included RiversElbeFlood. We then compared the causal discovery performance on the RiversElbeFlood graph with the performance on the full time-series and with the performance on month with almost no recorded precipitation (Oktober 2021) in the selected region. Concerning subgraphs, simply sampled all possible graphs with five nodes, equal to the sampling strategy "Random" from Experiment Set 1, and performed the same Hyperparameter optimization. We provide the results of this comparison in Figure 3a
Interestingly, we found mixed results, as some methods seem to benefit from lower exogenous influ- ences (PCMCI, "No rain") while others benefit from the additional strong distributional changes in "Flood". Especially noteworthy, Dynotears seems to struggle with the "No Rain" set while perform- ing reasonably well on the other sets. Our hypothesis here is that Dynotears, as a gradient-based method, struggles most with data that has little innovation. This could also be the reason why showed the worst performance in Experiment Set 1, as there are more geographic locations with lit- tle elevation included. Next, we note that the performance on this subset of the ground truth causal graph was generally little higher than in Experiment Set We attribute this to the geographical location (more elevation) of the nodes included in RiversElbeFlood We conclude that focusing proper time-series subselection strategy might be an interesting way forward make causal discov- ery methods more robust in real-world applications, it can strongly influence the performance of various methods. Furthermore, the property subselection seems to depend on the causal discovery method itself. Finally, we hope that these results encourage future works to put explicit effort into finding proper subselection strategies for the CausalRivers time-series that go beyond what we have shown here.

---

Published as a conference paper at ICLR 2025
Causal pretraiting finetuning performance
(a) Changes in method performance depending on the provided time-series data. We found mixed re- sults with some pronounced differences, e.g., for Dynotears. Notably, both data regimes, and Flood, only include four weeks of data while Full TS includes the complete five years.
0.9
+0.37
+0.36
+0.66
+0L1G
0.7
rain
c.a
without finstuu
Fineturing
0.5
0.9
.0.0
10.02
+0.07
Method
Full TS
No Rain
Flood
0.8
0.7
RP
0.78
0.78
0.70
CC
0.81
0.74
0.80
Fretusing
0.5
RPCC
0.81
0.79
0.74
3
Host Caise Rost Cause 5
Randeme's Randame
Confounder
Coolisunde'' Randem
Randemi
VAR
0.85
10.86
0.83
Varlingam
0.72
0.67
70.74
(h) Performance achieved through finetuning CP domain samples. Finetuning CP networks with random five vari- able samples from RiversBavaria strongly boosts its perfor- mance, even other more specified graph sets. This do- main adaptation often Ieads to improvements over the best approaches from Experiment Set
PCMCI
0.60
10.69
0.60
Dynotears
0.79
0.60
*0.80
CDMI
0.83
CP
0.65
0.66
70.74
Figure 3: AUROC scores for Experiment Set 2 (a) and Experiment Set 3 (b). We mark increases and performance with t. Further, the highest performance per method is marked bold.
4.3 EXPERIMENT SET DOMAIN ADAPTION
As a final evaluation, we leveraged the fact that we include two distinct ground truth graphs to provide results on whether domain adaptation can be leveraged to improve causal discovery perfor- mance. As this area of research is not yet widely explored, we wanted to provide first example of domain adaptation via Causal Pretraining (CP), method that specifically allows for it, as causally pre-trained neural networks can be finetuned whether the previously reported performance of CP on the RiversEastGermany dataset can be im- proved. To exccute this, we leveraged RiversBavaria and sampled training and validation examples (identical to the strategy "Random-5") to finetune pre-trained network provided by Stein et al. (2024). We performed a small Hyperparameter search, testing for different values the learning rate, weight decay, batch size, time-series resolution, normalization, and the CP architecture. After training, we selected the network that achieved the highest F1 max on "Random 5" (a GRU on 12H resolution and no normalization) and evaluated it all graph sets that were evaluated during Exper- iment Set 1. We report the results in Figure 3b and refer to for details. We found that fine-tuning on random samples with 5 variables from RaversBavaria improves the performance of CP on all graph sets, suggesting that the learned udaptation is not restricted to the specific fine-tuning set. Further, this performance increase was sufficient to outperform the best causal strategy of Experiment Set on 50% of the datasets. We take this as strong evidence that adapting casual discovery strategies to the target domain is highly beneficial and should be investigated thoroughly in future work.
supervised manner. We, therefore, investigated
5 CONCLUSION
In this paper, we presented CausalRivers, the largest in-the-wild causal discovery benchmarking kit for time series data to date. After motivating the need for such a benchmark by summarizing al- ternative datasets, discussed the benchmarking kit and its unique challenges and opportunities. Further, we conducted a set of experiments, aiming at an evaluation of causal discovery approaches in real-world applications and exploration of potential beneficial strategies. Our experiments showed, that many well-established causal discovery methods underperform real-world applica- tions and are even occasionally matched by simple but robust baseline strategies. With this, we conclude that more research is necessary, focusing on in-the-wild robustness, potentially through selecting relevant sections of a given time-series, and domain adaptation. We hope that this work provides the foundation for spires the development of other benchmarking approaches.
beachmmrk--riveeee development of causal discovery methods and in-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We gratefully recognize the support of iDiv (German Centre of Integrative Biodiversity Research), which is funded by the German Research Foundation (DFG-FZ) 118, 202548816) 16). Gideon Stein funded by the iDiv flexpool (No 06203674-22). Maha Shadaydeh supported by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) Individual Research Grant SH 1682/1-1. Special thanks to the following German state and federal agencies for the provision of discharge data and their cooperation in this project:
Thüringer Landesamt für Umwelt, Bergbau und Naturschutz
Sächsisches Landesamt für Umwelt, Landwirtschaft und Geologie
Landesamt für Umwelt Brandenburg
Landesbetrieb für Hochwasserschutz und Wasserwirtschaft Sachsen-Anhalt
Landesamt für Umwelt, Naturschutz und Geologie Mecklenburg- Vorpommern
Wasserstraßen un Schiffabrtsverwaluung des Bundes
Senatsverwaltung für Mobilität, Verkehr Klimaschutz und Umwelt
Bayerisches Landesamt für Umwelt
We thank Tim Büchner for helping with online resources and Michel Besserve for providing valuable input on the first draft of the paper. Raw data sources fall under the Datenlizenz Deutschland.
REFERENCES
Wasim Ahmad, Valentin Kashurg, Nina Kukowski, Maha Shadaydeh, and Joachim Denzler. Deep- Learning Based Causal Inference: A Feasibility Study Based on Three Years of Tectonic-Climate Data From Moxa Geodynamic Observatory. Earth and Space Science, 11(10)=20003EA0004433 2024a. ISSN 2333- doi: 10.10099/0023E4003333. URL UBLLhttpps///onnneeelbbo
wi waLeyy.coo//b///bb///00000//2212
Wasim Ahmad, Maha Shadaydeh, and Joachim Denzler. Regime Identification for Improving Causal Analysis in Non-stationary Timeseries, April 2024b. httlhttpp//axxiv..rm wtt.g/////./.....///////
02315
arXiv:2405.022
[stat].
Md Zahangir Alom, Tarck M. Taha, Christopher Yakopeic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Brian Van Esesn, Abdul S. Awwal, and Vijayan K. Asari. The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches, September 2018. [es].
URL http:://axa http://aaww.vvoooo//////////// arggabbs/8803 01164 1xXiv:8003.0
Mohammed AlQuraishi. AlphaFold at CASP13. Bioinformatics, 35(22):4862 4865, November 2019. ISSN 1367-4803. doi: 10. 1.00930oooinoommttcsszz22222 URL https://doi...... org/10.
Samanch Aminikhanghahi and Diane Cook. A Survey of Methods for Time Series Change Point Detection. Knowledge and information systems, 51(2):339-367, May 2017. ISSN 0219- 1377. doi: SMEiiclee/MMC54444
URL htt.https/// /wwww ncb.. nih. gtv/ pmc/
Holly Andersen. When Expect Violations of Causal Faithfulness and Why Matters. Philosophy of Science, (77-63766,, 2013. doi: 10. 1086//6773937
Charles K. Assaad, Emilie Devijver, and Eric Gaussier. Survey and Evaluation of Causal Discovery Methods for Time Series. Journal of Artificial Intelligence Research, 73:767-811 February 2022. ISSN 1076-9757. doi: 10.1613/jair 13142. URL (tt.sttts::/// www. jai org/ indexxixxx. php/ fair/article/vieww/
Stefan Bauer, Bernhard Schölkopf. and Jonas Peters. The Arrow of Time in Multivariate Time Series. In Proceedings of The 33rd International Conference Machine Learning, pp. 2043 2051. PMLR, June 2016. URL [atpps: httpp:://pprceeeinng mlr.. wttp:/...eee//////uuuea.e html

---

Published as a conference paper at ICLR 2025
O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Amint-Amand, R. Soricut, L. Specia, and A. Tamchyna. Findings of the 2014 Workshop on Statistical Machine Translation. Stroudsburg, PAAssociation for Computational Linguisties, 2014. ISBN 9781941643174. URL http:: /daaee:///deeuuee htt_httss::/daree P-O1TT---F22------3335-aa-------
uva. www.all/aa... ///////arrhhdediie htt.ee.w...//aaachh..deee. fier=
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote Sensing Image Scene Classification: Bench- mark and State of the Art. Proceedings of the IEEE, 105(10): 1883, October 2017. ISSN
1558-2256. doi: 10. 100/0/PR...20017255559999999 h htttttt////e:eee//eeeee wew.eeeeieeeee.................. abst Cbostract/docmmett//ea
Yuxiao Cheng, Ziqian Wang. Tingxiong Xiao, Qin Zhong, Jinli Suo, and Kunlun He. CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery. October 2023. URL
https://opennev// http:///ggpppaeeeeeeeeeeerrrrrrr
net
yyyGme)
Mathieu Chevalley, Yusuf H. Roohani, Arash Mehrjou, Jure Leskovec, and Patrick Schwab. Causal- Bench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data. September 2022. URL httpttt:////peeeetaaaeee//
net/ feeee//ooomiidddieee http://ww....e.ooooooe/////o....
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proceedings of the IEEE Con- ference on Computer Vision and Puttern Recognition, pp. 3213-3223, 2016. URL
thecvf com/ wtt.c/////oocttttcoeooo ctnt/et_tttttteeeeeeeeeeee 10rpr_20111 / The Mhe_Cityscaper Mhee_CCtyyccaps__atteeeee.c Dateeee__eeeteeeee_______ 6a001_6pppr.htmm
Payam Dibaeinia and Saurabh Sinha. SERGIO: A Single-Cell Expression Simulator Guided by September 2020. ISSN 2405-4712.
Gene Regulatory Networks. Cell Systems, 11(3):52-221.e S,
doi: 10. 20..0166f.coss.222000... 003. URL http:::/ wwtp::/wwwwwwwwwwwwc..cccmc..mme htt.::/ceeeccceeett...cocceeeaae
Vincent Dorie, Jennifer Hill, Uni Shalit, Mare Scott, and Dan Cervone. Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data
Analysis Competition. Statistical Science, 34(1):43-66 68, February 2019. ISSN 0883- 4237, 2168-8745.
doi: 10.1214/1-STT667. bre/joorrall/statiiiic.. wew.aiaallsiiiieeoooooeeeo 34/issue-isssl1 Aut wutlomated--vezsus-seee It Yourse Io- Metthods F Inference/
URL
10.1214/18-ST9611
Philipp M. Faller, Leena C. Vankadara, Atalanti A. Mastakouri, Francesco Locatello, and Dominik Janzing. Self-Compatibiltty: Evaluating Causal Discovery without Ground Truth. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, pp. 4132-4140. PMLR, April 2024. URL [attps:
ml wtt.://sreee//22////llee..
html
Mingming Gong, Kun Zhang, Bernhard Schoelkopf. Dacheng Tao, and Philipp Geiger. Discov- ering Temporal Causal Relations from Subsampled Data. In Proceedings of the 32nd Interna- tional Conference on Machine Learning, pp- 1898-1906. PMLR, June 2015. URL https:
/ //proceedings. mll press v3 gogggb1s 15 html
Mingming Gong, Kun Zhang, Bernhard Schölkopf. Clark Glymour, and Dacheng Tao. Causal Dis- covery from Temporally Aggregated Time Series. Uncertainty in artificial intelligence: proceed- ings of the conference. Conference on Uncertainty in Artificial Intelligence, 2017:269, August 2017. ISSN 1525-3384. URL https: /wwww ncbi. nnm. fom..ihhh........ gov. http://ccaarriiess.s PMC5995575/1
Alex Greenfield, Aviv Madar, Harry Ostrer, and Richard Bonneau. DREAM4: Combining Genetic and Dynamic Information to Identify Biological Networks and Dynamical Models. PLOS ONE, 5(10)=e1339) October 2010. ISSN 1932-6203. doi: 10. 10. /foural/joo
0013397. URL /sttpa://journaas plos. org http:///ww.oeeeeaattcccee/eddaa. Journal 1 pone. 0013397
12

---

Published as a conference paper at ICLR 2025
Isabelle Guyon, Constantin Aliferis, Greg Cooper, André Elisseeff, Jean- Phillippe Pellet, Peter Spirtes, and Alexander Statnikov. Design and Analysis of the Causation and Prediction Chal- lenge. In Proceedings of the Workshop the Causation and Prediction Challenge at WCCI 2008, pp. 1-33. PMLR, December 2008. URL http:://ppooceedings...ooo guyon08a. html
Konstantin Göbler, Tobias Windisch, Mathias Drton, Tim Pychynski, Martin Roth, and Steffen Sonntag. (aattttccaallsssebblyy $: Generating Realistic Production Data for Benchmarking Causal Discovery. In Proceedings of the Third Conference on Causal Learning and Reason- ing. pp. 609-642. PMLR, March 2024, httl:ttssppppcceeeeeee /2tpoeeeeinnnniiireeee//3333336 /236 mobler24a.htm..
mlr
P. Richard Hahn, Vincent Dorie, and Jared S. Murray. Atlantic Causal Inference Conference (ACIC) Data Analysis Challenge 2017, May 2019. URL http:://aaii..cr//a////00....0 arXiv:1905.099 [stat].
Aapo Hyvärinen, Kun Zhang, Shohei Shimizu, and Patrik O Hoyer. Estimation of Struc- tural Vector Autoregression Model Using -ou--aussia Journal of Machine Learning Re- search, 17956)) 2010. ISSN 1533-7928. URL http:/ // wwt./oog/ppp./.
v11 v11/hyvarinen 10..html
Azam Ikram, Sarthak Chakraborty, Subrata Mitra, Shiv Saini, Saurabh Bagchi, and Mu- rat Kocaoglu. Root Cause Analysis of Failures in Microservices through Causal Dis- covery. Advances in Neural Information Processing Systems. 35:31158 35:31158-31170, Decem- ber 2022. URL URLhttpp://prrccediii http:://ww.....er.....c.pppappee cc/paper lomneer_fileee wtwess/aapee//0222222
Abstracct-----------a--------eee Conference html
Andrew Jesson, Peter Manshausen, Alyson Douglas, Duncan Watson-Parris, Yarin Gal, and Philip Stier. Using Non-Linear Causal Models to Study Aemoso-CCloud Interactions in the Southeast Pa- cific. CoRR, January 2021. URL (ttth:tp//// /ttpt::::///eeeeeiiiiie http://ewwwwiieeeeeeeeeeeeeeeeee 20
Yoshiki Kuramoto. Self-entrainment of population of coupled non- linear oscillators. Mathematical Problems in Theoretical Physics, 39:420-422, Jamuary 1975. doi: 10.1007/BF6001336 URL
https: adwaas.aa.......... waw.aaaaaaaaaa...aaaaaaa//////a 1975LNP....39 39..p..KKK
... ................................ 420K ADS Bibcode:
L. Lauritzen and D. J. Spiegelhalter. Local Computations with Probabilities on Graphical Struc- tures and Their Application to Expert Systems. Journal of the Royal Statistical Society: Se- ries B (Methodological), 50(2):157-194, 1988. ISSN 2517-6161. doi: 10.1111/j.517-61611
198812001 URL httpp::/ee http://nnnnnnebbbaa wiley. com/ http/dcc/aaae bci//bs//10 cww.//ca/aab//..111111.. 5.2517-6161. 1988 tb01721.x
Daniel McDuff, Yale Song, Jiyoung Lee, Vibhav Vineet, Sai Vemprala, Nicholas Alexander Gyde, Hadi Salman, Shuang Ma, Kwanghoon Sohn, and Ashish Kapoor. CausalCity: Complex Sim- ulations with Agency for Causal Discovery and Reasoning. In Proceedings of the First Con- ference on Causal Learning and Reasoning, pp. 559-575. PMLR, June 2022. URL https: //proceedings. mlr. www..ppeess/ve htwp117/mmmufff2f22 html
Giovanni Menegozzo, Diego Dall' Alba, and Paolo Fiorini. CIPCaD- Continuous Indus- trial Process datasets for benchmarking Causal Discovery methods. In 2022 IEEE 18th Inter- national Conference on Automation Science and Engineering (CASE), pp. 2124-2131, August URL
2022. doi: 10. (0.11099CAAE499999.002 (ww.:/4999999222.99999996
ieee. org/
Destrract/dbumeent////
ISSN: 2161-8089.
Søren Wengel Mogensen, Karin Rathsman, and Per Nilsson. Causal discovery in a complex indus- trial system: A time series benchmark. In Proceedings of the Third Conference on Causal Leam- ing and Reasoning, pp. 1218-1236. PMLR, March 2024. URL htttp::/ ml wttpsss//3./oogssssseseeeee
Francesco Montagna, Atalanti Mastakouri, Elias Eulig, Nicoletta Noceti, Lorenzo Assumption
Rosasco, Dominik Janzing. Bryon Aragam, and Francesco Locatello. violations in causal discovery and the robustness of score matching. Advances in
13

---

Published as a conference paper at ICLR 2025
Neural Information Processing Systems, 36:47339 December 2023.
URL
neuripsssnnnnsnnnu.pesp.. www.:eeerppp.ppppppeeeeeeeeeeeee cc/paper httpppppe_..Llee//app/////////// htt https::/prooaecngg..eeeii..
Joris M. Mooij. Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Dis-
tinguishing Cause from Effect Using Observational Data: Methods and Benchmarks. Jour-
nal of Machine Learning Research, 17(32)11-10,, 2016. ISSN 1533-7928. URL http:
// jm prtp:rooor///pppp///////////////
html
Muñoz-Mari, G. Mateo, Runge, and G. Camps-Valls, CauseMe: An online system for bench- marking causal discovery methods., 2020. In preparation (2020).
Brady Neal, Chin- Huang, and Sunand Raghupathi. RealCause: Realistic Causal In-
ference Benchmarking. May 2023. URL htttps://openrett m2 m28F5RN64h11
net/forum?id
Wenjin NIu, Zijun Gao, Liyan Song, and Lingbo Li. Comprehensive Review and Empirical Evaluation of Causal Discovery Algorithms for Numerical Data, July 2024. URL http:
arXiv:2407.1305: [cs].
Roxana Pamfil, Nisara Sriwattanaworachan Shaan Desai, Philip Pilgerstorfer, Konstantinos Geor- gatzis, Paul Beaumont, and Bryon Aragam. DYNOTEARS: Structure Learning from Time-Series Data. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pp. 1695-1605. PMLR, June 2020. httptppss//ppccceengggg//////e// .ml mlr. press/
v108/pamte 1200a. html
Judea Pearl. Causal inference in statistics: An overview.
Statistics Surveys,
(none):9--166 January 2009, ISSN 1935-7516. doi: 10.1214/09-55057 URL
09 S5057. full
Jonas Peters, Dominik Janzing, and Bernhard Scholkopf. Causal Inference Discrete Data Using Additive Noise Models. IEEE Transactions on Pattern Analysis and Machine Intelligence. 33(12): 2436-2450 December 2011. ISSN 1939-3539,
doi: 10. 10..0 URL https:
//ieeexplore leee org/ abst ract/ http://dmmmmeeteeeeee.ee
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: infedracee.oundatiion foundations and learning algorithmx. The MIT Press, 2017.
Rajpurkar Pranav, Zhang Jian, Lopyrev Konstantin, and Liang Percy. SQuAD: 100,000+ Questions for Machine Comprehension of Text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016. doi: 10. 10.18833/v1/d1--1 1264. URL Intpps://cil
nii. ac htt.:pppcrrd11100011111
Aditya Pratapa, Amogh Jalihal, Jeffrey N. Law, Aditya Bharadwaj. and T. Murali. Benchmark- ing algorithms for gene regulatory network inference from single-cell transcriptomic data. Nature Methods, 17(2):177-154 February 2020. ISSN 1548-7105. doi: 00.1008/441592-00
Alexander Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the Simu- lated DAG! Causal Discovery Benchmarks May Be Easy to Game In Advances in Neural Information Processing Systems, volume 34, pp. 27772-27784, Curran Asso- ciates, Inc., 2021. URL http:///pproeeddnnneeeeesssspppp http:/pp.....pppeee/////aaa/////
Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdinovic. Detecting causal associations in large nonlinear time series datasets. Science Advances, 5(1 5(11)xeau4496. November 2019. ISSN 2375-2548. doi: (t.11///ciadvvaauu4900 arXiv:
abs/ abs/1702. 07007
Iphysics, stat].
Karen Sachs, Omar Perez, Dana Pe' Pe'er, Douglas A. Lauffenburger, and Garry P. Nolan. Causal Protein- Signnainng Networks Derived from Multiparameter Single-Cell Data. Science, 308(5721): 523-529, April 2005. doi: 10. 10.1126/science.. 105809. URL https: ststtps://ww.. science.............o .orgeeeeeeeeeeeeeeeeeoeeee doi
1105809
14

---

Published as a conference paper at ICLR 2025
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-2522 December 2022. URL http::ppss//rroceedingg..cc inerooceeedngggg.eeiippppps nwwrips. ottpas///ppeeeednssss..p..peeppp http://eeeeeerrrc//p/ppppeaaeeaa a htt2://2h224..11ee23333333333333 and Benchmarks. .tt.rksssllmmmlmm.mmmmm
Datasets.
Yishai Shimoni, Chen Yanover, Ehud Karavani, and Yaara Goldschmnidt. Benchmarking Frame- work for Performance-Evaluution of Causal Inference Analysis, March 2018. URL http:
arXiv:1802.05046 [es, stat].
Ali Shirali, Rediet Abebe, and Moritz Hardt. Theory of Dynamic Benchmarks. September 2022.
URL https:://ppppreeep/e//a net. forum - 18L
Stephen M. Smith, Karla I.. Miller, Gholamreza Salimi-Khorshide Matthew Webster, Christian F. Beckmann, Thomas E. Nichols, Joseph D. Ramsey, and Mark W. Woolrich. Network modelling methods for FMRI. Neurolmage, 54(2):875-891, January 2011. ISSN 1053-8119, doi: 10. hwtp://rwrriiaaeee.........0.00 atticle/pp1//1005111100000
URL https: / /wwww//wwww/..//////////////ww sc www.eeeeeetcccooccmoooo wwwp://ttt..cccc/cccccccccccce
David J. Spiegelhalter, A. Philip Dawid, Steffen L. Lauritzen, and Robert G. Cowell. Bayesian
Analysis in Expert Systems. Statistical Science, 83):229, 1993. ISSN 0883-4237, URL
https:// www. jstor.
Peter Spirtes. An Anytime Algorithm for Causal Inference. In International Workshop Artificial Intelligence and Statistics, pp. 278-285. PMLR, January 2001. URL https://
proceedings ngg www.mrr....... www.....es........ /ttprees////spirtesllaa...
Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, Prediction, and Search. MIT Press, Books-ID: OZOVEAAAQBAJ.
January 2001. ISBN 9780262527927.
Gideon Stein, Maha Shadaydeh, and Joachim Denzler. Embracing the black box: Heading towards foundation models for causal discovery from time series data, February 2024. URL http:
//axxiv.orgaabs//22.v
09305 arXiv:2402.09305 [cs].
Michael Strevens. The Knowledge Machine: How Irrationality Created Modern Science. Liveright Publishing, October 2020. ISBN 9781631491382. Google-Bookks I.w.II::ISSSAAA...... ISXWDWAAQBAJ.
Ruibo Tu. Kun Zhang. Bo Bertilson, Hedvig Kjellstrom, and Cheng Zhang. Neu- ropathic Pain Diagnosis Simulator for Causal Discovery Algorithm Evaluation. In Advances Neural Information Processing Systems, volume 32. Curran Associates, www.ffilleefeeeeee selee/paaprr2219//aaae Inc., 2019. URL otg.aa/sce....eeeeeeeeeeeaaaaaad Abstract.httm
4fdaal httaa//1112222222eeeeee/e/////e.
Tim Van den Bulcke, Koenraad Van Leemput, Bart Naudts, Piet van Remortel, Hongwu Ma, Alain Verschoren, Bart De Moor, and Kathleen Marchal. SynTReN: a generator of synthetic gene expression data for design and analysis of structure learning algorithms. BMC Bioinformatics, (1):43, January 2006. ISSN 1471-2105. doi: 10.1186/1471-2105.. orq/ 10.
URL httpss///do..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httppss//proccednngs neurips cc/ ccppppper/017//aaee... 335ae22337663111000ecccaaaaaaa33 Abst html
Matthew J. Vowels, Necati Cihan Camgoz, and Richard Bowden. D&#x2019:ya Like DAGs? A Survey Structure Learning and Causal Discovery. ACM Computing Surveys, Strreys,55(4::221113232 November 2022. ISSN 0360-0300. doi: 10. 145/335/5577544
URL htt.tttts::///i..///////
1145/3527158
15

---

Published as a conference paper at ICLR 2025
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. September 2018.
Tracy Sellers Warwick Nash. Abalone, 1994. URL https://archive. uci edu dataset/1
B. A. Wickel, B. Lehner, and N. Sindorf. HydroSHEDS: A global comprehensive hydrographic
dataset. American Geophysical Union, Fall Meeting, 2007:H1 HH-HHHHH-00 05, December 2007. URL
https: /ttps://ui.. adsabs. wwt.:aaddd.d.abb///000aa.. FHt.GGFF................1..111.1 .05W
hwwpa///.saaaaaa.....aaaaaaa.a 2007 1007AGUUM..H1H..OSM
ADS Bibcode:
Steven Fenves Yoram Reich. Pittsburgh Bridges, 1989. URL https::/arcciie...tt. ics uci,
edu/dataset/1B
Wei Zhou, Hong Huang, Guowen Zhang, Ruize Shi, Kehan Yin, Yuanyuan Lin, and Bang Liu. OCDB: Revisiting Causal Discovery with a Comprchensive Benchmark and Evaluation Frame- work, June 2024a. http:///://rxx wwt.oorg////b/oeee awwg/bbb/2405..
04598 arXiv:2406.0455 [es].
Yu Zhou, Xingyu Wu, Beicheng Huang, Jibin Wu, Liang Feng, and Kay Chen Tan. CausalBench: A Comprehensive Benchmark for Causal Learning Capability of Large Language Models. CoRR, January 2024b. URL https: http:///ppppwwwe../tooo net/ forum? DE-DEIIEEV177
16


Reviews:
[{"review_id": "N5Yk7WY1X9", "reviewer": "Reviewer_qr8D", "summary": "The authors present a high resolution dataset of river discharges that covers two large geographical regions in Germany with the purpose of benchmarking causal discovery algorithms. The presented dataset is interesting and presents a significant improvement over existing benchmarks in terms of scale and resolution and has the potential to be a significantly contribution to the development of causal discovery algorithms.", "questions": "- see weaknesses above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The dataset is interesting and improves over existing benchmarks both in terms of size and resolution. \n-  The authors provide a reliable baseline for the causal relations in the dataset. \n-  The paper is well written and easy to follow. \n-  The authors provide an extensive set of experimental baselines along with multiple software tools for analysing and processing the dataset.  \n- The authors commit to publishing the full pipeline used to construct the data set ensuring reproducibility.", "weaknesses": "- The authors could have discussed the completeness and reliability of the dataset, such as data quality checks and missing data handling more extensively. \n- Allowing the reviewers access the dataset and software resources would have been beneficial but is understandable given the nature of the article. \n- Although the authors mention that the data is compiled the from multiple sources these are not specified in the paper, providing a list of specific data sources would enhance transparency and reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gE1lXFggGN", "reviewer": "Reviewer_TrPo", "summary": "This paper introduces the CausalRivers benchmarking kit for comprehensive time series data analysis of river discharge in Germany. With high-frequency sampling every 15 minutes, the dataset includes data from a significant flood event on the Elbe River, making it suitable for testing causal discovery under real-world conditions with distributional shifts. The kit supports a range of causal discovery methods, from traditional approaches like Granger causality to advanced models like PCMCI, VARLINGAM, Dynotears, and CDMI, and allows sampling of subgraphs for diverse benchmarking cases. \nThe experimental evaluation uses data from 666 and 494 stations across eastern Germany and Bavaria, respectively, from 2019 to 2023. \nThe results revealed that while advanced methods struggle with real-world complexities like non-stationarity and high dimensionality, simple baselines performed robustly in identifying causal links. Domain adaptation via fine-tuning also showed performance gains.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper’s main strength is the extensive, realistic testing of causal discovery methods in the wild, which provides valuable insights into performance in a complex problem. CausalRivers supports various graph structures and sampling techniques, making it adaptable for different scenarios. It is obvious that benchmarks are important, and CausalRivers’ advanced methods allow for comprehensive performance comparisons across various scenarios.", "weaknesses": "Although this paper has an intensive assessment of models, its main weakness is that it is a technical benchmark. That is, the dataset’s complexity, along with various methods, may present implementation challenges for causal discovery researchers, and thus, it offers scientific opportunities. However, the scientific insight from the paper is missing (unless ICLR has changed its structure and now accepts also non-research but technical contributions, in which case this paper would be a good fit)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o4hXLcCLiI", "reviewer": "Reviewer_5VQx", "summary": "This paper provides a large-scale benchmark dataset for causal discovery to estimate causal relationships between time series. The dataset consists of river flow measurements from East Germany and Bavaria, collected at 15-minute intervals from 2019 to 2023, and also includes flood data from the Elbe River area. Benchmark experiments using various causal discovery methods on the time series data revealed that a simple baseline method demonstrated the most consistent performance, outperforming many other causal discovery techniques. Additionally, the use of deep learning-based causal discovery methods showed a significant improvement in performance through domain adaptation across different datasets. This dataset represents an unprecedented benchmark for causal discovery in time series data and is expected to greatly contribute to advancing research in this field.", "questions": "As noted in Table 1, there is still a lack of sufficient benchmark datasets for causal discovery in non-time-series domains, even for real-world data. Would it be feasible to construct similar benchmark datasets for causal discovery in non-time-series settings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper presents a large-scale benchmark dataset for causal discovery to estimate causal relationships between multivariate time series. The dataset provides an unprecedented scale of ground-truth data for causal discovery in time series, and it is expected to contribute to the advancement of research in this area significantly. Furthermore, the benchmark experiments conducted using this dataset revealed that a simple baseline method outperformed many other causal discovery techniques, offering important insights into the field, which has traditionally relied on evaluations using artificial data or simpler real-world datasets.", "weaknesses": "The benchmark dataset provided in this paper is of unprecedented scale and will undoubtedly contribute significantly to the field of causal discovery. However, the task may be somewhat simplified because the dataset is tied to geographical information. Predicting causal relationships between nearby river basins is relatively easy, meaning that causal discovery might only need to focus on a limited subset of the series. This could slightly diminish the dataset's overall value. The experiments seem to focus only on causal discovery within subsets of the time series. A broader evaluation of causal discovery across all series could lead to a more in-depth discussion.\n\nAdditionally, there are a few minor typographical errors. For instance, in Section 3.2.1, the phrase \"benchmarking kid\" should be corrected to \"benchmarking kit.\" Moreover, variables with upper bars are used in the same section without explanation, and the notation should be clearly defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)