WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series
Authors: Jean-Christophe Gagnon-Audet
Kartik Ahuja
Mohammad Javad Darvishi Bayazi
Pooneh Mousavi
Guillaume Dumas
Irina Rish
Venue: Accepted by TMLR ()

Abstract: Deep learning models often fail to generalize well under distribution shifts. Understanding and overcoming these failures have led to a new research field on Out-of-Distribution (OOD) generalization. Despite being extensively studied for static computer vision tasks, OOD generalization has been severely underexplored for time series tasks. To shine a light on this gap, we present WOODS: 10 challenging time series benchmarks covering a diverse range of data modalities, such as videos, brain recordings, and smart device sensory signals. We revise the existing OOD generalization algorithms for time series tasks and evaluate them using our systematic framework. Our experiments show a large room for improvement for empirical risk minimization and OOD generalization algorithms on our datasets, thus underscoring the new challenges posed by time series tasks.

Published in Transactions on Machine Learning Research (08/2023)
WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series
Jean-Christophe Gagnon-Audet
benn-brrstpph..ggonn--dddddddddd queeee
Mila Québec AT Institute
University of Montreal
Kartik Ahuja
aurtik. &irrela quebee
Mila Québec AI Institute
University of Montreal
Mohammad-Javad Darvishi-Bayazi
moohammaa..byyzziämmiii quebbr
Mila Québec AT Institude
University of Montreal
Pooneh Mousavi
Gina Cody School of Engineering and Computer Science Concordia University
Guillaume Dumas
Mila Québec AI Institude
CHU Sainte-Justine Research Center, Department Psychialry University of Montreal
Irina Rish
irina.rish @mmila. quebec
Mila Québec AI Institute University of Montreal
Reviewed on OpenReview: https: oppenrevew.... net/ form?? id= t TTQ
Abstract
Deep learning models often fail to generalize well under distribution shifts. Understanding and overcoming these failures have led to research field Out-of-Distribution (OOD) generalization. Despite being extensively studied for static computer vision tasks, OOD generalization has been severely underexplored for time series tasks. To shine light on this gap, we present WOODS: 11 challenging time scries benchmarks covering a diverse range of data modalities, such as videos, brain recordings, and smart device sensory signals. We revise the existing OOD generalization algorithms for time series tasks and evaluate them using our systematic framework. Our experiments show large room for improvement for empirical datasets, thus underscoring
risk minimization and OOD generalization algorithms the new challenges posed by time series lasks.

---

Published in Transactions on Machine Learning Research (08/2023)
1 Introduction
In the last decade, the success of deep learning has led to impactful applications spanning many fields (Krizhevsky et al.] 2012 Vaswani et al. 2017b Silver et al. 2016 Jumper et al. 2021 Brown
et al. 2020b]. However, parallel to this surge, there is growing evidence that deep learning models exploit undesired correlations due to selection biases, confounding factors, and other biases in the data (Geirhos et al. 2020: Shen et al. 2021 Ye et al. 2021a). These biases can often create shortcuts that help the model arrive at low empirical risk on a dataset. Nevertheless, a prediction rule relying on these shortcuts will not generalize out of its training distribution as it uses spuriously correlated factors instead of causal factors (Rojas-Carulla et al. 2015; Schölkopf et al.) 2021). Such failure becomes very concerning in real-life applications that
directly impact human lives, such medicine [Razzak et al. 2018 Ching et al, 2018
Rajkomar et al. 2018
or self-driving cars [Baduc et al. 2021 Janai ct al. 2020).
Spurious TCMNIST TOMNIST
CAP
SEDFx
PCL
LSA64
HHAR PedCount AusElec IEMOCAP
Fourier Source
Time
Domains Data Task
Classification
Classification
Classification Classification Classification Classification
Classification videos sign
Classification eccel/gvro activity aroeeni
Forecasting pedestrian crossing Eeee www.ereeeeooo
Forecasting X: consumption tn Peneonoiieen
Classification AV text Y: emetion
signal
X: wideo
X:
EEG
EEG
EEG
froquency
Y: pantyy
img
800
A...
Spurious
Spurious digit color correlation
Spurious digit color
EEG
group
Dataset
Signers
wetch
Locations
Emotion
correlation
frequency
80%%
90%
60%
correlation
90%
B SAN
D
20-40
40:60
90-99
Lee19
Cho17
5a6t1
3+4
7
63mm
gear
LG
T02
T01
0
9
Dacember
January C
shift
Test:L00
Test 10%
Test: 10%
Schalk04 gi1
0.10f!
Nexus4
T65
O
Holiicas
Rare shift
Domain Generalization
Subpop. Shift
Synthetic challenge
Real-world datasets
Figure 1: Summary of WOODS benchmark: tasks, modalities, domains and distribution shifts.
Let us explain an important failure mode with a common example from the work of Beery et al. (2018). Consider the task of distinguishing cows and camels in pictures. The training dataset is heavily tainted by selection bias, as the vast majority of cow images were taken in green pastures, and the vast majority of camel images were taken in sandy areas. A model trained to minimize empirical risk over the training dataset leverages the selection bias and ends using green background to classify cows and beige backgrounds to classify camols. As a way to capture different failures of deep learning models, much work has gone into finding and standardizing datasets with distribution shifts (Gulrajani & Lopez-Pas 2020; Ye el al. 2021bb] Koh et al.] (222).. These datasets provide a direction for research efforts in the field of OOD generalization. Gulrajani & Lopez-Paz/ (2020) gathered seven standard image datasets with distribution shifts and concluded that no OOD generalization algorithm considerably outperformed ERM, highlighting the need for better and more versatile solutions. Ye et al. (2021b) showed that some algorithms outperform ERM on specific types of shifts, highlighting that different algorithms might be needed for different type of distribution shifts. Koh et al. (2021) created a set of benchmarks of in-the-wild distribution shifts, highlighting the challenges in real-world applications. Further related works can be found in Appendix B
The above mentioned works have led to crucial empirical and theoretical insights towards addressing the OOD generalization failure in deep learning. However, they have been predominantly focused on static computer vision tasks, leaving the field of time series severely underexplored despite being essential to
various applications such as computational medicine Toooo 2019 Yang et al., 2021 Jarrett et al.] 2021
natural sciences (Stoffer & Ombao 2012 Tanaka el al. 2021) finance (Sezer el al. 2020 Heaton et al.
2016 6
Andersen et al. 2005), climate (Mudelsee) 2019). retail (Böse et al.} (017), ecology (Capinha et al.
2021
Christin et al., 2019)), energy (Deb et al. 2017) and many more TTorres et al.) 2021 Lim Zohren 2021) In
this work, we take the first step towards deeper understanding of distribution shifts in time series data.
Our key contributions are:
2

---

Published in Transactions on Machine Learning Research (08/2023)
We propose WOODS: benchmark of 3 synthetic challenge and 8 real-world datasets, totaling 11 datasets spanning a wide array of critical problems and data modalities, such as videos, brain recordings, and smart device sensory signals (See Figure
We develop systematic framework for easy evaluation of new time scries datascts and algorithms. The framework includes adaptation of existing OOD generalization algorithms for time series datasets.
We conduct extensive experiments on the above datasets with ERM and various OOD generalization algorithms. Our findings lead us to conclude that OOD generalization in time scries brings its own set of challenges and that there is a large room for improvement as shown in Table [
Why OOD generalization in time series? Table 1: Generalization gap between the In-Distribution (ID) performance and the OOD performance of ERM on
Recently, work in the deep learning community the WOODS benchmarks. See Section 6.2 for more details.
have shown that large scale pretrained models such as CLIP (Radford et al. 2021a) show con- siderable improvements when it comes to OOD generalization performance for static computer vision tasks (Cha et al., 2022). Since large scale raited models for time series data do not ex- ist yet, whether or not large scale pretraining on time scries data helps address OOD generalization challenge of time series remains to be determined. We hope our datasets and benchmarks help shed light on this important question.
Dataset
Performance
(Perf. is accuracy
ID
OOD
Gap
unless specified)
Spur--Fourier
74.5 (0.1)
9.8
64.7
-SM--ource TCM--Time
68.4
10.2
58.2
89.4 (0.0)
10.0
79.3
CAP
75.1 (0.7)
62.8
12.3
SEDFx
72.5 (0.4)
(7.8
5.2
PCT
73.6 73.6
(4.3
9.3
In the next section, discuss problem formu- lation, followed by discussion on the various datasots we use. In Scction 5 we describe the adaptation of existing methods for time series set- tings. In Section Ø we discuss the results followed by the conclusion and limitations in Section
HHAR
93.4
84.4(0.6)
9.0
LSA64
86.6 (1.0)
53.4
33.2
PedCount (rmse)
99.1 (2.7)
204.9
105.8
AusElec (rmse)
232.0
397.2(8.4)
165.2
IEMOCAP
69.1 (0.4)
51.7
11.4
2
Problem formulation
2.1 Static tasks
Consider the standard OOD generalization setting for static supervised learning tasks. Data samples: (X, Y) consists of the input observation X and the corresponding label Y. We gather the datasets Dd from the domains d E grrain which follow the follows the distribution [X, Y). Datasets from these domains form the training dataset Dtrain which follows the training distribution perain Edegman q grain where qraain e gle* is the vector of training mixture weights and gtrain is the mixture weight for domain d. We define predictor f. The performance of f on domain d is measured in terms of the risk (²*(f) E [((((X) Y)], where Ed is the expectation over the distribution Id and E + Rzo denotes the loss function. We evaluate the predictor on a set of test domains denoted as gall. The goal of OOD generalization is to use the training dataset Dtrain and construct a predictor f that can perform well on the test domains. We write this objective formally below.
Problem 2.1. Find predictor f that solves min, max Rd(f)
In the above problem, some restrictions are necessary on the set of testing domains gall to make Problem 2.1 of practical interest. Otherwise, the best predictor is random guessing, nothing can be assumed about the test domains. Many works (Arjovsky et al. 2020 Chen et al., 2021 Sun & Sacnko 2016) provide guarantees
of generalizing to OOD domains by assuming that the relationship between the label and some su subset of features (potentially a nonlinear transform of the observation
Slojas-Carulla et al.) 2018 Ahuja et al. 2020b) is invoriant across all domains. We call this subset of features the invoriant features, and any other features that might be correlated with the label are called spurious features. The predictor f solving Problem 2.1 is said to be OOD-optimal;; f" relies on the invariant features that generalize to all domains in gall (Koyama &
3

---

Published in Transactions on Machine Learning Research (08/2023)
Yamaguchi 2020 Because the set of training domains Errain is much smaller than the of testing domains Eall learning features that generalize to all test. domains is a challenging task.
In practice, we aim to solve Problem 2.1 to avoid the predictor to fail at test time when evaluated on the test dataset Drest which follows the test distribution pptest = Edeggm Ld gestpd where grsat E ml is the vector of training mixture weights and gft the mixture weight for domain d. There exists 2 significant ways the distribution ppte can shift:
Domain generalization (Arjovsky et al. 2020) The test domains are a superset of the training
domains, such that gtrain C grest We seek to generalize to the unseen domains Errain Erest (Gulrajani 2022
& Lopez-Paz
2020 Wang ct al.
Subpopulation shift (Koh et al. 2021 There are no unseen domains, such that Errain 2 grest
however, the test domains mixture different than the training distribution such that g +f g2 test We seek minimize the maximum domain error in grest (Sagawa et al. 2020 Yang et al., 2023 Santurkar et al. 2020
2.2 Time series tasks
Data samples consist of the input time series observation X [XaJeEs,, where S, is the set of time steps, and the set of labels Y = [V]]es,, where S, S St is the set of labeled time steps. The performance of the predictor f is measured in terms of the risk R E.d [<(f(X), Y)], where the expectation is taken over time samples from domain d. We formalize the OOD generalization problem in time series as Problem 2.1
In time series, similar to static tasks, the distribution shift can occur across data sources. Additionally, the distribution can also shift over time. As concrete real-world example of this charactoristic, consider a predictor monitoring a person's 's health from vital signs gathered with a smart watch.
Example 2.2 (Source-domains). Wrist characteristics such as size or hair vary across person, or sources. The solution to Problem 2.1 with persons as source domains would be a predictor that does not rely on spurious wrist characteristics and thus generalizes new persons. We call this formulation of domains Source-domains time series are taken from different sources, see Figure bb)
Example 2.3 (Time-domains). Heart rate is lower during the night when we are asleop and higher during the day when we are awake. However, when we are working during the night, our heart rate might be higher than on a typical night. A predictor that relies on spurions features like the time of day could make a false alarm regarding our health on an atypical day. The solution to Problem 2.1 with time of day as time domains d would be a predictor that does not rely on spurious features, and thus generalizes different activities different times. We call this way of defining domains Time-domains, the data distribution changes through time, see Figure c).
(a) Raw Data Samples
(b)
Source Domains (c)
Time Domains
Domain Domain
Domain
Domain Domain
Domain
X,
X
Samples
time
time
time
Figure 2: Illustration of the Source- and Time-domain definitions.
3 Synthetic challenge datasets
3.1
Spurious-Fouriée Fourier:: Spurious features encoded in the frequency domain
Colored MNIST (CMNIST) (Arjovsky et al., 2020) presented the failure mode of ERM under distribution shift. in the image domain. This was accomplished by creating training domains with strongly predictive
4

---

Published in Transactions on Machine Learning Research (08/2023)
(a) Spurious Invariant
LA
L B
H A
Hg
A
A
B
Figure 3: (a) Fourier spectrum construction in the Spurious-Pourier dataset. Signals have one low- frequency peak and one high-frequency peak. Signals are constructed from the Fourier spectrum with an in- verse Fourier transform. (b) Examples of reconstructed signals, both signals have the same high frequency, but different low frequencies, which are hard to distinguish visually.
or
or
(b)
{10, 80, 90)% correlation
75% freq correlation
A
time
spurions features and weakly predictive invariant features. The spurious correlation would be flipped at test time while the invariant correlation was kept the same. The correlation flip made it clear which features the model relied on to make predictions.
We create dataset composed of one-dimensional signals, where the task is to perform binary classification based the frequency characteristics. Signals constructed from Fourier spectra with one low-frequency peak (L n)) and one high-freee greh-freeuency peak (H A n), see Figure Domains D Iae lde0% (10%, 80%, 90%) contain signal-label pairs, where the labels are created such that the information carried by the low-frequency signal are d% correlated with the label (varies by domain), while the information carried by the high-frequency signal is 75% correlated with the label.
In the training dataset De (80%, 90%), the low-frequency signal are stronger predictor of the label (85%) than the high-frequency signal (75%). Therefore, minimizing the empirical risk fails at learning the invariant high frequencies as the low frequencies achieve the lower risk.
Appendix C.1 provides more information about the dataset.
3.2 Temporal Colored MNIST: A study of domain definitions in sequential data
(a)
di 10% 80%
Figure 4: Domain definition of both TCMNIST (a) Source and (b) Time datasets. Data samples are videos of four colored MNIST digits where the task to predict whether the sum of the current and previous digits in the sequence is odd or even. The color is spuriously correlated with the label.
2 4
3
(b)
d=90%
7
d=90% d=80% 1=10%

---

Published in Transactions on Machine Learning Research (08/2023)
In Temporal CMNIST (TCMNIST), extend the CMNIST dataset to binary classification task of video frames in order to investigate both domain definition paradigms presented in Section 2.2. Source-domains (Example 2.2 and Time-domains (Example 2.3 Videos are sequences of four colored MNIST digits where the goal is to predict whether the sum of the current and previous digits in the sequence is odd or even, see Figure 4 Prediction made for all frames except. for the first one. The labels are created such that the information carried by the color of the digits are d% correlated with the label (varies by domain), while the information carried by the value of the digit is 75% correlated with the label.
TCMNIST-Source Domains are created such that the color correlation is constant among the frames of a video, but varies between video from different domains E {10% 80%, 90%3. The domain definition is depicted in Figure [a).
Appendix
C.2 provides more information about the dataset.
TCMNIST-Time Domains are created such that the color correlation varies across frames. However, videos all have the same sequence color correlation, where the first labeled frame correlation is 90%, second is 80% and third 10%. The domain definition is depicted Figure ()..
Appendix provides information about the dataset.
4
Real-world datasets
4.1
CAP: Sleep classification across different machines
Figure Summary of the CAP dataset. (a) The task to perform sleep stage classification from EEG mea- surements. (b) The dataset has five source domains, where each domain contains data gathered with dif- ferent machine. The goal is to generalize to unseen machines.
(a)
(b)
Machine C
Z7
Machine o Machine E
Z
Machine
Machine
A recurrent problem in computational medicine is that models trained on data from given recording device will not generalize to data coming from another device, even when both devices are from similar equipment provider. Failure to generalize to unseen machines can cause critical issues for clinical practice because a false sense confidence
model could lead a false diagnosis (Kim et al., 2018 Engemann et al.] 2018).
We study these machinery-indueee distribution shifts with the CAP TTeraano al., 2001 Goldberger et al, 2000) dataset (Figure 5]-
We consider the sleep stage classification task from electroencephaloggaaa (EEG) measurements. The dataset has five source domains, where each domain contains data gathered with different machine. The goal is to generalize to unseen machines.
A Appendix
C.4 provides more information about the dataset.
4.2
SEDFx: Sleep classification across age groups
Figure 6: Summary of the SEDFx dataset. (a) The task to perform sleep stage classification from EEG mea- surements. (b) The dataset has four source domains, where each domain contains data from participants of a certain age group. The goal is to generalize to unseen age groups.
(a)
(b)
ZZ
Age 60-80
Age 80-100
Age 20-40
Age 40-60
6

---

Published in Transactions on Machine Learning Research (08/2023)
In clinical settings, train model on the data gathered from limited number of patients and hope this model will generalize to new patients in the future (Pfohl et al. 2022). However, this generalization between observed patients in the training dataset and new patients is not guaranteed. Distribution shifts caused by shifts in patient demographics (c.g., age, gender, and ethnicity) can cause the model to fail. We study age demographic shift with the SEDFx (Kemp et al. 2000) Goldberger al., 2000) dataset (Figure
We consider the sleep classification task from EEG measurements. The dataset has four source domains, where each domain contains data from participants of certain age group. The goal is to generalize to unseen age groups.
Appendix
C.5 provides more information about the dataset.
4.3
PCL: Motor imagery classification across data Mata-gathering procedures
Figure Summary of the PCL dataset. (a) The task is to perform motor imagery classification from EEG measurements. (b) The dataset has three source domains, where each domain contains a dalaset from different research group carrying out the same task. The goal is to generalize to unseen datasets of the same task.
(a)
(b)
Lee2019 MI
hor fo
Cho2017
PhysionetMI
Aside from changes in the recording device and shifts in patient demographics, human intervention in the data gathering process is another contributing factor to the distribution shift that can lead to failure of clinical models (e-g., Camelyon17 (Koh al., 2021 Sagawa et al., 2021 ). This challenge especially prevalent temporal modical data (e.g., EEG, MEG, and others) because recording devices are complex tools greatly affected by nonlinear effects and modulations. These effects
often caused by context and preparations made before the recording [Engemann al. 2018). We study these procedural shifts with the PCL (Lee
ct al. 2019 Cho et al.] 2017 Schalk et al. 2004 Jayaram & Barachant, 2018) dataset (Figure 7
We consider the motor imagery lask from EEG measurements. The dataset has three source domains, where each domain contains a dataset from a different research group carrying out the same task. The goal is to generalize to unscen datascts of the same task.
Appendix C.6 provides more information about the dataset.
4.4 LSA64: Sign language video classification across speakers
(a)
(b)
5 & 6
Figure 8: Summary of the LSA64 dataset. (a) The task is to perform signed word classification from videos. (b) The dataset has five source domains, where each domain contains videos of different signers. The goal to generalize to unseen signers.
3 & 4
9 & 10
1 & 2
7 & 8
Communication an individualistie way convey information through different media: text, speech, body languago, and many others. However, some media are more distinctive and challenging than others. For example, text communication has less inter-individual variability than body language or speech. If deep learning systems hope to interact with humans effectively, models need to generalize to new and evolving mannerisms, accents, and other subtle variations in communication that significantly impact the mcaning of the message conveyed. We study the ability of models to recognize information coming from unseen individuals with the LSA64 (Rochetti et al. 2016) dataset (Figure 8).
We consider the video classification of signed words in Argentinian Sign Language. The dataset has five source domains, where each domain contains videos of different signers. The goal to generalize to unseen signers.

---

Published in Transactions on Machine Learning Research (08/2023)
Appendix C.7
provides more information about the dataset.
4.5
HHAR: Human activity recognition across smart devices
(a)
(b)
Figure 9: Summary of the HHAR dataset. (a) The task to perform human activity classification from smart devices sensory data. (b) The dataset has five source domains, where each domain contains data gathered with different smart device. The goal is to generalize to unseen smart devices.
G. S3 mini
Galaxy
$3
Sam. Gear watch
Nexus
Accelerometer
Gyroscope
The intrinsic biases from inaccurate and poorly calibrated sensors of smart devices, along with the accumulated biases from everyday use makes human activity recognition a notoriously difficult task when task when done across devices (Stisen et al. 2015 Blunck et al. 2013 Contrary to static tasks where uninformative features can often be segmented out from the input features (e-g., background when classifying an animal from an image), invariant features in time series are often highly convoluted with other spurious features. We study the ability of models to ignore spurious information from complex signals with the HHAR (Stisen et al. 2015
Dua & Graff 2017) dataset (Figure 9
We consider the human activity classification task from accelerometer and gyroscope measurements of smartphones and smartwatches. The dataset has five source domains, where each domain contains data gathered with a different device. The goal is to generalize to unseen smart devices.
Appendix C.8 provides more information about the dataset.
4.6 PedCount: Forecasting of pedestrian crossings throughout locations
(a)
(b)
Summary of the PedCouni dataset. (a) The task is to forecast the count of pedestrian crossing streets of Melbourne. (b) The datasct has 65 source domains, where each domain contains pedestrian counts of a different street crossing. The goal to perform well on unseen street crossings.
Figure 10:
00
T01
T61
Data gathered from the behavior of a population follows scasonal (daily, weekly, yearly) trends. An example of this is the movement of population within a city, either by walking, public transport or car. These trends form from the daily life of the population, e.g., the influx in the morning, outflux in the evening, and absence on the weekend. However, these trends can shift when the data is gathered from different sources in a city. We study the impact of those trend shifts with the PedCount (City of Melbourne 2017 Godahewa. et al. 2021 dataset (Figure 10
The dataset has 65 source domains, where each domain contains pedestrian counts of different street crossing. The goal is to perform well on unseen street crossings. Specifically, investigate the OOD generalization to location T22 and T25.
Appendix
C.9 provides more information about the dataset.
4.7 AusElec: Forecasting of energy consumption throughout the year
Seasonality the property of time series where recurring characteristics appear every cycle of a fixed period, o.g., weekly. A common practice in the forocasting field to provide models with additional information, e.g., day of week in order allow models to leverage scasonality for better predictions. However, holidays is a seasonality of time series that is very sparse which models often fail to capture. We study the performance of
8

---

Published in Transactions on Machine Learning Research (08/2023)
(a)
(b)
Figure 11: Summary of the AusElec dataset. (a) The task is to forecast electricity consumption. (b) The dataset has 13 time domains, where each domain con- tains data from different months and holidays. The goal is to perform well on all seasonalities.
Jan
Feb
models sparse scasonality with the AusElec (Hyndman & Athanasopoulos 2018 Godahewa et al. 2021 dataset (Figure I1p
We consider the electricity consumption forecasting task. The dataset has 13 time domains, where each domain contains data from different months and holidays. The goal to perform well on all scasonalities.
Appendix C.10 provides more information about the dataset.
4.8
IEMOCAP: Emotion recognition across different conversational emotion shifts
Summary of the IEMOCAP dataset. (a) The task is to perform emotion recognition from multi modal data (video, sound, text). (b) The dataset has 11 time domains, where each domain contains data from a different emotion shifts during conversations. The goal to perform well on all conversational emotion shifts.
Figure 12:
(a)
(b)
neutral/angry
sad/frustrated
neutral/excited
sad/neutral
neutral/frustrated
happy/excited
angryffrustrated
happy/angry
No shifts
happy/neutral
Rare shifts
28.00 (0)) 22 020 (2)
Speakers tend to maintain an emotional state over conversation. However, external stimuli can invoke shift in the emotional state of speakers (Poria et al. 2019). Such emotion shift are often sparsely represented in the data, making it hard for models to classify them adequately. Recent work emotion recognition models (Poria et al. 2019 2018 Majumdor et al. 2019) show the failure of cxisting models to adapt to those emotion shilt. We study the performance of models on emotional shift with the IEMOCAP (Bulut el al., 2008) dataset (Figure 12
We consider the emotion recognition task. The dataset has 11 time domains, where cach domain contains data from different emotion shift during conversations. The goal is to perform well all conversational emotion shifts.
Appendix C.11 provides more information about the dataset.
5 Adaptation of OOD generalization algorithms to time series
Many algorithms were proposed to address the failure of machine learning models under distribution shifts. However, they were formulated for the image domain and require adaptation to be used with time series. We now describe how we adapt them to the time series settings.
On top of Empiriaal Risk Minimization (ERM, Vapnik 1998)). ), we have selected commonly used al- gorithms from the OOD generalization rescarch field to adapt and evaluate on WOODS benchmarks: Invariant Risk Mininization (IRM, Arjovssy et. al. (2020)), Group Distributionally Robust. Optimiza- tion (GroupDRO, Sagawa et al. (2020)), Variance Risk Extrapolation (VREx, Krueger al. (2021)),
Spectral Decoupling (SD, Pezeshki et al. (202)) Information Bottlencck Empirical Risk Minimization (IB-ERM, Ahuja et al. 2021)), ), Transfer (Transfer, Zhang et al. (2021a) ), Contrastive Adversarial Domain bottlemeck (CAD, Ruan et. al. (2021)), Conditional CAD (CondCAD, Ruan et al. (2021)) Conditional Contrastive Domain Generalization (CCDG, Ragab et al. 2022 ), Diversify (Diversify, Lu et al. (0023)
The loss function of above algorithms (except GroupDRO and Transfer) comprises of two terms: the empirical risk for domain (f) and penalty function P(f). For the empirical risk of domain d, average the risk
9

---

Published in Transactions on Machine Learning Research (08/2023)
across the set of labeled time steps of time series belonging to domain d: Sp.
R0) - T
Σ Σ (X. IED
((f(X1a))))
(1)
TESH
where nd is the number of samples from domain in the dataset D. Tn the case of Source-domains, all time steps of time series belongs to the same domain, while for the Time-domains there can be time steps belonging to different domains in the time series. IRM and VREx use penalty that relies on the risk across domains, we use the risk from Equation ① in the corresponding penalties.
P(f) nd |
Σ (X
X(f,X Yi),
(2)
rESS
where P is the penalty applied at each prediction point, e.g., P(f, X1a, Y) = [X for SD. Equations and 2 are simplifications of the adaptation; in Appendix D we provide more general formulation along with explicit penalty definitions for all algorithms used in this work.
6 Experiments
Our framework follows the DomainBed (Gulrajani & Lopez-Paz 2020) workflow for hyperparameter search and model selection for a fair and systematic evaluation of OOD generalization algorithms. We perform a random search over hyperparameter configurations, which we repeat three times for error estimation. We then report the performance of the model chosen with our model selection methods (see Section 6.1). Table 2
summarizes the technical characteristies and backbone used for every datasets in our experimentation (See Appendix C for full details of each dataset.)
Appendix provides more information on the on the framework along with hyperparameter search spaces.
Table 2: Technical characteristic summary of WOODS datascts
Spurr-Frvrrier
TCM-Souren
TCM-Time
CAP
SRDPc
PCL
ISA64
HHAR
PodConnt
AusElue
TEMOCAP
Tusk
Claseiiation
Cassilaation 17,000
Clasifiction 17,000 2
Clasifestion 01,230
Clasxticaton 218.712
Classiiationn 22,58 2
Cazsiliationn 3,200
Clasifectioo 18.74 G
Eeresating
Frescaninu
ChesiBontoos 7,438
Nuu. Sumpls
12/000
Nuu. canoes
G4
Donoll type
Source
Source
Time
Source
Souree
Sauce
Source
Souree
Souree US 5000 (1+-33
Sanrce
Time 11
Num. domains
Sepuenee hirth
3000010001i
3000010011
7520055011> (98)
S0000tie (0)
5000 (1+11)
Stepe sluge
(1)
(3.88
(2.00 Yrd [ths]
(1)
(3,038
(TI))
Prediction times
and
steps
strpe
stupe
Budchene
CNT+
Trundfenaa
Trandumeer
Multineda
Appee
Apps
Appe
Appe C.4
Aupen
Appa
Apps
Ange
Appel
Appe
6.1
Model selection methods
For domain generalization We split all the training domains into training and validation sets. With train-domain validation, we choose the model that gets the best average validation performance across training domains. With test-domain validation, we choose the model with the best performance on the test domain, however, we restrict the test domain queries the final training checkpoint only, effectively disallowing early stopping. With oracle train-domain validation, choose the model with the best performance the test. domain, however, restrict the test domain queries to the training checkpoint with the best performance the validation set of the training domains.
For subpopulation shift We split all domains into training, validation and test sets. With domain- average validation, we choose the model with the best average validation performance across domains. With worst-domain validation, we choose the model with the best worst domain performance.
Appendix G provides more details the model selection methods, and why chose them.
10

---

Published in Transactions on Machine Learning Research (08/2023)
Table 3: Summary of baseline algorithms performance on the real-world domain generalization datasets.
Train-domain validation
traio-domain
Objective
(ecrszacy)
(oostrayy)
Average
Objective
[semasee)]
CAP
SEDFFs
(assuey))
PCL.
ISAG4
HHAR
Average
FRM
Q7.5
53.4
(0.6)
(0.5
(0:9)
IIM
(L.7
(0.7
60.0
65.0
(0.9)
62.6
IRM
(0.5
64.3(0.6)
(6.400.4)
63.6 (2.0)
63.4
63.2
VREy
(8.6(1-7)
56.1
(0.3
46.8 (2.9)
(3.5
50.6
VREE
(1.3
(7.0
63.3 (0.3)0.3)
5000
(1.2)
0216
GruuDROO
62,000.8)
(0.2
64.8
(2.1
(0.4)
64.5
GrouDDRO
(2.9
(0.1
64.5(0.3)
(4.0
$4.3(0.4)
66.4
(3.2
(0.5
(1.33
67.3
(0.7)
(6.2(00.)
70.8(004)
65.0
(9.8(1.0)
SD
00.8 (0.9)
(9.8
(0.2)
30.7 (1.7)
85.6 (0.3)
66.2
63.2 (0.4)
70.8-0.44
65.3 (0.12)
(1.60
(1.22
68.8
CAD
06.1 (1.1
50.3 (0.6)
(3.0)
65.6
CAD
(0.8
(7.5
64.5
(7.5
84.8 (0.3)
67.7
ConCAAD
(0.5(0.8)
ConCCAU
06.6(0.5)
(1.3)
55.0(1.3)
61.0
(0.3(
47.3 (1.3)
84.4 (0.5)
(1.5
(1.9(022)
51.3
CCDG
61.7(1.)
65. (0.6)
64.3
84.7 (1.0
66.4
CCDG
63.1 (0.1(
(0.2
(0.5)
(6.0
85.8 (0.3)
67.7
Diverstly
07.4(1.9)
76.9 (0.3)
(0.4))
68.5
Divesilyy
(2.3
(0.3
(0.2
50.6 (1.3)
86.7 (0.15
68/2
Table 4: Summary of baseline algorithms performance on the synthetic challenge domain generalization datasets.
Train-domain validation
Test-domain validation
Spur-Fourrer (scruracy)
TCM-Source
TCM-Time (accuracy]
Spur-Fourrier
TCM-Source
TCM--Time (acrunacy)
Objective
(sccurscy]
Average
Objective
[sccurscy]
(acemnary)
Average
ERM
9.9
10.1 (0.0)
(.9
10.0
ERM
12.1 (2.0)
30.3 (0.8)
(2.4
23.7
IRM
(0.3
9.8 (0.1)
(0.3
10.1
IRM
58.8
52.7 (0.6)
50.6 (0.2)
54.1
VREx
(0.4
(08
9.8.).1)
63.7 (0.7)
49.7
(0.6
GroupDRO
(0.2)
(0.0)
10.1 (0.1)
10.2
GroupDRO IB-ERM
21.5(2.1)
33.5
24.3(3.9)
26.6
IB-ERM
(..().3)
(0.0
(0.2(0.0)
9.8
18.6(4.0)
28.1
33.7
26.8
SD
9.7 (0.2)
10.1(0.1)
(0.2
10.0
SD>
10.0(0.1)
27.5
31.8(5.1)
23.0
CAD
(0.4)
(.8().
9.91)
CAD
20.4(2.4)
26.3
27.2(2.3)
CondCAD
(0.3
(0.1
9.9.0.2)
10.1
CondCAD
16.0(1.))
20.6
22.7(2.9)
20.1
Thansfer
(0.1).
(0.2)
9.80
9.8
Transfer
13.4(2.8)
18.3
24.2
21.6
CCDG
(1.2(0.7)
(0.0)
(.9
10.4
CCDG 50.6(0.2)
(9.3
49.2
49.9
Diversify
12.2(0.9)
(.9
(0.2(0.0)
10.7
Diversify
67.0(3.2)
29.1
(9.8(2.1)
42.0
6.2 0OD generalization algorithms results
WOODS datasets have a significant generalization gap Table summarizess the generalization gap for all WOODS datascts, along with the In-Distribution (ID) and OOD performance. We compute the generalization gap to be an upper bound of the attainable performance on the Lest. domains. This is positively indicative that there is significant improvements to be made over ERM.
Appendix provide more details on how the generalization gaps are obtained.
Marginal improvement over ERM on WOODS real-world datasets Table 3 and summarrzes
the baseline results on our real-world datasete We observe a marginal improvement over ERM on several datasets with the adapted algorithms.
Performance SD, IRM, CAD, CondCAD, and Transfer reported forecasting datasets, because their adaptation to a forecasting task is not possible without significant alterations to their formulations.
Table 5: Summary of baseline algorithms performance Table 6: on subpopulation shifts datasets.
Summary of bascline algorithms perfor-
mance on forecasting domain generalization datasets.
Domain-average validation
Worst-domain validation
Train-domain validation
Oracle train-domain val.
Objective
AusElee
(mmee)
IEMOCAP
Objective
AusElee
(rmen)
IEMOCAP
(ocourscy)
PedCount (rmse)
PedCount (rmec)
Objective
Objective
ERM
(9)
(1.9)
ERM
(7)
56.3 (2.8)
IHM
X
(1.2)
IIM
X
58.9
ERM
204.1 (11.4)
ERM
223.2 (7.1)
VREx
415 (10)
59.1(1.4)
VREx
409 (4)
57.7 (3.1)
GroupDRO
409 (2)
(1.2)
GroupDRO
424 (13)
58.8(1.0)
VREx
201.6
VREx
213.1 (3.1)
IB. BR-E
394
(9.5(
BRER
391
58.5(1.5)
CroupDRO
243.2 (13.0)
GroupDRO
242.1 (9.9)
SD
X
(8.0)
SD
X
(6.1)
IB-ERM
213.1
IB-ERM
205.7 (11.3)
11

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithms fail on synthetic challenge dataset with train-domain validation Table _ summarizes the baseline results on our synthetic challenge datasets. We observe that IRM and VREx significantly outperform ERM on WOODS synthetic datasets with test-domain validation. However, all algorithms fail with train-domain validation as chosen models learned to rely on the spurious features which were anti-correlated with the label during lesting. This caused accuracies of 10,, significantly below the random guessing accuracy of 50%. We also observe that IRM and VREx under perform on real-world dataset.
6.3 Discussion and future research directions
Recent advancements in the fields of computer vision (CV) and natural language processing (NLP) have witnessed remarkable performance gains through the utilizat ion of large-scale web data for training mod-
els (Brown al. 2020a Rae al. 2021 Chowdhery al. 2022 Radford et al.) 2021b Wei et al. 2021 p.
This approach, which involves training models on diverse and abundant data (Kaplan et al.] 2020 has demonstrated enhanced OOD generalization capabilities across various tasks and domains (Miller et al. 2021 However, when it comes to time series data, the transferability of concepts learned from self-supervision is not guaranteed (Ma et al. 2023a), posing challenges for achieving similar scaling benefits in time series OOD genoralization. As a result, it remains uncertain whether a solution analogous to CV or NLP cxists for effectively addressing OOD generalization research advances are able achieve universal pre-training in time series, we hope WOODS can be a reliable evaluation ground for the development of these foundation models.
the context of time series data. In the scenario where
In the scenario that foundation models in time series remain unattainable for the near future, opportunities for alternative rescarch directions aimed at tackling the OOD generalization problem arisc. One promising avenue for exploration involves the deliberate construction of pre-training datasets tailored specifically for downstream tasks with distribution shift (Kostas et al., 2021 Malkiel et al. 2022p. By designing datasets that align closely with the characteristics of the target tasks, could be possible to maximize the transfer of learned representations to tackle distribution shift. This approach offers a potential proxy for achieving the benefits of large-scale pre-training. Consequently, investigating the construction and utilization of task-specific pre-training datasets represents an interesting direction for further research in this area.
Morcover, gaining a deeper understanding of the distribution shifts that occur in time series OOD generalization is essential for advancing the field. Similar sludies conducted in the domain of computer vision Ye et al. 2021b: Ruan et al. 2021 Ahuja et al.) 2020b) have yielded valuable insights into the characteristics of distribution shift. Building upon this knowledge and exploring how these insights can be translated and applied to the unique challenges and characteristics of time series data could unlock more effective methodologies and algorithms.
Several factors contribute to the complexity of establishing a universal representation for time scries data. Characteristics such as seasonality, frequencies, sampling rate, signal amplitude, and dimensionality introduce inherent challenges that impact the generalizability of models. However, maybe such factors could be leveraged with expert knowledge as part of the solution. By incorporating domain-specific insights and inductive biases derived from these characteristics, significant improvements OOD generalization performance might be possible.
Furthermore, investigating the influence of model architecture OOD generalization is another key aspect that warrants thorough analysis. Understanding the relationship between the architectural choices of models and their ability to generalize OOD can provide valuable insights for addressing the challenges of OOD generalization in time series. This knowledge could guide the development of novel architectures or modification strategies that are specifically tailored to enhance OOD generalization capabilities in time series data.
7 Conclusion & Limitations
This work introduced WOODS: a benchmark of 11 datascts for OOD generalization time series. We formulated the Source- and Time-domain settings for dealing with different scenarios of distribution shifts in time series. We adapted OOD algorithms to the time series setting, and provided their performance on
12

---

Published in Transactions on Machine Learning Research (08/2023)
WOODS datasets using our fair and systematic evaluation framework. With WOODS, take the first step and lay the groundwork towards understanding and solving distribution shifts failure mode of deep learning in time series.
While this work proposes an initial set benchmarks for OOD generalization in time series, our benchmarks are inherently biased toward Source-domains problems, ilassification tasks, and neurophysiology modalities. We hope for WOODS to be a platform continue building towards complete set of benchmarks with datasets covering those missing settings and other data modalities not currently studied in WOODS.
Broader Impact Statement
Failures deep learning models under distribution shifts are very concerning in real-life applications that directly impact human lives, such as medicine or self-driving cars. The WOODS benchmark hopes to give researchers and engineers meaningful measure of generalization performance to test new algorithms and alleviate potentially dangerous failures. However, it is possible that our benchmark does not accurately reflect OOD generalization performance for all possible applications. This could lead to false confidence in deployed system that could be dangerous to human life.
13

---

Published in Transactions on Machine Learning Research (08/2023)
References
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshncy, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning, pp. 145 155. PMLR, 2020a.
Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney. Empirical or invariant risk minimization? a sample complexity perspective. arXiv preprint arXiv:2010.16413 2020b.
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Chri--hhistophe Gagnon-Audet, Yoshua Bengio, Joannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. Advonces in Neural Information Processing Systems, 34, 2021.
Torben G Andersen, Tim Bollerslev, Peter Christoffersen, and Francis X Diebold. Volatility forecasting, 2005.
Martin Arjovsky, Léon Bottou, Tshaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2020.
Benjamin Aubin, Agnieszka Slowik, Martin Arjovsky, Leon Bottou, and David Lopez-Paz Paz. Linear unit-tests for invariance discovery, 2021.
Claudine Badue, Rånik Guidolini, Raphael Vivaequa Carnciro, Pedro Azevedo, Vinicius B Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M Paixao, Filipe Mutz, et al. Self-driving cars: survey. Expert Systems with Applications, 165:113816, 2021.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01277 2018
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gut/frund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Loic Barrault, Ondřcj Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koelm, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zamppeei.. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 1-61, Florence, Italy, August 2019. Association for Computational Linguisties. doi: URL http://///ccaaaaloooooooo///////
Sara Beery, Grant van Horn, and Pietro Perona. Recognition in terra incognita, 2018
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
Henrik Blunck, Niels Olof Bouvin, Tobias Franke, Kaj Grønbæk, Mikkel B Kjaergaard, Paul Lukowicz, and Markus Wüstenberg. On heterogeneity in mobile sensing applications aiming at representative data collection. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication, pp. 1087-1098, 2013.
Hend-kk Böse, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Dustin Lange, David Salinas, Sebastian Schelter, Matthias Seeger, and Yuyang Wang. Probabilistic demand forecasting at scale. Proceedings of the VLDB Endoument, 10(12):1694-1700 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. 2020a.
Advances in neural information processing systems,
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. A Aduances in neural information processing systems, 33:1877-1901, 2020b.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadch, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Temocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335 359, 2008.
César Capinha, Ana Ccia-Hasse, Andrew M Kramer, and Christiaan Mcijer. Deep learning for supervised classification of temporal data in ecology. Ecological Informalics, 61:101252, 2021.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo LapezoGazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-muliinnguaa and cross-lingual focused cvaluation. arXiv preprint (XX:::100.0000 2017.
Junbum Cha, Kyungjac Lee, Sungrac Park, and Sanghyuk Chun. Domain generalization by mutual-information regularization with pre-trained models. arXiv preprinl Xrxiv:2203. 10t029229..179,, 2022.
Yining Chen, Elan Rosenfeld, Mark Sellke, Tengyu Ma, and Andrej Risteski. Iterative feature matching: Toward provable domain generalization with logarithmic environments. arXiv preprint arXiv:2106 09919, 2021.
Yongqiang Chen, Youggang Zhang, Han Yang, Kaili Ma, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Invariance principle meets out-of-distribution generalization graphs. arXie preprint 0rXiv:2202.05441, 2022.
Travers Ching, Daniel S Himmelstein, Brett K Beaulieu-Jones, Alexandr A Kalinin, Brian T Do, Gregory P Way, Earico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M Hoffman, et al. Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface, 15(141): 20170387, 2018.
Hohyun Cho, Minkyu Ahm, Sanglae Ahm, Moonyoung Kwon, and Sung Chan Jun. Eeg datasets for motor imagery brain computer interface. GigaScience, 6(7):gix034, 2017.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXic preprint arXiv:2204. 02311, 2022.
Sylvain Christin, Éric Hervet, and Nicolas Lecomte. Applications for deep learning ecology. Methods in Ecology and Evolution, 10(10):1632 1644, 2019.
City of Melbourne. Pedestrian counting system 2009 to present (counts per hour), 2017. URL https://
vic. http:///www...a/////o//c///eaaaa
counts-/
b2ak-trbp
Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. The devil is in the detail: Simple tricks improve systematic generalization of transformers. arXiv preprint or Xix:22108. 12284, 2021.
Chirag Deb, Fan Zhang, Junjing Yang, Siew Eang Lee, and Kwok Wei Shah. A review time series forccasting techniques for building energy consumption. Renewable and Sustainable Energy Reviews, 74: 902 924, 2017.
Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. Adarnm: Adaptive learning and forecasting of time scries. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 402 411, 2021.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http::/archivveicc.uuu m1
Denis Engemann, Federico Raimondo, Jean-Rémi King, Benjamin Rohaut, Gilles Louppe, Fródéric Faugeras, Jitka Annen, Helena. Cassol, Olivia Gosseries, Diego Fernandez-Slezak, et al. Robust eeg- cross-site and cross-protocol classification of states of consciousness. Brain, (11) -192, 2018.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Florian Eyben, Martin Wöllmer, and Björn Schuller. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Mullimedia, pp- 1459 1462, 2010.
Haoqi Fan, Tullie Murrell, Heng Wang, Kalyan Vasudev Alwala, Yanghao Li, Yilei Li, Bo Xiong, Nikhila Ravi, Meng Li, Haichuan Yang, Jitendra Malik, Ross Girshick, Matt Feiszli, Aaron Adcock, Wan- Yen Lo, and Christoph Feichtenhofer. PyTorchVideo: deep learning library for video understanding. Proceedings of the 29th ACM International Conference on Multimedia, 2021. hatps://pyyorccviioo..om
Robert Geirhos,
Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nalure Machine Intelligence, 2(11): 665 673, Nov 2020. ISSN 2522-5839, doi:
020-00257-2 URL http::/dx..... www.looo.oogr... org/10
1038/s4255-022-0022--
Muhammad Ghifary, W Bastiaan Kleiju, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551-2559, 2015.
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey Webb, Rob J Hyndman, and Pablo Montoro-Manso. Monash time series forecasting archive. arXiv preprint arXiv:2105. 1r5.:22105.0664, 2021.
A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkin and PhysioNet: Compo- nents of new research resource for complex physiologic signals. Circulation, 101 (23):e2221 e220, 2000. Circulation Electronic Pages: http://drr.aaaajouuals.oor http:///waaa...rooantenne///e//e 101.//00.......1........
PMID:1055218; doi:
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization, 2020.
Lin Lawrence Guo, Stephen R Pfohl, Jason Fries, Alistair EW Johnson, Jose Posada, Catherine Aftandilian, Nigam Shah, and Lillian Sung. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Scientific reports, 12(1):1-10, 2022.
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. Conversational memory network for emotion recognition in dyadic dialogue videos. In Proceedings of the conference. Association for Computational Länguistics. North American Chapter. Meeting, volume 2018, pp. 2122. NIH Public Access, 2018.
Ruining He and Julian McAuley, Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference world wide web, pp. 007-517, 2016.
Yue He, Zheyan Shen, and Peng Cui. Towards non-i.idd image classification: A dataset and baselines, 2019.
JB Heaton, Nicholas G Polson, and Jan Hendrik Witte. Deep learning in finance. ar preprint arXiv:1602.06561, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.122: 2019.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzie, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve distt--iiittiitti robustness. arXiv preprint arXit:2004.06100, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340 8349, 2021a.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEER/CVF Conference on Compuler Vision and Paltern Recognition, pp- 2021b.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory, Neural computation, 9(8):1735 1780, 1997.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovee. Open graph benchmark: Datasets for machine learning on graphs. Advonces in newral information processing systems, 33:22118-22 1211333, 2020.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalise? Journal of Artificial Intelligence Research, 67:757- 795, 2020.
Robin John Hyndman and George Athanasopoulos. Forecasting: Principles and Practice. OTexts, Australia, 2nd edition, 2018
Joel Janai, Fatma Güney, Aseem Behl, Andreas Geiger, et al. Computer vision for autonomous vehicles: Problems, datasets and state of the art. Foundations and Trends& in Computer Graphics and Vision, 12 (1-3):1-300) 2020.
Daniel Jarrelt, Jinsung Yoon, Toana Bica, Zhaozhi Qian, Ari Ercole, and Mihaela van der Schaar. Clairvoyance: A pipeline toolkit for medical time scries. In International Conference on Learnöng Representations, 2021. URL https://openrrviiet ntt::formmidd=nnCVveeieee
Vinay Jayaram and Alexandre Barachant. Moabb: trustworthy algorithm benchmarking for bcis. Journal of neural engineering, 15(6):066011, 2018.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvuakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 589, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 0rXiv:2001.08361, 2020.
Bob Kemp, Aeilko H Zwinderman, Bert Tuk, Hilbert AC Kamphuisen, and Josefien JL Oberye. Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg. IEEE Transactions Biomedical Engineering, 47(9):1185-1199) 2000.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measuring compositional generaliza- tion: A comprehensive method on realistic data. arXie preprint xr 09713,, 2019.
Najoung Kim and Tal Linzen. Cogs: A compositional generalization challenge based on semantic interpretation. arXiv preprint arXiv:2010.08 05465, 2020.
Yang-Min Kim, Jean- Baptiste Poline, and Guillaume Dumas. Experimenting with reproducibility: case study of robustness in bioinformatics. GigaScience, 7(7). 06 2018. ISSN 2047-217X. doi: 10.1093/
sraascienee/giy077. URL https:://oi..rrr http://.....///aaaaeieeeeeeee wwt.////ggaaccinn////iior////// giy077.
Masanari Kimura, Takuma Nakamura, and Yuki Saito. Shift15m: Multiobjective largo-scale fashion datasct with distributional shifts. arXiv preprinl ar Xiw:21008. 12992, 2021.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICMT.), 2021.
17

---

Published in Transactions on Machine Learning Research (08/2023)
Demetres Kostas, Stephane Aroca- CrocleOuellette, and Frank Rudzicz. Bendr: using transformers and contrastive self-supervised learning task to learn from massive amounts of eeg data. Fronliers in Human Neuroscience, 15:653659, 2021.
Masanori Koyama and Shoichiro Yamaguchi. When invariance useful in an out-of-distribution gencralization problem? arXiv preprint arx:::0008.0 01883, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenct classification with deep convolutional neural networks. Advances in newral information processing systems, 25, 2012.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex), 2021.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequenceeto-ssqquence recurrent networks. In International conference on machine learning, pp. 1873-2882 PMLR, 2018.
Vernon J Lawhern, Amelia J Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance. Fegnet: compaci convolutional neural network for eeg-based brain compu-c interfaces. Journal of neural engineering, 1555)1056013, 2018.
Angeliki Lazaridou, Adhi Kuncoro, Flena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomas Kocisky, Sebastian Ruder, al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 34, 2021.
Min-Ho Lee, O-Yeon Kwon, Yong-Jeong Kim, Hong-Kyung Kim, Young- Lee, John Williamson, Siamac Fazli, and Seong-Wiin Whan Lee. Eeg dataset and openbmi toolbox for three bei paradigms: an investigation into bei illiteracy. GigaScience, 8(5):giz002, 2019.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Out-of-distribution generalization on graphs: A survey. Xiv preprint arXiv:2202.00900 2022.
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A, (7992194) 2021.
Evan Z Liu, Behzad Haghgoo, Annie Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781 6792. PMLR, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730 3738, 2015.
Chaochao Lu, Yuhuai Wu, Jose Miguel Hernández=Looatoo and Bernhard Schölkopf. Nonlinear invariant risk mininization: A causal approach. arXiv preprint ar Xio:2102. 1 12353, 2021.
Wang Lu, Jindong Wang, Xinwei Sun, Yiqiang Chen, and Xing Xie. Out-of-distribution representation learning for time series classification. In The Fleventh International Conference on Learning Representations, 2023.
Qianli Ma, Zhen Liu, Zhenjing Zheng, Ziyang Huang, Siying Zhu, Zhongzhong Yu, and James T Kwok. A survey on time-series pre-trained models. Xixiv preprint arXio:2305. 10716, 2023a.
Qianli Ma, Zhen Liu, Zhenjing Zheng, Ziyang Huang, Siying Zhu, Zhongzhong Yu, and James T Kwok. A survey time-series pre-trained models. Xr preprint arXio:2305.1077 107161 2023b.
18

---

Published in Transactions on Machine Learning Research (08/2023)
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associalion for computational linguistics: Muman language technologies, pp. 142 150, 2011.
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. Dialoguernn: An attentive run for emotion detection in conversations. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6818 6825, 2019.
Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark JF Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, et al. Shifts: A dataset of real distributional shift across multiple large-scale tasks. arXiv preprint arXiv:2107.07455, 2021.
Itzik Malkiel, Gony Rosenman, Lior Wolf, and Talma Hendler. Self-supervised transformers for fmri representation. International Conference on Medical Imaging with Deep Learning, pp. 895-913 913. PMT.R, 2022.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Imago- based recommendations styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43 02, 2015.
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between distri-iistiibution and in-distribution gencralization. In Marina Meila and Tong Zhang (eds.), Proceed- ings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Reseorch, pp. 7721 7735. PMLR, 18 24 Jul 2021. URL httpp::/pproeeedings mlr press/v139/ mi ailler2bb.0m html
Manfred Mudelsee. Trend analysis of climate time series: A review of methods. Earth-science reviews, 190: 1.00-22,,20 2019.
Jens Müller, Robert Schmier, Lynton Ardizzone, Carsten Rother, and Ulrrich Köthe. Learning robust. models using the principle of independent causal mechanisms. In DAGM German Conference on Pattern Recognition, pp. 79-110. Springer, 2021.
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. arXiv preprint 0rXiv:2009.00330 2020.
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669 688, 1995.
Judea Pearl. Causality. Cambridge university press, 2009.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference compuler 2019.
vision, pp.
Jonas Peters, Peter Bühhmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 785444147- 2016.
Mohammad Pezeshki, Sékou-Oumat Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks, 2021.
Stephen R Pfohl, Haoran Zhang, Yizhe Xu, Agata Foryciarz, Marzyeh Ghassemi, and Nigam H Shah. A comparison of approaches to improve worst-case predictive model performance over patient subpopulations. Scientific reports, 12(1):1 13, 2022.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateck Vij. A deeper look into sarcastic tweets using deep convolutional neural networks. Xix preprint or Xin 1610. 0.08815, 2016.
19

---

Published in Transactions on Machine Learning Research (08/2023)
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihal- cea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. ar Xiviv preprinl arXiv:1810.02508, 2018.
Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. Emotion recognition in conversation:
Research challenges, datasets, and recent advances. IEEE Access, 7:100943 100953, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 18 24 Jul 2021a. URL httpp::/pproceedngss press/v139/radford221 htmm
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXie:2103.0000 2021b.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.14444 2021.
Mohamed Ragab, Zhenghua Chen, Wenyu Zhang, Emadeldeen Eldele, Min Wu, Chee-Keong Kwoh, and Xiaoli Li. Conditional contrastive domain generalization for fault diagnosis. IEEE Transactions Instrumentation and Measurement, 71:1-12, 2022.
Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, al. Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine, 1(1):1 10, 2018.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of- distribution generalization. arXie preprint orXiv:2109. 029339 2021.
Muhammad Imran Razzak, Sacoda Naz, and Ahmad Zaib. Deep learning for medical image processing: Overview, challenges and the future. Classification in BioApps, pp. 323 350, 2018
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenct classifiers generalize to imagenel? In International Conference on Machine Learning, pp. 5389 -4000 PMLR, 2019.
Alexander Robey, George Pappas, and Hamed Hassani. Model-based domain generalization. Advances in Neural Information Processing Systems, 34, 2021.
Mateo Carulla,,,,,,, Bernhard Scholkopf, Richard Turner, and Jonas Peters. A causal perspective on domain adaptation. stat, 1050:19, 2015.
Mateo Rojas- Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1300-1342 2018.
Franco Ronchetti, Facundo Quiroga, César Armando Estrebou, Laura Cristina Lanzarini, and Alejandro Rosete. Lsa64: an argentinian sign language dataset. In XXII Congreso Argentino de Ciencias de la Computación (CACIC 2016)., 2016.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient. for out-of-distribution generalization. ar preprint XXXi:2222200656 2022.
Yangjun Ruan, Yann Dubois, and Chris J Maddison. Optimal representations for covariate shift. arXiv preprinl ar 0xxx:2221.00057 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2020.
20

---

Published in Transactions on Machine Learning Research (08/2023)
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, et al. Extending the wilds benchmark for unsupervised adaptation. arXiv preprint arXie:2112.060 05090, 2021.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breods: Benchmarks for subpopulation shift.
arXiv preprint orXiv:2008.0455 2020.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmect Kohli. Analysing mathematical reasoning abilities of neural models. arXiv preprint aiv: eetei.1440055500 2019.
Gerwin Schalk, Dennis J McFarland, Thilo Hinterberger, Niels Birbaumer, and Jonathan Wolpaw. Bci2000: general-purpose brain-computer interface (bei) system. IEEE Transactions biomedical engineering, 51 (6):1034 1043, 2004.
Robin Tibor Schirrmeister, Jost Tobias Springenberg. Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human Brain Mapping, aug 2017. ISSN 1097-0193. doi: 10.1002/hbm..25 23730. URL http://dxx.. doi wwg/oogg///.. 100200022hhm 23730
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal represenlation learning. Proceedings of the IEEE, 109(5):612 634, 2021.
Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. Financial time series forecasting with deep learning: A systematic literature review: 2005 2019. Applied soft compuling, 90:106181, 2020.
Soroosh Shahtalebi, Tean-Chhistoon Gagnon- Touraj Laleh, Mojtaba Faramarzi, Kartik Ahuja, and Trina Rish. Sand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization. aiv preprint arXiv:2106. 02266, 2021.
Hossein Sharifi-Noghabi, Parsa Alamzadeh Harjandi, Olga Zolotareva, Colin C Collins, and Martin Ester. Velodrome: Out-of-distribution generalization from labeled and unlabeled gene expression data for drug response prediction. bioRriv, 2021.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of- distribution generalization: A survey, arXiv preprint ar Xix:22108..1 13624, 2021.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7588) 2016.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 1631 1642, 2013.
Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM conference on embedded networked sensor systems, pp. 127 140, 2015.
David S Stoffer and Hernando Ombao. Special issuc on time series analysis in the biological sciences, 2012.
Eric V Strobl, Kun Zhang, and Shyam Visweswaran. Approximate kernel-based conditional independence tests for fast non- nom-maraettric causal discovery. Journal of Causal Inference, 7(1), 2019.
Baochen Sun and Kale Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443-40-400 450. Springer, 2016.
Akinori Tanaka, Akio Tomiya, and Köji Hashimoto. Deep Learning and Physics. Springer, 2021.
21

---

Published in Transactions on Machine Learning Research (08/2023)
Mario Giovanni Terzano, Liborio Parrino, Adriano Sherieri, Ronald Chervin, Sudhansu Chokroverty, Christian Guilleminalt, Max Hirshkowitz, Mark Mahowald, Harvey Moldolsky, Agostino Rosa, et al. Atlas, rules, and recording techniques for the scoring of cyclic alternating pattern (cap) in human sleep. Sleep medicine, 2(6):537- -533, 2001.
Eric J Topol. High-performance medicine: the convergence of human and artificial intelligence. Nature medicine, 25(1):44 56, 2019.
Antonio Torralba and Alexei A Efros. Unbiased look dataset bias. In CVPR 2011, pp- 1521-1528 TEFE, 2011.
José F Torres, Dalil Hadjout, Abderrazak Scbaa, Francisco Martnnzz-Alvarez, and Alicia Troncoso. Deep learning for time series forecasting: survey. Big Data, 9(1):3 21, 2021.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489-4444488 4489-4497, 2015.
Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interssience, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017a.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention all you need. Advances in nearra information processing systems, 30, 2017b.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pollern recognition, pp. 5018-5027, 2017.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust. global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseon domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
Simeon Warner. Open archives initiative protocol development and implementation at arxiv. orXiv preprint cs/0101027, 2001.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0100 2021.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXim:1704 00426. 2017.
Zhiyuan Wu, Ning Liu, Guodong Li, Xinyu Liu, Yue Wang, and Lin Zhang, A variational bayesian approach for fast adaptive air pollution prediction. In 2021 IEFE International Conference on Big Data (Big Dala), pp. 1748 1756. IEEE, 2021.
Kai Xiao, Logan Engstrom, Andrew Tyas, and Alcksander Madry. Noisc or signal: The role of image backgrounds in object recognition. arXiv preprint XrXiv:200.0098 09999, 2020.
Yilun Xu and Tommi Jaakkola. Learning representations that support robust transfer of predictors. arXiv preprint ar Sft:2110.09944 2021.
22

---

Published in Transactions on Machine Learning Research (08/2023)
Sijie Yang. Fei Zhu, Xinghong Ling, Quan Liu, and Peiyao Zhao. Intelligent health care: Applications of deep learning in computational medicine. Frontiers in Genelics, 12:444, 2021.
Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is hard: closer look subpopulation shift. arXiv preprint ar arx:22302. 12254, 2023.
Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards a theoretical framework of out-of-distribution generalization. Advances in Newral Information Processing Systems, 34, 2021a.
Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li. Ood-bench: Benchmarking and underslanding out-of-distribution generalization datasets and algorithms, 2021b.
Rui Ye and Qun Dai. A relationship-aligned transfer learning algorithm for time series forecasting. Information Sciences, 593:17 34, 2022.
Guojun Zhang, Han Zhao, Yaoliang Yu, and Pascal Poupart. Quantifying and improving transferability domain generalization. Advances in Neural Information Processing Systems, 15:10957-10977 2021a.
Haoran Zhang, Natalie Dullerud, Laleh Seyyed-Kalantari Quaid Morris, Shalmali Joshi, and Marzyeh Ghassemi. An empirical framework for domain generalization in clinical settings. In Proceedings of the Conference on Health, Inference, and Learning, pp. 279 290, 2021b.
Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence lest and application in causal discovery. arXiv preprinl trXii:1202.377 2012.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 128855810.12885, 2018.
23

---

Published in Transactions on Machine Learning Research (08/2023)
Ethical concern address
Comments on personally identifiable information or sensitive personally identifiable informa- tion
EEG datasets (CAP, SEDFx and PCL) The dataset does not hold personally identifiable informa- tion.
HHAR: The dataset does not hold personally identifiable information.
LSA64 The dataset holds videos of signers, which might be considered as personally identifiable information, However, because of the naturs of the data gathering process, i.e. deliberate head shot of the signers and that the dataset is openly distributed by the author at http://facundog gi thub to/datasees/lsa64/ we can assume that participants gave permission to distribute their videos. Efforts were made speak with Ronchetti al. (2016) on the subject, but we were not able make contact.
AusElec The dataset does not hold personally identifiable information.
Pedestrian The dataset does not hold personally identifiable information.
IEMOCAP The dataset holds videos and voice recording of actors, which might be considered as personally identifiable information. However, is reasonable assume that actors gave consent the data gathering because of the nature of the data, i.e., acted dialogues by actors. In any case, the data is protected by release and will not be distributed by to protect property and identity of actors.
Comments on consent use or share the data
EEG datasets (CAP, SEDFx and PCL) the datasets were accessed through open forums of dataset. (Physionet (Goldberger et al. 2000 for CAP and SEDFx, and MOABB (Jayaram Barachant 2018) for PCL) and thus we can assuine that we have consent to use. Additionally, licenses allows us to use and distribute derived products of the data.
HHAR was accessed through the UCI Dua & Grall 2017 open forums and thus we assume that. we have consent to use.
LSA64 is openly distributed by the author at http:///acuuooggiittto//////att
which allows us to assume that we have consent to use. The dataset has a Attribution Commerciaal Ahare.llike 4.0 International license which gives us permission to distribute derived products. Efforts were made to speak with Ronchetti et al. (2016) to confirm assumptions on the subject, but we were not able to make contact.
AusElec was accessed through the Monash time series archive (Godahewa et al. direct consent from Godahewa et al. (2021) to use the dataset in our work.
2021 We obtained
Pedestrian accessed through the Monash time series archive (Godahewa et al. 2021 obtained direct consent from Godahewa et al. (2021 to use the dataset in work.
We
IEMOCAP: We obtained direct consent to use under the license agreement on their website /https://sail usc. edu/iemocap/). However, we will not ourselves be distributing this dataset, users will need to sign the release form and obtain the dataset themselves from SAIL at USC hhttps:/saall wtw.aaallls.eeei.....o eds/) in order to use in WOODS. This is to protect property and identity of actors.
24

---

Published in Transactions on Machine Learning Research (08/2023)
A Organization
In Appendix B we provide additional related works, including works in both OOD gencralization algorithms and existing datasets in the field. In Appendix C we provide further details on all WOODS datasets, along with model architecture choices and licenses.
Appendix D we provide a general formulation for OOD generalization algorithms adaptation to time series, along with explicit penalty value function definitions for we give further details on how we define the generalization gaps in our datasets. In Appendix F we describe our evaluation framework. In Appendix G model selection strategies used in this work.
the algorithms used this work. In Appendix E
we discuss the
B Related works
In the main text, we covered important benchmarks in the field of OOD generalization. In this section, we detail broader horizon of datasets in the field along with OOD generalization algorithms.
B.1 OOD generalization algorithms
Several algorithms were recently proposed to address the OOD generalization failures of deep learning (Arjovsky
et al.
2020
Krueger et al.) 2021 Pezeshki et al. 2021 Ahuja et al. 2021 Sagawa et al.
2020
Parascandolo
et al.,
2020;
Shahtalebi et al.,
2021 Koyama & Yamaguchi 2020
Robey et al.
2021
Ruan et al.) 2021
Rame et al. 2021 Ahuja et al. 2020a Xu & Jaakkola, 2021 Müller et al. 2021
Liu et al.
2021
Lu et al.
2021 Rosenfeld et al. 2022; Chen et al. 2022 Sharifi-Noghale et al.) 2021
Ragab et al.
2022)
Several
of these algorithms adopt the invariance principle from causality (Pearl 2009
1995 Peters et al.
2016 to
creale predictors that rely on the causes of the label to make predictions. Invariance leveraged because
it is more flexible and scalable alternative conditional independence testing typically used for causal discovery (Zhang et al.; 2012; Strobl ct al.] 2019). An optimal predictor that relies the cause will be min-max optinal (Ahuja et al.) 2020b Müller et al. 2021 Rojas-Carulla et al. 2018) under a large class of distribution shifts. Some works have also been proposed to address the distribution shift that arises through
time in time series forecasting tasks (Du et al.
2021
Wu et al. 2021
Ye & Dai 2022). Other works look at
representation learning for time series OOD generalization (Lu ct al. 2023
Ma ct al. 2023b).
B.2 Existing benchmarks for OOD generalization
Synthetic datasets Many synthetic and semi-synthetic datasets were created to gain a better understanding of generalization failure in deep learning, e.g., CMNIST (Arjovsky et al. 2020) investigates our motivating cow or camel classification problem, RMNIST (Ghifary et al. 2015) investigates invariance with respect to rotation
of images, and Invariance Unit Tests (Aubin al. 2021 investigates six different types of distribution shifts for linear models.
Image datasets Many real (i.c., non-synthetic) image datasets were proposed, some with naturally occurring distribution shifts and some with artificially induced distribution shifts. Several of these datasets are composed of different renditions of the same underlying labels, e.g., PACS Li et al. 2017 (Photo, Art, Cartoon, Sketch), DomainNet (Peng et al. 2019) (Clipart, Infographic, Painting, Quickdraw, Photo, Sketch), Office-Home (Venkateswara et al. 2017) (Art, Clipart, Product, Photo), and ImageNet-R. (Hendrycks HuaggeeeeRR.Hendryyds et al. 2 (art, cartoons, graffiti, embroidery). Others focus on the generalization across different datasets with same rendition, such as many altered versions of ImageNet, e.g., ImageNet-A (Hendrycks et al. 2021b comprises of ImageNet images that are missclassified by ResNet models, ImageNet-Ci (Hendrycks & Dietterich
2019 comprises algorithmically corrupted images from the original ImageNet, ImageNNet-Sketch (Wang et al,
2019 comprises samples through Google Image queries, ImageNet-V2 (Recht et al., 2019) compriscs similar
images to ImageNet collected by closely following the original labeling protocol, BREEDS (Santurkar et al.) 2020) comprises of ImageNet subclasses that are held out during training. Others created datasets similar renditions but different sources, c.g., VLCS (Torralba Efros 2011) comprises images from four different. photo datasets, ObjectNet (Barbu et al.] 2019) comprises images from different predefined viewpoints, Terra Incognita. (Beery al. 2018) comprises images from multiple different traps. Another dataset class has
25

---

Published in Transactions on Machine Learning Research (08/2023)
strong spurious features that create shortcuts to minimize the empirical risk, e.g., in CelebA Liu et al. 2015) hair color a spurious attribute to gender classification task, while in NICO He et al. 2019 Waterbirds (Sagawa et al. 2020) and backgrounds challenge (Xiao et al. 2020 use the background as a spurious attribute of animal classification task. Finally, some other datasets were created to study specific problems, e.g,, 15mft15m (Kimura et al. 2021) that looks al OOD generalization the large data regime.
Language datasets Natural language is prone to distribution shifts because of interindividual variability, consequently, many works investigated OOD generalization in language. The Machine Translation dataset: from the work of Malinin et al. (2021) investigates generalization to atypical language usage in a translation task. Csordás et al. (2021) explored the systematic generalization of transformers with five datasets, ie., SCAN (Lake & Baroni] 2018) uses splits of different sentence lengths, CFQ (Keysers et al. 2019) uses splits of different lext structures, PCFG (Hupkes et. al. 2020) uses different split definitions to investigate different aspects of generalization, COGS (Kim & Linzen 2020 uses splits that can be addressed with compositional generalization, and the Mathematics dataset (Sxxoon et al. 2019) uses extrapolation sets to measure gencralization. Hendrycks et al. (2020) showed that pretrained transformers help OOD generalization compared to other language models. They use three imett analysis datasets, i.e., generalization between SST-2 (Socher et al. 2013)) and 1MDb (Maas et al.,
2011], ), the Yelp Review dataset with food types as domains, the Amazon Review dataset (McAuley ct al., 2015 He & McAAuley] 2016) with domains composod of clothing categories. They also used three reading comprehension datasets, i..., STS-B (Cer et al. 2017) has text of different genres (news and captions), ReCoRD (Zhang al.) 2018) has news paragraphs from different news sources (CNN and Daily Mail), and MNLI (Williams et al. 2017) has text from differently communicated interactions such as transcribed telephone and face-to-face conversations.
Temporal datasets Some works looked at temporal distribution shifts in different settings. In natural language procossing, [azaridou et al. (2021) investigated the ability of language models to generalize to future utterances beyond their training period on the WMT (Barrault et al., 2019) and ArXiv (Warner 2001 datasets. In the clinical setting, both ZZhng et al. (2211)) and Guo et al. (022 investigated shifts when
data is grouped according to the year in which they were gathered: the former used in-hospital mortality
records and X-rays of the lungs, while the later used patients health record in the ICU. Malinin et al. 2021 investigated temporal shifts in large amounts of weather data.
Other modalities from Li et al. (2022) and the OGB-MoIPCBA (Koh et al. 2021 dataset adapted from the Open Graph Benchmark (Hu et al. 2020).
There have been efforts in studying OOD generalization on graphs, such as works
As mentionned in Section multiple works focused on gathering and standardizing datasets for unified measure of OOD generalization algorithm performance. Gulrajani & Lopez-Paz (2020) introduced DomainBed: collection of seven image datasets (i.e., CMNIST, RMNIST, PACS, VLCS, Office-Home, Terra Incognita, DomainNet) for systematic OOD performance evaluation of algorithms. Ye et al. (2021b) built on top of DomainBed and added three datasets (i.c., C to group the datasets according to their distribution shift. Koh el al. (2021) introduced WILDS: a benchmark of several new in-the-wild distribution shifts datasets across diverse data modalities, i.e., IWildCam2020- WILDS, Camelyon17-WILDS RxRxI-WIIDDS, OGB-MoICCBA, GlobalWheat WI.D,, CivilComments- WILDS, FMoW-WILDS, PovertyMap- WILDS, Amazon-WILDS and Py150 WILDS. WILDS was recently extended with unlabeled samples for multiple of its datasets (Sagawa et al.
NICO, and CelebA), along with mcasure
2021
26

---

Published in Transactions on Machine Learning Research (08/2023)
C Additional dataset information
C.1 Spurious-Fond Fouurer
(a) Spurious
Invariant (b)
HALLL
HA+LB
LA
HA
H B
B
A
A
A
or
or
{10, 80, 90}% correlation
75%
freq
time
correlation
Figure 13: Description of the Spurious- Spurious-Fourier dataset. Signals have one low-frequency peak and one high-frequency peak. They are then constructed from the Fourier spectrum with an inverse Fourier transform. (b) Examples of reconstructed signals, both signals have the same high frequency, but different low frequencies, which are hard to distinguish visually.
C.1.1 Setup
Motivation Recall the cow or camel classification problem from Section D where deep learning model trained distinguish cows from camels learns rely on the background properties (e.g.. grass or sand) instead of the animal characteristic features (e.g., color) to make a prediction. Arjovsky al. (2020) proposed Colored MNIST (CMNIST) to recreate the the cow camel classification problem into a simple benchmark in the image domain. We propose the Spurious-Fourier dataset which is an adaptation of the cow camel classification problem to time series.
Problem setting We create a dataset composed of one-dimensional signals, where the task is to perform binary classification based on the frequency characteristics. Signals are constructed from Fourier spectra with one low-frequency peak (LA- 2Hz or LB 4Hzz) and one high-frequency peak (Hl 7Hz or Hi 9Hz), see Figure 13 Domains (10%, 80%, 90%) contain signal-label pairs, where the label is noisy function of the low- and high-frequencier such that cww-frquency peaks bear varying correlation of d with the label and high-frequency peaks bear an invariant correlation of 75% with the label.
Data We first create four Fourier spectra with all combinations low- and high-frequency peaks. From each of the spectra, we perform an inverse Fourier transform to get a 1 dimensional signal of 100 seconds sampled at 100Hz. We then split this long signal into smaller overlapping sequences of 50 timc-steps, i.c., half a second. We then recreate the Colored MNIST (Arjovsky et al., (2200) dataset characteristie. We build datasets d by repeating the following protocol 4000 times. First, sample y from Bernoulli distribution = 0.5. Second, we obtain by flipping with probability of 25%, this gives us our high-frequency component (ū = 0 - HA, 5 = 1 HB). Third, we sample from a Bernoulli distribution of parameter = d, this gives us our low-frequency component. (z = 0 + LA, = + Ln). Finally, we add to the domains dataset Dd a random signal of configuration + h with the label ÿ.
Domain information Table details the distribution of labels for every domain in the Spurious- Fourier dataset.
27

---

Published in Transactions on Machine Learning Research (08/2023)
Table 7: Distribution labels for every domain in the Spurious-Fourd Fourier dataset
Domain
7Hz
9Hz
Total
10%
2043
1957
4000
80%
2013
1987
4000
90%
1991
2009
4000
Total
6047
5953
12000
Architecture choice For this simple task, we use the LSTM (Hochreiter & Schmidhuber 1997| model because it is simple model well accepted in the time series /squuntial prediction field. We stack on top of the LSTM a fully connected (FC) layer used to make predictions at the last time step of the time series. Layers are detailed in Table
Table 8: Model architecture used for the Spurious- Fourier dataset
#
Layer
15
LSTM(in=1, hidden 2idden_sizo==0, num layors-2)
16
Linear(in=20, oul=20)
17
ReLU
18
Lincar(in=20, out=2)
C.1.2 Detailed results
Oracle task Investigating the impact of spurious correlation in a dataset is meaningless if the underlying invariant task is impossible to solve with given model hyperparameter configuration. In order to avoid this, we provide the Basic-Fourier dataset in the WOODS repository. It consists of the oracle task of the Spurious-Fourie dataset, i.c., classifying 7Hz and 9Hz signals with no label noise or spurious features. We create two Fourier spectra with 7Hz and 9Hz frequency peaks respectively. From both of the spectra, we perform an inverse Fourier transform to get a one-dimensional signal of 100 seconds sampled at 100Hz. We then split this long signal into smaller overlapping sequences of 50 time-steps, i.e., half a second. While this is a domain generalization task, the Basic-Fourier dataset is included in the WOODS repository as a sanity check that the underlying invariant task of the Fourier dataset is possible with the model and hyperparameter configuration using. We show the results of ERM the Basic-Fouris datasct in Table g
Table 9: Result for the Basic-Fourier dataset.
Objective
Performance
ERM
100.00 (0.00)
ID evaluation We evaluate the performance of ERM with access to all domains Dd (1l%e(100..00 10%...0%%000.................... 90%)- We oblain these results by doing hyperparameler search with the methodology detailed in Appendix F with no held-out test domain and choose the model with train-domain validation. In other words, the training is done with all domains; thus, all domains are ID. The columns correspond the validation accuracy of the chosen model in each domain. We see that the model learns the invariant solution the task because the high frequencies (75%) are stronger predictor of the label than the low frequencies (60%).
28

---

Published in Transactions on Machine Learning Research (08/2023)
Table 10: ID results for whhe Spurious-Fourie dataset
Algorithm
10%
80%
90%
Average
ID ERM
74.46 (0.07) 74.79 (0.03) 73.54 (0.07)
74.26
Benchmark results We present the detailed evaluation of OOD generalization algorithms on the Spurious- Fourier dataset. Important note: We evaluate performance only when holding out the 10% domain as it is the only domain of meaning, and including the other domains only dilutes the information carried by this datasct.
Table 11: OOD gencralization algorithms performance on the Spurious-Fourts datasct
Train-domain validation
Test-domain validation
Objective
10%
Objective
10%
ERM
9.91 (0.12)
ERM
12.07 (1.99)
TRM
10.30 (0.09)
IRM
58.82 (1.98)
VREx
10.36 (0.23)
VREx
63.69 (0.70)
GroupDRO
10.06 (0.19)
GroupDRO
21.49 (2.12)
IB-ERM
9.21 (0.31)
IB-ERM
18.65 (4.01)
SD
9.67 (0.20)
SD
9.97 (0.11)
C.2 Temporal Colored MNIST with source domains
(a)
(b)
d=10% d=80% d=90%
1
2
4
3 Y==ODD (4+3=7)
No label
Y2=ODD (1+2=3)
Yz=EVEN (2+4=6)
1
2
3
Figure Description of the Temporal Colored MNIST dataset with source domains. (a) Data samples are videos of four colored MNIST digits where the task is to predict whether the sum of the current and previous digits in the sequence is odd or even. (b) Spuriously correlated color is added to each digit such that the correlation is constant among the frames of a video, but varies between video from different domains d E (10% 80%, 90%).
C.2.1 Setup
Motivation
Arjovsky et al. (2020) proposed the CMNIST dataset synthetic investigation of the cow camel classification problem. We propose an extension of this widely used dataset to time series to investigate both domain definition paradigms presented in Section 2.2 Source-domains (Example 2.2h and Time-domains (Example 2.3 In this section, we give more details on the Source-domain formulation of the the dataset.
Problem sctting In Temporal Colored MNIST with source domains (TCMN))))Source)) we create a binary classification task of video frames. Videos are sequences of four colored MNIST digits where the goal is to predict whether the sum of the current and previous digits in the sequence is odd even, see
29

---

Published in Transactions on Machine Learning Research (08/2023)
Figure 14(a)) aa.. Prediction is made for all frames except for the first one. The label is noisy function of the digit and color, such that the color bears varying correlation of d with the label of the frame, and the digit. sums bears an invariant correlation of 75% with the label of the frame. Domains are created such that the color correlation is constant among the frames of a vidco, but varies between video from different domains (10%; 80%, 90%]. The domain definition is depicted Figure 14
Data We create videos by concalenating four digits logether and attributing labels y to the second, third and fourth frames following the parity task, see Figure [4(a). For every labeled frame i in a sequence from the domain d f10%, 80%, 90%), define the final label of that frame y; by flipping the label y, with probability of 25%. Second, we define as y; flipped with probability equals to the domain definition (10%, 80% or 90%). Finally, we color the digit red if 2 = 0 or green if 2 == 1.
Domain information Table 12 details the distribution of labels for every domain in the TCMNIST-SSurce dataset.
Table 12: Distribution of labels for every domain in the TCMNIST-Souree datasct
Domain
Even
Odd
Domain Total
10%
8603
8899
17502
80%
8583
8916
17499
90%
8563
8936
17499
Total
25749
26751
52500
Architecture choice For this task, we use a combination of CNN and an LSTM architecture. Table 18
details the layers of the model architecture. Its parameters were hand tuned to perform well on this task.
Table 13: Model architecture used for the TCMNIST-Souree dataset
# Layer
Conv2D out=8, padding-1)
ReLU
Conv2D(in=8, out=32, stride-2, padding-1)
ReLU
MaxPool2d
Conv2D(in=32, out=32, padding-1)
7
ReLU
MaxPool2d
(in=2 out=32, padding
10
ReLU
Linear(in=288, out=64)
ReLU 12
13
Linear(in=64, out-332))
14
ReLU
15
LSTM(in=32, hidden -iz-=18_ num layers=1)
16
Linear(in=128, out-64)
17
ReLU
16
Linear(in=64, out=64)
17
ReLU
18
Linear(in=64, out=2)
30

---

Published in Transactions on Machine Learning Research (08/2023)
C.2.2 Detailed results
Oracle task Investigating the impact of spurious correlation in a dataset is meaningless if the underlying invariant task impossible solve with a given model or hyperparameter configuration. In order to avoid this, we provide the Temporal MNIST (TMNIST) dataset in the WOODS repository. It consists of the oracle task of the TCMNIST-Souree dataset, i..., classifying whether the sum of the current and last digit is odd even without label noise and without spurious features. We create videos by concatenating four digits together and attributing labels y to the second, third and fourth frames following the parity task. While this is not a domain generalization task, the TMNIST dataset is included in the WOODS repository as a sanity check that the underlying invariant task the Spurious- Spurious-Fourier dataset is possible with the model and hyperparameter configuration we are using. We show the results of ERM on the TMNIST dataset Table 14
Table 14: Result for the TMNIST dataset
Objective
Performance
ERM
98.77 (0.02)
We obtain these results by doing hyperparameter search with the methodology detailed in Appendix F with no held-out test domain and choose the model with train-domain validation. In other words, the training is done with all domains; thus, all domains are ID. The columns correspond to the validation accuracy of the chosen model in each domain.
ID cvaluation We show the ID results of ERM for TCMNIST-Souree in Table 15
Table 15: ID results for the TCMNIST-Souree dataset
Algorithm
10%
80%
90%
Average
ID ERM
68.36 (0.13) 73.49 (0.13) 74.85 (0.16)
72.23
Benchmark results in Table [6]
We show the detailed benchmark results of the adapted OOD generalization algorithms
Table 16: OOD generalization algorithms performance the TCMNIST-Source dataset
Train-domain validation
Test-domain validation
Objective
10%
Objective
10%
ERM
10.07 (0.02)
ERM
30.34 (0.82)
IRM
9.82 (0.13)
IRM
52.74 (0.59)
VREx
9.83 (0.22)
VREx
49.69 (0.25)
GroupDRO
10.39 (0.02)
GroupDRO
33.52 (2.95)
IB-ERM
9.97 (0.07)
IB-ERM
28.12 (1.12)
SD
10.24 (0.12)
SD
27.35 (3.51)
31

---

Published in Transactions on Machine Learning Research (08/2023)
C.3
Temporal colored MNIST with time domains
(a)
(b)
1
2
4
No label
Y2=ODD
Y3=EVEN
YD=ODD (4+3=7)
(1+2=3)
(2+4=6)
3
7
d=90% d=80% d=10%
Figure 15: Description of the Temporal Colored MNIST dataset with time domains. (a) Data samples are videos of Tour colored MNIST digits where the task to predict whether the sum of the current and previous digits in the sequence is odd or even. (b) Spuriously correlated color is added to each digit such that the correlation varies across frames. However, videos all have the same sequence of color correlation, where the first labeled frame correlation is 90%, second is 80% and third is 10%.
C.3.1 Setup
Motivation Arjovsky et al. (2020) proposed the CMNIST dataset synthetic investigation of the
camel classification problem. We propose an extension of this widely used dataset to time series to investigate both domain definition paradigms presented in Section 2.2 Source-domains (Example 2.2 and Timo-domains (Example 2.3 In this section, we give more details on the Tine-domain formulation of the the dataset.
Problem setting Temporal Colored MNIST with time domains (TCMNIST-Time)) we create a binary classification task of video frames. Videos are sequences of four colored MNIST digits where the goal is predict whether the of the current and previous digits in the sequence is odd even, see Figure 15 a). Prediction is made for all frames except for the first one. The label is noisy function of the digit and color, such that the color bears varying correlation of with the label of the frame, and the digit sums bears an invariant correlation of 75% with the label of the frame. Domains are created such that the color correlation varies across frames. However, vidcos all have the same sequence of color correlation, where the first labeled frame correlation is 90%, second is 80% and third 10%. The domain definition is depicted in Figure 15 b).
Data We create videos by concatenating four digits together and attributing labels y to the second, third and fourth frames following the parity task, Figure 15 a). For every labeled frame i 3, 4] of all videos in the dataset, we define the final label of that frame y; by flipping the label with probability of 25%. Second, we define as y, flipped with a probability equals to the domain definition for that frame index (i = 2 + 90%, i = 3 + 80% or i = + 10%). Finally, we color the digit red if 2 = 0 or green if 2 = 1.
Domain information Table 17 details the distribution of labels for every domain in the TCMNIST-Time datasct.
Table 17: Distribution of labels for every domain in the TCMNIST-Time dataset.
Domain
Even
Odd
Domain Total
10%
8564
8936
17500
80%
8765
8735
17500
90%
8613
8887
17500
Total
25942
26558
52500
32

---

Published in Transactions on Machine Learning Research (08/2023)
Architecture choice For this task, we use combination of CNN and LSTM architecture. Table 18
details the layers of the model architecture. Its parameters were hand tuned to perform well on this toy task.
Table 18: Model architecture used for the TCMNIST-Time dataset
#
Layer
Cinv2D out=8, padding-1)
ReLU
Conv2D(in=8, out=32, stride=2, padding=1)
ReLU
MaxPool2d
Conv2D(nn=32, out=32, padding=1)
7
ReLU
MaxPool2d
(in=22 out=32, padding=1)
10
ReLU
11
Linear(in=288, out=64)
12
ReLU
13
Lincar(in=64, out=333)
14
ReLU
15
LSTM(in=32, hidden size=128, num layers=1)
16
Lincar(in=128, out=64)
ReLU
Linear(in=64, oul=64)
17
ReLU
18
Lincar(in=64, out=2)
C.3.2 Detailed results
Oracle task Investigating the impact of spurious correlation a dataset is meaningless if the underlying invariant task impossible solve with a given model or hyperparameter configuration. In order to avoid this, we provide the Temporal MNIST (TMNIST) dataset in the WOODS repository. It consists of the oracle task of the SEMNIST-Time dataset, i.e., classifying whether the sum of the current and last digit is odd even without label noise and without spurious features. We create videos by concatenating four digits together and attributing labels y to the second, third and fourth frames following the parity task. While this is not a domain generalization task, the TMNIST dataset is included in the WOODS repository as a sanity check that the underlying invariant task the Spurious- Spurious-Fourier dataset is possible with the model and hyperparameter configuration we are using. We show the results of ERM on the TMNIST dataset Table [9
Table 19: Result for the TMNIST dataset
Objective
Performance
ERM
98.77 (0.02)
ID evaluation We show the ID results of ERM for TCMNIST-Time in Table 20 by doing a hyperparameter search with the methodology detailed Appendix F with no held-out test domain and choose the model with train-domain validation. In other words, the training is done with all domains; thus, all domains are ID. The columns correspond the validation accuracy of the chosen model in each domain.
We obtain these results
33

---

Published in Transactions on Machine Learning Research (08/2023)
Table 20: ID results for the TCMNIST-Time dataset
Algorithm
10%
80%
90%
Average
ID ERM
89.97 (0.00) 80.98 (0.02) 91.20 (0.00)
87.38
Benchmark results in Table 21
We show the detailed benchmark results of the adapted OOD generalization algorithms
Table 21: OOD generalization algorithms performance the TCMNIST-Time dataset
Train-domain validation
Test-domain validation
Objective
10%
Objective
10%
ERM
9.88 (0.11)
ERM
28.61 (2.41)
GroupDRO
10.09 (0.14)
GroupDRO
24.85 (3.91)
IB-ERM
10.19 (0.04)
IB-ERM
33.70 (6.49)
IRM
10.27 (0.05)
IRM
50.65 (0.17)
SD
10.19 (0.14)
SD
31.76 (5.15)
VREx
9.80 (0.06)
VREx
50.57 (0.59)
C.4 CAP
(a)
(b)
Machine C
Z
Machine B
Machine E
Machine A
Machine D
Figure 16: Summary of the CAP dataset. (a) The task is to perform sleep stage classification from EEG measurements. (b) The dataset has five source domains, where each domain contains data gathered with a different machine. The goal is generalize to unseen machines.
C.4.1 Setup
Motivation A recurrent problem in computational medicine is that models trained on data from given recording device will not generalize to data coming from another device, even when both devices are from a similar equipment provider. Failure to generalize to unseen machines can cause critical issues for clinical practice because a false sense of confidence in a model could load to a false diagnosis (Kim et al. 2018 Engemann et al. 2018). We study these machinery- distribution shifts with the CAP (Terzano et al. 2001 Goldberger et al. 2000) dataset (Figure
Problem setting We consider the sleep stage classification task from electoeencephallggapiic (EEG) measurements. The dataset has five source domains, where each domain contains data gathered with a different machine. The goal is to generalize to unseen machines.
Data The dataset is composed of 300 gathered on 41 participants. Each participants had one night of sleep recorded. The inputs X recordings of 30 seconds each with 19 channels sampled at 100Hz. The
34

---

Published in Transactions on Machine Learning Research (08/2023)
channels include EEG but also include Electromyographhy (EMG), Electrocardiography (ECG), and heart rate measurements. The labels Y consist of 6 sleep stages: Awake, Non-REM 1, Non-REM 2, Non-REM 3, Non-REM 4, and REM. The domains d are the 5 EEG machines: Machine A, Machine B, Machine C, Machine D, and Machine E.
Preprocessing This section details the preprocessing steps taken for the CAP dataset. The raw CAP dataset contains data from 15 machines, each with different channels and sampling frequency characteristies We only use recordings from the five machines with the most data. Removing machines with less data allows retain a reasonable number (19) of shared channels between them. Next, we resample the data to a standard sampling frequency of 100Hz for all five machines. We then apply bandpass filter from 0.3Hz to 30Hz. This bandpass filter removes frequency bands generally considered uninformative for sleep stage classification. Next, we split the nights of sleep into sequences of 30 seconds for training and testing. Finally, we then detrend and normalize the 30 second recordings with a standard scaler applied to the channels individually.
Domain information Table 22 details the number of participants domain and some demographic information; each had a single night of sleep recorded. Table 23 details the proportion of samples and labels across domains.
Table 22: Number of participants and demographic information of the CAP dataset
Domain
Number of participants
Male
Female
Age
Machine A
33.1 + 13.9
Machine B
26.4 + 8.2
Machine C
73.4 +: 6.42
Machine D
10
5
30.7 (8.9
Machine E
3
5
36.8 + 16.7
Total
24
17
37.3 + 18.1
Table 23: Domain proportions labels in the CAP dataset
Domain
Awake
NREM 1
NREM 2
NREM 3
NREM 4
REM
Domain Total
Machine A
1448
350
4986
1533
2110
2342
12769
Machine B
318
171
1933
595
706
971
4694
Machine C
1318
294
1168
595
810
547
4732
Machine D
1114
580
3547
1273
1606
1810
9930
Machine E
967
276
3377
711
1251
1683
8265
Total
5165
1671
15011
4707
6483
7353
40390
Architecture choice For this dataset, we use deep convolution network model defined in work from |Schirmeister et al. (2017). We use the implementation of the BrainDecode (Schirrmeister et al. 2017 Toolbox. We chose this model because it is the perfect combination of performance stability and recogni- tion from the EEG community. The implementation is available at https://githun http://ghiiicccm..//ttrrrrreg/// braindecode
C.4.2 Detailed results
ID evaluation We show the results ERM for the CAP dataset in Table 24 We obtain these results by doing hyperparameter search with the methodology detailed in Appendix with no held-out test domain and choose the model with train-domain validation. In other words, the training is done with all domains;
35

---

Published in Transactions on Machine Learning Research (08/2023)
thus, all domains are ID. The columns correspond to the validation accuracy of the chosen model in each domain.
Table 24: ID) results for the CAP dataset.
Algorithm
Machine A
Machine B
Machine C Machine D
Machine E
Average
ID ERM
78.26 (0.52)
78.14 (1.00)
63.39 (1.38) 78.73 (0.34)
77.09 (0.37)
75.12
We show the detailed benchmark results of the adapted OOD generalization algorithms in Table 25 Each results is obtained by holding out one domain during training and reporting the performance of the chosen model from the hyperparameter sweep on that held out domain, more details in Appendix F
Benchmark results
Table 25: OOD generalization algorithms performance on the CAP dataset.
Train-domain validation
Objective
Machine
Machine
Machine
Machine
Machine
Average
ERM
68.93 (0.54)
61.98 (0.53)
40.10 (0.75)
73.10 (0.83)
70.13 (0.33)
62.85
IRM
67.59 (0.55)
48.40 (3.14)
41.01 (1.10)
69.52 (1.03)
66.86 (0.71)
58.68
VREx
57.97 (1.92)
38.96 (0.42)
33.81 (1.19)
52.53 (3.49)
59.71 (1.53)
48.60
GroupDRO
68.07 (0.33)
59.22 (1.53)
41.38 (0.52)
72.25 (0.70)
69.12 (0.90)
62.01
IB-ERM
70.20 (0.71)
62.03 (1.79)
40.66 (0.58)
72.73 (0.18)
70.57 (0.83)
63.24
SD
69.29 (0.25)
55.53 (1.45)
41.36 (1.78)
71.14 (0.22)
66.48 (0.92)
60.76
Oracle train-domain validation
Objective
Machine A
Machine B
Machine C
Machine D
Machine F,
Average
ERM
69.00 (0.51)
65.21 (1.38)
43.11 (0.30)
73.31 (0.67)
70.34 (0.16)
64.19
IRM
67.59 (0.55)
55.09 (2.08)
41.20 (1.19)
70.72 (0.46)
67.87 (0.26)
60.49
VREx
57.79 (2.06)
39.49 (0.65)
36.38 (0.57)
52.95 (3.15)
59.71 (1.53)
49.26
GRoupDRO
68.73 (0.22)
60.39 (1.23)
43.19 (0.56)
72.51 (0.49)
69.91 (0.50)
62.95
IB-ERM
71.06 (0.37)
66.99 (0.93)
43.21 (0.76)
73.64 (0.50)
70.88 (0.59)
65.16
SD
69.38 (0.20)
61.84 (0.39)
43.97 (1.09)
71.41 (0.14)
69.41 (0.29)
63.20
C.4.3 Credit and license
This dataset adapted from the work of Terzano et al. (2001). as made available on the online Phys- ionet (Goldberger al. 2000) platform. This dataset is licensed under the Open Data Commons Attribution License v1.0.
36

---

Published in Transactions on Machine Learning Research (08/2023)
C.5 SEDFx
(a)
(b)
Age 60-80
Age 80-100
Age 20-40
Age 40-60
Figure 17: Summary the SEDFx dataset. (a) The task is to perform sleep stage classification from EEG measurements. (b) The dataset has four source domains, where each domain contains data from participants of certain age group. The goal to generalize to unseen age groups.
C.5.1 Setup
Motivation In clinical settings, we train a model on the data gathered from limited number of patients and hope this model will generalize to new patients in the future (Pfohl et al., 2022). However, this generalization between observed patients in the training dataset and new patients not guaranteed. Distribution shifts caused by shifts in patient demographics (o.g., age, gender, and ethnicity) can cause the model to fail. We study age demographic shift with the SEDFx (Kemp al., 2000 Goldberger al. (0000) dataset (Figure 17
Problem setting We consider the sleep classification task from EEG measurements. The dataset has four source domains, where each domain contains data from participants of certain age group. The goal is to generalize to an unseen age demographic.
Data The dataset is composed of 238 712 recordings gathered on 100 participants. Every participant had 2 nights of sleep recorded. The inputs X are recordings of 30 seconds each with four EEG channels sampled at 100Hz. The channels include 2 EEG channels, one Electromyography (EOG) channel, and one Elcctrocardiography (ECG) channel. The labels Y consist of 6 sleop stages: Awake, Non-REM 1, Non-REM 2, Non-REM 3, Non-REM 4, and REM. The domains are the four disjoint age groups: Age 20-40, Age 40-60, Age 60-80, and age 80-100.
Preprocessing This section details the preprocessing steps taken for the SEDFx datasct. The raw SEDFx datasel. contains data from 2 machines with different channels and sampling frequency characteristics. We use the data from both machines and keep only the four channels they have in common. First, resample the data to a standard sampling frequency of 100Hz for both machines. We then apply bandpass filter from 0.3Hz to 30Hz. This bandpass filter removes frequency bands generally considered uninformative for sleep stage classification. Next, we crop the unlabeled onset and end of the complete recordings. Next, we split the nights of sleep into shorter sequences of 30 seconds for training and testing. Finally, we detrend the data and normalize the 30 second recordings with a slandard scaler applied to channels individually.
Domain information The data from the different machincs consists of data from disjoint sets of partici- pants. Table 27 details the number of participants per domain and some demographic information. Table details the proportion of samples and labels across domains.
37

---

Published in Transactions on Machine Learning Research (08/2023)
Table 26: Number of participants and demographic information of the SEDFX dataset
Domain
Number of participants
Male
Female
Age
Age 20-40
32
14
18
27.6 + 4.7
Age 40-60
29
12
17
53.3 + 3.4
Age 60-80
23
10
13
69.2 + 3,5
Age 80-100
16
8
90.5 (4.5
Total
100
56
54.7 22.6
Table 27: Domain proportions of labels the SEDFx datasct
Domain
Awake
NREM
NREM
NREM 3
NREM
REM
Domain Total
Age 20-40
10505
4222
28105
4830
4254
12348
64264
Age 40-60
20405
7182
27222
3243
1423
10007
69482
Age 60-80
14708
7087
19186
2830
1400
6917
52128
Age 80-100
25358
6684
14410
1288
186
4912
52838
Total
70976
25175
88923
12191
7263
34184
238712
Architecture choice For this dalaset, we use a deep convolution network model as defined work from Schirrmeister al. (2017). We use the implementation of the BrainDecode (Schirrmeister et al.) 2017 Toolbox. We chose this model because it is the perfect combination of performance, stability, and recogni- tion from the EEG community. The implementation is available at http:://gtthub.ccm///PPPPs. braindecode
C.5.2 Detailed results
ID evaluation We show the results of ERM for the SEDFx dataset in Table 28 doing a hyperparameter scarch with the methodology detailod in Appendix with no held-out test domain and choose the model with train-domain validation. other words, the training is done with all domains; thus, all domains are ID. The columns correspond the validation accuracy of the chosen model in each domain.
We obtain these results by
Table 28: ID results for the SEDFx datasct
Algorithm
Age 20-40
Age 40-60
Age 60-80
Age 80-100
Average
ID ERM
74.11 (0.12)
74.19 (0.47)
72.39 (0.41) 69.22 (0.49)
72.48
We show the detailed benchmark results of the adapted OOD generalization algorithms in Table 29 Fach results is obtained by holding out one domain during training and reporting the performance of the chosen model from the hyperparameter sweep on that held out domain, more details in Appendix F
Benchmark results
C.5.3 Credit and license
This dataset was adapted from the work of Kemp al. (2000). as made available on the online Phys- ionet (Goldberger al. 2000) platform. This dataset is licensed under the Open Data Commons Attribution license v1.0.
38

---

Published in Transactions on Machine Learning Research (08/2023)
Table 29: OOD generalization algorithms performance on the SEDFx dataset
Train-domain validation
Objective
Age 20-40
Age 40-60
Age 60-80
Age 80-100
Average
ERM
65.90 (2.06)
70.59 (0.54)
68.48 (0.17)
64.18 (0.33)
67.29
IRM
60.76 (0.94)
67.69 (0.80)
63.04 (0.81)
59.13 (0.43)
62.65
VREx
57.21 (2.39)
58.80 (0.94)
56.81 (0.97)
51.74 (1.23)
56.14
GroupDRO
67.01 (0.90)
68.31 (0.73)
65.08 (0.42)
60.48 (1.07)
65.22
IB-ERM
69.41 (0.12)
72.58 (0.46)
69.79 (0.42)
66.16 (0.87)
69.48
SD
69.87 (1.34)
73.18 (0.19)
69.14 (0.16)
67.18 (0.26)
69.84
Oracle train-domain validation
Objective
Age 20-40
Age 40-60
Age 60-80
Age 80-100
Average
ERM
69.87 (0.41)
71.03 (0.20)
68.88 (0.18)
64.36 (0.26)
68.53
IRM
66.02 (0.55)
67.69 (0.80)
63.16 (0.65)
60.14 (0.32)
64.25
VREx
59.63 (0.76)
58.80 (0.94)
56.78 (0.95)
52.73 (0.22)
56.99
GroupDRO
68.73 (0.37)
69.14 (0.40)
65.17 (0.35)
61.54 (0.85)
66.15
IB-ERM
70.42 (0.41)
72.79 (0.62)
70.25 (0.12)
69.08 (0.56)
70.64
SD
71.22 (0.60)
73.18 (0.19)
69.60 (0.04)
68.41 (0.60)
70.60
C.6 PCL
(a)
(b)
Lee2019_M MI
m or
m
Cho2017
PhysionetMI
Figure 18: Summary of the PCL dataset. (a) The task to perform motor imagery classification from EEG mcasurements. (b) The dataset has three source domains, where each domain contains dataset from different research group carrying out the same task. The goal is to generalize to unseen datasets of the same task.
C.6.1 Setup
Motivation Aside from changes in the recording device and shifts in patient demographics, human intervention in the data gathering process is another contributing factor to the distribution shift that lead to failure of clinical models (c.g., Camelyon17 (Koh et al. 2021 Sagawa et al. 2021)). This challenge is especially prevalent in temporal medical data (e.g., EEG, MEG. and others) because recording devices are complex tools greatly affected by nonlinear effects and modulations. These effects are often caused by context and preparations made before the recording (Engemann al. (018)). We study these procodural
shifts with the PCL (Lee al. 2019 dataset (Figure 18
Cho et al. 2017 Schalk al. 2004 Jayaram & Barachant 2018
39

---

Published in Transactions on Machine Learning Research (08/2023)
Problem setting We consider the motor imagery task from elecrrenceephalggappicc (EEG) measurements. The dataset has three source domains, where each domain contains a dataset from different research group carrying out the same task. The goal is to generalize to unseen data gathering processes.
Data The dataset is composed of 22 598 recordings gathered with 215 participants. The inputs X are recordings of three seconds each with 48 EEG channels sampled at 250Hz. The 48 channels contain only EEG measurements, The labels Y are two imagined movements: left hand and right hand. The domains d are three different motor imagery datasots: Schalk04 (Schalk al. 2004). Cho17 (Cho et al. 2017 and Lee19 (Lee al. 2019).
The 48 channels are: AF7, CP5, AF4, P4, P8, P2, FC6, Fz, C5, 01, Fpl, Fp2, F4, CP4, PO3, C1, FCI, T8, Pz, Oz, TP7, Cz, FC2, CP6, CP2, POz, PO4, C6, P7, AF3, FC4, TP8, CP1, 02, C2, F8, FC3, P3, AF8, FC5, F7, F3, T7, C4, CP3, CPz, C3, Pl. The channel locations are shown in Figure 19
Figure 19: International 10-10 system EEG channel labeling.
Preproccssing This section details the preprocessing steps taken for the PCL dataset. The raw PCL datasel contains data from 2 machines, Schalk04 (Schalk et al. 2004) and Chol7 (Cho et al. 2017) both used a BCI2000 system (Schalk et al. 20044) while Lee19 (Lee et al. 2019) used an undefined machine. Both machines have different channels and sampling frequency characteristics. We take only the channels they have in common and we resample the data a standard sampling frequency of 250Hz for both machines. We then apply bandpass filter from 0.3Hx to 30Hz. This bandpass filter removes frequency bands generally
40

---

Published in Transactions on Machine Learning Research (08/2023)
considered uninformative for the motor imagery task. Finally, then detrend the data and normalize the three second recordings with a standard scaler applied to the channels individually.
Domain information Table 30 details the number of participants per domain and some demographic information; we put N/A for unavailable demographic information. Table 31 details the proportion of samples and labels across domains.
Table 30: Number of participants and demographic information of the PCL dataset
Domain
Number of participants
Male
Female
Age
Schalk04
109
N/A
N/A
N/A
Cho2017
52
33
19
24.8 + 3.9
Lee19
54
29
25
[24, 35)
Total
215
N/A
N/A
N/A
Table 31: Domain proportions of labels in the PCL dataset
Domain
Left Hand
Right Hand
Domain Total
Schalk04
2480
2438
4918
Cho2017
4940
4940
9880
Lee19
3900
3900
7800
Total
11320
11278
22598
Architecture choice For this dataset, we use deep convolution network model as defined in work from Lawhern et al. (2018). We the implementation of the BrainDecode Schirrmeister et al. (2017) Toolbox. We chose this model because it is well recognized by the EEG community. It also has smaller architecture that better fits the data amount and task complexity of the PCL dataset. The implementation is available at http:////gggh.....o////eaaaarrrn wwt.:///TL/eeeburrrraaaaeeeeeedd
C.6.2 Detailed results
ID evaluation We show the results of ERM for the PCL dataset in Table 32 doing hyperparameter search with the methodology detailed in Appendix with no held-out test domain and choose the model with train-domain validation. other words, the training is done with all domains; thus, domains are ID. The columns correspond to the validation accuracy of the chosen model in each domain.
We obtain these results by
Table 32: ID results for the PCL dataset
Algorithm
Schalk04
Cho17
Lee19
Average
ID ERM
76.40 (0.19) 68.07 (0.09)
76.45 (0.23)
73.64
Benchmark results
We show the detailod benchmark results the adapted OOD generalization algorithms in Table 33 Each results is obtained by holding out one domain during training and reporting the performance of the chosen model from the hyperparameter sweep that held out domain, more details in A Appendix F
41

---

Published in Transactions on Machine Learning Research (08/2023)
Table 33: OOD generalization algorithms performance on the PCL dataset
Train-domain validation
Objective
Schalk04
Chol7
Lee19
Average
ERM
63.52 (0.92)
59.34 (0.23)
70.06 (0.46)
64.31
IRM
63.43 (0.37)
60.41 (0.07)
67.90 (0.27)
63.91
VREx
62.65 (0.29)
58.84 (0.26)
68.22 (0.33)
63.24
GroupDRO
63.97 (0.57)
60.24 (0.35)
70.34 (0.02)
64.85
IB-ERM
63.31 (0.16)
59.82 (0.38)
70.18 (0.41)
64.44
SD
63.72 (0.20)
59.31 (0.36)
70.15 (0.16)
64.40
Oracle train-domain validation
Objective
Schalk04
Chol7
Lee19
Average
ERM
64.52 (0.25)
60.41 (0.22)
71.11 (0.29)
65.35
IRM
63.28 (0.30)
61.09 (0.42)
68.77 (0.41)
64.38
VREx
62.41 (0.47)
59.28 (0.29)
68.08 (0.20)
63.26
GroupDRO
63.96 (0.36)
59.60 (0.38)
70.09 (0.16)
64.55
IB-ERM
64.63 (0.22)
60.22 (0.38)
70.27 (0.34)
65.04
SD
64.50 (0.05)
60.80 (0.17)
70.72 (0.14)
65.34
C.6.3 Credit and license
This dataset is built from 3 different motor imagery datasets (Schalk al. 2004 Cho al. 2017 Lee et al. 2019) as made available the online MOABB (Jayaram & Barachant 2018 platform. The Physionet.MI dataset is licensed under the Open Data Commons Attribution license v1.0.
C.7 LSA64
(a)
(b)
&
3 & 4
9 &
10
1 & 2
7 & 8
Figure 20: Summary of the LSA64 dataset. (a) The lask is to perform signed word classification from videos. (b) The dataset has five source domains, where each domain contains videos of different signers. The goal is to generalize to unscen signers.
C.7.1 Setup
Motivation Communication is an individualistic way to convey information through different media: text, speech, body language, and many others. However, some media are more distinctive and challenging than others. For example, text communication has less inter-individual variability than body language or speech. If deep learning systems hope interact with humans effectively, models need to generalize to new and evolving mannerisms, accents, and other subtle variations in communication that significantly impact the
42

---

Published in Transactions on Machine Learning Research (08/2023)
meaning of the message conveyed. We study the ability of models to recognize information coming from unseen individuals with the LSA64 (Ronchetti et al. 2016) dataset (Figure 20).
Problem setting We consider the video classification of signed words in Argentinian Sign Language. The dataset has five source domains, where each domain contains videos of different signers. The goal is to generalize to unseen signers.
The dataset consists of 3200 vidcos from 10 different signers signing in Argentinian Sign Language. The inputs X are videos of 20 frames with resolution (3, 224, 224). Sequences are two and a half seconds long. The labels Y consist of 64 words: Opaque, Red, Green, Yellow, Bright, Light-blue, Colors, Light-red Women, Enemy, Son, Man, Away, Drawer, Born, learn, Call, Skimmer, Bitter, Sweet milk, Milk, Water, Food, Argentina, Uruguay, Country, Last name, Where, Mock, Birthday, Breakfast, Photo, Hungry, Map, Coin, Music, Ship, None, Name, Patience, Perfume, Deaf, Trap, Rice, Barbecue, Cady, Chewing-gum, Spaghetti, Yogurt, accept, Thanks, Shut down, Appear, To land, Catch, Help, Dance, Bathe, Buy, Copy, Run, Realize, Give, and Find. The domains d are subgroups of signers: Signers & 2, Signers & 4, Signers 5 & 6, Signers & 8 and Signers 9 & 10.
Data
Preprocessing This section details the preprocessing steps taken for the LSA64 dataset. The raw LSA64 datasot contains 3200 vidcos, each about scconds long with the resolution of 1920x1080, at frames per second. We first crop all videos al precisely 2.5 seconds to have videos of the same length. This cropping does not impact the information content of the video signers pause at the end of their signed words. We then resize the frames to 224x224. Finally, use PyTorchVided (Fan et al. 2021 to uniformly sample 20 frames from each video in a sequence for prediction.
Domain information
Table 34 details the proportion of samples and labels across domains.
Table 34: Domain proportions of labels in the LSA64 dataset
Domain
64 words
Domain Total
Signer & 2 10 videos per word
640
Signer & 4 10 videos per word
640
Signer &
10 videos per word
640
Signer & 8
10 videos per word
640
Signer & 10
10 videos per word
640
Total
50 videos per word
3200
Architecture choice We Convolutional Recurrent Neural Network (CRNN) for this dataset. The CRNN model has 4 model blocks: Convolutional, Recurrent, attention, and prediction. First, we feod cach video frame through frozen Resnet550 model that is pretrained on Imagenet to extract. relevant features. We then feed these feature vectors sequentially to an LSTM model. Finally, push the output of the LSTM model for each frame through a self-attention layer which lincarly combines the LSTM output weighed by their attention scores. We the use fully connected network to make predictions. Table 35 details the layers of the model architecture.
43

---

Published in Transactions on Machine Learning Research (08/2023)
Table 35: Model architecture used for the LSA64 dataset
# Layer
1 Resnee550(mm3332222222 out-2048)
2 Linear(in=2048, out=512)
3 ReLU
4 BatchNormmnum. 512 momentum=0.01)
5
Linear(in=512, oul=512)
6 ReLU
7 (atuhNorrmnnum fcatures=512, momentum=0.01)
8
Linear(in=512, out=216)
9
ReLU
10
LSTM(in=256, hidden size=128, num layers-2)
11
SelfAttention(m==122, out=128)
12
Linear(in=128, out=64)
13
ReLU
14
Linear(in=64, out=64)
C.7.2 Detailed Results
ID evaluation We show the results of ERM for the LSA64 dataset in Table 36 doing hyperparameter search with the methodology detailed in Appendix F with no held-out test domain and choose the model with train-domain validation. In other words, the training is done with all domains; thus, all domains are ID. The columns correspond to the validation accuracy of the chosen model in each domain.
We obtain these results by
Table 36: ID results for the LSA64 dataset
Algorithm
Signers &
Signers 3 & 4
Signers & Signers 7 & 8
Signers 9 & 10 Average
ID ERM
90.10 (0.56)
89.58 (1.13)
80.21 (1.06)
35.16(1.61)
87.76 (0.77)
86.56
Benchmarks results We show the detailed benchmark results of the adapted OOD generalization algo- rithms in Table 37 Each results is obtained by holding out one domain during training and reporting the performance of the chosen model from the hyperparameter sweep on that held out domain, more details in Appendix
C.7.3 Credit and license
This dataset was adapted from the work of Ronchetti et al. (2016). The LSA64 dataset is under the Creative Commons
4.0 International License.
44

---

Published in Transactions on Machine Learning Research (08/2023)
Table 37: OOD generalization algorithms performance on the LSA64 dataset
Train-domain validation
Objective
Signers
Signers 3 &
Signers & 6
Signers
Signers 9 & 10
Average
ERM
48.50 (2.93)
50.65 (2.28)
47.53 (1.32)
57.49 (2.49)
62.96 (0.98)
53.42
IRM
44.34 (0.60)
43.16 (1.48)
38.28 (2.01)
46.88 (1.13)
52.47 (2.78)
45.03
VREx
42.19 (3.64)
45.57 (1.67)
42.06 (2.82)
51.82 (1.95)
52.21 (4.26)
46.77
GroupDRO
43.62 (3.95)
44.14 (1.87)
43.29 (1.58)
47.79 (1.70)
52.73 (1.36)
46.32
IB-ERM
55.66 (1.71)
56.71 (2.16)
49.80 (2.41)
64.52 (0.61)
59.70 (2.51)
57.28
SD
48.63 (2.46)
50.20 (1.60)
40.89 (0.84)
57.68 (2.54)
56.32 (1.19)
50.74
Oracle train-domain validation
Objective
Signers
& 2
Signers 3 & 4
Signers 5 & 6
Signers 7 & 8
Signers 9 & 10
Average
ERM
54.43 (1.11)
59.24 (0.23)
48.89 (1.45)
62.96 (1.25)
65.62 (0.42)
58.23
TRM
42.06 (1.17)
43.16 (1.48)
39.06 (1.39)
46.22 (3.83)
47.46 (1.95)
43.59
VREx
46.29 (1.81)
49.93 (0.45)
42.84 (0.75)
54.23 (0.59)
56.90 (0.53)
50.04
GroupDRO
50.52 (1.01)
54.49 (1.87)
45.12 (1.27)
56.51 (1.66)
63.22 (0.75)
53.97
IB-ERM
56.51 (1.38)
59.51 (0.91)
51.82 (0.96)
64.52 (0.61)
66.54 (1.27)
59.78
SD
56.58 (1.24)
60.68 (1.08)
49.35 (0.51)
62.43 (0.83)
64.06 (1.39)
58.62
C.8 HHAR
(a)
(b)
G. S3 mini
Galaxy S3
Sam. Gear
Nexus 4
LG watch
Accelerometer Gyroscope
Figure 21: Summary of the HHAR dataset. (a) The task is perform human activity classification from smart devices sensory data. (b) The dataset has five source domains, where each domain contains data gathered with a different smart device. The goal is to generalize to unseen smart devices.
C.8.1 Setup
The intrinsic biases from inaccurate and poorly calibrated sensors of smart devices, along with the accumulated biases from everyday use makes human activity recognition a notoriously difficult. task when task when done across devices (Stisen et al.] 2015 Blunck et al. 2013). Contrary to static tasks where uninformative features can often be segmented out from the input features (e.g., background when classifying an animal from an image), invariant features time series are often highly convoluted with other spurious features. We study the ability of models to ignore spurious information from complex signals with
Motivation
the HHAR (Sisen et al. 2015
Dua & Graff 2017 dataset (Figure 21
Problem sctting We consider the human activity classification task from accelerometer and gyroscope measurements of smartphones and smartwatches. The dataset has five source domains, where each domain contains data gathered with different device. The goal is to generalize to unseen smart devices.
45

---

Published in Transactions on Machine Learning Research (08/2023)
Data The dataset consists of 13674 recordings of 3-axis accelerometer and 3-axis gyroscope data from 5 different smart devices (3 smartphones and 2 smartwatches). The inputs X are five second recordings of a 6-dimensional signal sampled at 100Hz. The labels Y consist of 6 activities: Stand, Sit, Walk, Bike, Stairs up, and Stairs Down. Domains consist of five smart device models: Nexus 4, Galaxy $3, Galaxy S3 Mini, LG Watch, and Samsung Galaxy Gears.
Preprocessing This section details the preprocessing steps taken for the HHAR dataset. The raw data was gathered with 10 different smart devices (2 from each model). Different models have different sampling frequencies, plus recordings have gaps in the data samples where devices temporarily stopped recording, making the time series irregularly sampled. We first remove the recordings of any device that either is missing considerable amounts of signals or has less than 100 seconds of recording. We then sort the data points in each sequence according to their recorded time, instead of time the data was saved on the device. Next, we split the full recordings into sequences of five seconds and resample at 100Hz. Finally, we normalize the data with a standard scaler applied to the accelerometer and gyroscope channels separately,
Domain information Table 38
details the proportion of samples and labels across domains.
Table 38: Domain proportions of labels in the HHAR dataset
Domain
Stand
Sit
Walk
Bike
Stairs up
Stairs down
Domain Total
Nexus 4
760
911
1024
644
695
543
4577
Galaxy S3
664
889
944
560
635
474
4166
Galaxy S3 Mini
409
501
524
297
396
280
2407
LG watch
368
358
382
424
315
307
2154
Gear watch
21
23
78
42
120
86
370
Total
2222
2682
2952
1967
2161
1690
13674
Architecture choice As this data is similar to FEG recordings, we use the same deep convolution network model as in the CAP and SEDFx datasets. The architecture defined work from Schirrmeister et al.] (2017). We use the implementation of the BrainDecode (Schirrmeister et al. 2017) Toolbox. Temporal Convolutional Networks (TCN) are powerful tools for processing time series data (Bai et al.. 2018). The architecture we combines temporal and spatial convolution, which fits this data well. We found that it performed well on this task and obtained stable performance. The implementation is available at https://githuu: http://.......llroorro//rreneeee
C.8.2 Detailed results
We obtain these results by doing hyperparameter search with the methodology detailed in Appendix F with no held-out test domain and choose the model with train-domain validation. In other words, the training is done with all domains; thus, all domains are ID. The columns correspond to the validation accuracy of the chosen model in each domain.
ID evaluation We show the results of ERM for the HHAR dataset in Table 39
Table 39: ID results for the HHAR dataset
Algorithm
Nexus 4
Galazy S3
Galaxy S3 Mini
LG watch
Sam. Gear
Average
ID ERM
98.91 (0.24)
98.44 (0.15)
98.68 (0.15)
90.08 (0.28)
80.63 (1.33)
93.35
We show the detailed benchmark results of the adapted OOD generalization algorithms in Table 40 Each results is obtained by holding out one domain during training and reporting the performance of the chosen model from the hyperparameter sweep that held out domain, more details in Appendix F
Benchmark results
46

---

Published in Transactions on Machine Learning Research (08/2023)
Table 40: OOD generalization algorithms performance on the HHAR dataset
Train-domain validation
Objective
Nexus 4
Galazy S3
Galaxy S3 Mini
LG watch
Sam. Gear
Average
ERM
97.91 (0.03)
98.17 (0.18)
92.49 (0.26)
71.33 (0.67)
62.16 (1.69)
84.41
IRM
95.68 (0.47)
96.31 (0.53)
91.10 (0.35)
69.76 (1.44)
61.71 (1.56)
82.91
VREx
95.53 (0.55)
96.51 (0.16)
91.36 (0.43)
69.72 (0.29)
62.73 (1.15)
83.17
GroupDRO
96.49 (0.18)
96.79 (0.12)
92.13 (0.09)
71.64 (0.43)
63.74 (1.34)
84.16
IB-ERM
97.56 (0.06)
97.93 (0.21)
91.76 (0.57)
71.38 (1.02)
59.01 (1.86)
83.53
SD
98.14 (0.01)
98.32 (0.19)
92.71 (0.09)
75.12 (0.18)
63.85 (0.28)
85.63
Oracle train-domain validation
Objective
Nexus 4
Galazy S3
Galaxy S3 Mini
LG watch
Sam. Gear
Average
ERM
97.64 (0.06)
98.05 (0.07)
93.18 (0.20)
73.11 (0.77)
64.64 (1.20)
85.32
IRM
96.81 (0.14)
96.43 (0.09)
91.26 (0.23)
70.61 (0.51)
61.82 (2.21)
83.39
VREx
96.60 (0.24)
96.68 (0.29)
92.00 (0.65)
71.67 (0.84)
59.23 (1.17)
83.24
GROupDRO
96.54 (0.23)
96.94 (0.15)
91.62 (0.34)
71.33 (0.68)
64.86 (0.69)
84.26
IB-ERM
98.16 (0.09)
98.22 (0.09)
93.18 (0.16)
73.40 (0.68)
64.64(0.09
85.52
SD
98.48 (0.01)
98.67 (0.11)
94.36 (0.24)
75.12
64.86 (0.28)
86.30
C.8.3 Credits and license
This dataset was adapted from the work Stisen et al. 20155) as made available on the online UCI Machine Learning Repository (Dua & Graff 2017 This datasct licensed under the Open Data Commons Attribution license v1.0.
C.9 PedCount
(a)
( (b)
007
A
T01
T11
T61
Figure 22: Summary of the PedCount dataset. (a) The task is to forecast the count of pedestrian crossing streets of Melbourne. (b) The dataset has 65 source domains, where each domain contains pedestrian counts of a different street crossing. The goal is to perform well on unseen street crossings.
C.9.1 Setup
Motivation Data gathered from the behavior of a population follows scasonal (daily, weekly, yearly) trends. An example of this is the movement of population within a city, either by walking, public transport or car. These trends form from the daily life of the population, e.g., the influx the morning, outflux the evening, and absence on the weekend. However, these trends can shift when the data gathered from different sources in city. We study the impact of those trend shifts with the Pedestrian CCiyy of Melbourne 2017
Godahewal
et al. 2021 dataset. (Figure 22
47

---

Published in Transactions on Machine Learning Research (08/2023)
Problem setting The dataset has 65 source domains, where each domain contains pedestrian counts of different street crossing. The goal is to perform well on unseen street crossings. Specifically, we investigate the OOD generalization to location T22 and T25.
Data The dataset consists of 65 time scries comprising pedestrian crossing counts in the city of Melbourne, Australia. The time series are gathered from various parts of the city. The time series are gathered up to 30/04/2020, and the start of the data gathering process range from 1/5/2009 to 13/3/2020. The imputs X are seven days of pedestrian count sampled hourly, which we add 40 lag features and four time features. Lag features are past pedestrian count values that go past the seven day context given to the model. The time leatures
time indicators: hour of the day, day of the week, day of the month, and day of the year. The labels Y is the pedestrian count for the day following the seven days of context. Domains d consist of 65 different counters (T1-T65).
Preprocessing We do not perform any preprocessing for this dataset, this was already accomplished by prior work from Godahewa et al. (2021).
Domain information Information on start date and end date of data gathering can be found in Table along with some statistics such as time series average and maximum value.
Architecture choice For this datasct, we use forecasting Transformer architecture closely following the original formulation of Vaswani et al. (0177)). We found that it performed well on this task and obtained stable performance. Details are in Table 42
Table 42: Model architecture used for the Pedestrian dataset
# Layer
1 TransoornerEncoodrr(d model=48, nhead=2, num encoder layers-2,
dim feedforward=32, dropout=0.1, activation
2
Transforme:Decoderr (d_model==4, nhcad-2, num can_encoder 2
dim 322 dropout=0.1, activation=gelu)
C.9.2 Detailed Results
ID evaluation We show the in-distribution (ID) ERM results for the Podestrian datasct in Table 43 Wo oblain these results by doing hyperparameler search with the methodology detailed Appendix F with no held-out test domain and choose the model with train-domain validation. In other words, the training is done with all domains; thus, all domains are ID. The columns correspond the validation accuracy of the chosen model in each domain.
Table 43: ID results for the Pedestrian dataset
Algorithm
T22
T25
T1-T65 \TT22,125}
Average
ID ERM
96.40 (4.46)
101.73 (1.02)
61.48 (1.15)
62.65
Benchmark results
We show the detailod benchmark results the adapted OOD generalization algorithms in Table 44 Each results is obtained by holding out one domain during training and reporting the performance of the chosen model from the hyperparameter sweep that held out domain, more details in A Appendix F
48

---

Published in Transactions on Machine Learning Research (08/2023)
Table 41: Domain information in the Pedestrian dataset
Domain
Start date
End date
Time acries length
Time scrtes average
Time series maximum
Ti
1/5/2000
13/12/2018
S4331
1157
5573
T2
1/5/ 1/5/2009
20/4/2020
96187
1074
70:5
T3
19/5;2000
30/10/ 2019
91594
1207
5890
'T4
1/5/2008
17/8/2011
907281
1480
9052
T5
1/5/ 2009
12/12/200 20199
53068
1081
7391
T6
1/5/200 1/5/2009
12/4/2020
1193
0508
17
25/9/20009
15/8/2011
77921
306
117
118
21/6/2000
14/3/2020
151
3275
TO
1/5/20009 20009
30/4/ 2012
244224
518
5873
T10
1/5 2009
12/4/200 22/4/002
93085
176
3113
T11
1/5/2009 20009
27,2 2/ 2010
91908
99
9805
T12
1/5 22009
6/12/20110 2011
92911
2012
11284
T13
1/5/ 2009
26/4/2017 2017
70028
743
7510
'34
1/5 22009
1/1//
90030
308
T300
1/5/ 20009
23/6//2011
85700
900
1550
T16
1/5 2009
3/7/22014
45359
713
46400
1/5 2009
10/8/2011 2019
90091
3936
T18
1/5 22009
30/4, 00/0//0020
961123
311
$759
1/4/
1/3//3020
540469
506
25:4
T20
6/9/ 2013
20/3/2000 2020
57307
372
2231
T21
1/9 2013
12/4/ 22020
57979
606
T22
1/9/2013
1/12/200 2018
16030
1581
5655
122
1/0 22013
30/ 1/ 200M
562227
3354
38445
1/9/2013 20013
10/33/200 2012
57187
1195
1880
T25
1/9 2013
12/2019
54826
561
7064
28/9 2/033
/4/ 20120
515
2957
T27
1/9 2013
20/3/ 2010
57426
127
868
21/9/2203
19/3/2022
58918
981
795-1
T29
11/10/ 22013
16/12/2019
54186
441
7:194
10 2003
13/112/2019
530096
506
2801
T31
9/10/2013
15/4/2000
53139
282
3040
T32
12/12/2013
25/12/2001
28445
1057
9791
T33
25/ /1/2014
15/9/2011
47315
156
2218
133
8/6/ 22114
30/4/ 22020
51688
144
3708
12/4/2016
25,44 20/44/2000
33398
15445
9012
T36
21/1/2015
19/1/2020
43795
201
1380
1/2/2015
30/4/2022 20720
13978
159
1656
T338
1/1 /2016
/217
17662
2448
6965
23/8/2014
10/11/2019
45724
226
1823
T40
20/1/20055
9/44/0020
45765
287
2103
T41
1/7 2011
20/4/2019
158115
7138
T42
15/4/2015
24/3/202
43396
247
2202
T43
15/4/2015
30/4 30/4/2021
41219
201
1539
T44
15/ 15/4/2005
29/4 29/4//0220 20020
442077
U7
M06
T45
1/7 2017
17/12/22010
21,00
883
4736
8/4/2017 2017
12/4/2020
23472
100
678
T47
25/ 25/8/2017
26/4/2022
23411
950
4532
3/10/2017
1/4/2020
21876
2.19
2375
1149
30/11 01/11/2017
4/44/20120
207784
181
2167
T50
1/7 27/2017
4,4//0020
24215
255
2271
T51
1/12/2017
1/5/2002
21168
124
564
T52
1/8/ 2017
25/4 2//2/2021
23974
395
1918
1133
1/10/2005
13/4/20000
39765
GUG
3738
T54
1/7 20118
20/4 /4/ 2012
10033
148
1316
1/9/2018 2201
23/3/2019
5616
817
2006
T56
1/8 2018
18/4/ /2120
1.5027
310
1219
1/9/ 2018
1//3/2020
13368
798
15979
T58
1/10/2018
1/5./2020
13872
745
$352
13/2/2019
1/5/22020
252
39-19
TT80
18/4/2019
23/7//200
22211
1600
5424
T61
1/7/2019
1,85/2020
7320
447
2084
T62
1/10/2019
1/5/22020
5112
120
600
T63
8/1 1//2200
23/3, 20021
1777
291
2106
1664
17/1/2020
14/4/2220
23112
168
1157
T65
13/3/2020
1/55/20020
1176
136
1496
Total
2682
2952
1967
2161
C.9.3
Credits and license
This dataset was adapted from the work of City of Melbourne (0177) as made available on the online Monash time series archive (Godahewa et al. 2021). This dataset is licensed under the Creative Commons Attribution 4.0 International License.
49

---

Published in Transactions on Machine Learning Research (08/2023)
Table 44: OOD generalization algorithms performance on the Pedestrian dataset.
Train-domain validation
Objective
T22
T25
Average
ERM
196.07 (10.21)
212.11 (12.65)
204.09
VREx
197.98 (7.31)
205.19 (4.74)
201.58
GroupDRO
243.53 (16.90)
242.94 (9.17)
243.23
IB-ERM
224.55 (15.43)
201.60 (6.41)
213.07
Oracle train-domain validation
Objective
T22
T25
Average
ERM
226.78 (11.88)
219.71 (2.25)
223.24
VREx
203.52 (2.82)
222.63 (4.39)
213.07
GroupDRO
261.10 (12.11)
223.04 (7.84)
242.06
IB-ERM
201.43 (10.94)
209.89 (11.65)
205.66
C.10 AusElec
(a)
(b)
Jan
Feb
Figure 23: Summary of the AusElec datasct. (a) The task is to forecast electricity consumption. (b) The dataset has 13 time domains, where each domaii contains data from different months and holidays. The goal is perform well on all seasonalities.
C.10.1 Setup
Motivation Seasonality the property of time series where recurring characteristics appear every cycle of fixed period, e.g., weekly. A common practice in the forecasting field is to provide models with additional information, c.g., day of week in order to allow models to leverage scasonality for better predictions. However, holidays a seasonality of time series that is very sparse which models often fail to capture. We study the
performance of models on sparse seasonality with the AusElec (Hyndman & Athanasopoulos 2018 Godahewal
et al. 2021 dataset (Figure 23
We consider the electricity consumption forecasting task. The dataset has 13 time domains, where each domain contains data from different months and holidays. The goal is to perform woll on all seasonalities.
Problem setting
Data The dataset consists of five time series comprising 13 years of electricity demand across five states in Australia: Victoria, New South Wales, Queensland, Tasmania and South Australia. The inputs X are seven days of electricity demand sampled half hourly to which we add lag fcatures and 5 time features. Lag features are past electricity demand values the goes past the seven day context given to the model. The time features time indicators: minute of hour, hour of day, day of week, day of month, and day of year.
50

---

Published in Transactions on Machine Learning Research (08/2023)
The labels Y is the electricity demand for the day following the seven days of context. Domains d consist of time intervals throughout the year: January, February, March, April, May, June, July, August, September, October, November, December, and holidays.
Preprocessing We do not perform any preprocessing for this dataset, this was already accomplished by prior work from Godahewa et al. (2021).
Domain information We define the time interval of the Holidays domain as union of the following Australian holidays: New Year's Day, Australia Day, Good Friday, Easter Monday, Anzac Day, Christmas Day, Boxing Day.
Architecture choice For this dataset, we use forecasting Transformer architecture closely following the original formulation of Vaswani et al. (2017)). We found that it performed well on this task and obtained stable performance. Details are Table 45
Table 45: Model architecture used for the AusElec dataset
# Layer
1 Transoormerrncoodrr(d model=48, nhead-2, num encoder layers-2,
dim feedforward=32. dropout=0.1, activation-gell
2
Transforme/Decodeer( (40 nhead-2, num cun_encoddr 2
dim feedforward=32, dropout=0.1, activation-gelu)
C.10.2 Detailed Results
Unbalanced results has been reported in prior work (Koh al., 2021) that OOD generalization algorithms such IRM outperforms ERM on subpopulation shifi. datasets. However, it is unclear whether the improvements originates from the nature OOD generalization algorithms to upsample minority domains when computing the empirical risk or because the algorithm is performing well. In this work, we create an Unbalanced dataset of the subpopulation shift dataset which is agnostic of the domain definition during training. This allows us to compare the gain performance obtained by upsampling the minority domain when minimizing the empirical risk. We show those results in Table 46
Table 46: Results for the AusElecUnbalaneed dataset
Average validation
Worst-domain validation
Objective
Average
Worse
Objective
Average
Worse
ERM
22..73(2.64
409.80 (4.21)
ERM
235.40 (4.38) 395.99 (5.49)
Benchmark results in Table 47 Each line obtained by training on all domains of the dataset and reporting the average and worst domain performance of the chosen model, more details in Appendix
We show the detailed benchmark results the adapted OOD generalization algorithms
C.10.3 Credits and license
This dataset was adapted from the work Hyndman & Athanasopoulos (2018) as made available on the online Monash time series archive (Godahewa et al. 2021). This datasct is licensed under the Creative Commons Attribution 4.0 International License.
51

---

Published in Transactions on Machine Learning Research (08/2023)
Table 47: OOD generalization algorithms performance on the AusElectricity dataset
Average validation
Worst-domain validation
Objective
Average
Worse
Objective
Average
Worse
ERM
232.01 (2.60)
397.27 (8.48)
ERM
247.08 (7.59)
403.56 (6.57)
VREx
237.96 (2.53)
415.01 (9.92)
VREx
247.09 (2.19)
408.87 (3.97)
GroupDRO
237.09 (3.63)
408.83 (2.37)
GroupDRO
252.95 (7.58)
424.44 (13.34)
IB-ERM
232.03 (2.68)
393.56 (2.41)
IB-ERM
235.87 (3.11)
391.13 (5.44)
C.11
IEMOCAP
(a)
(b)
neutral/angry
sad/frustrated
neutral/excited
sad/neutral
neutral/frustrated
happy/excited
angry/frustrated
happy/angry
No shifts
happy/neuttal
Rare shifts
Figure 24: Summary of the IEMOCAP dataset. (a) The task is to perform emotion recognition from multi modal data (video, sound, text). (b) The dataset has 11 time domains, where each domain contains data from a different emotion shifts during conversations. The goal is to perform well on all conversational emotion shifts.
C.11.1 Setup
Motivation Speakers tend to maintain an emotional state over a conversation. However, external stimuli can invoke a shift the emotional state of speakers (Poria et al. 2019). Such emotion shift are often sparsely represented in the data, making it hard for models to classify them adequately. Recent work on emotion recognition models (Poria et al., 2019 2018 Majumder et al. 2019) show the failure of existing models to adapt. to those emotion shift. We study the performance of models emotional shift with the
IEMOCAP (Bulut et al. 2008 dataset (Figure 12
Problem setting We consider the emotion recognition task. The dataset has 11 time domains, where each domain contains data from a different emotion shift during conversations. The goal is to perform well on all conversational cmotion shifts.
Data The dataset consists of 151 videos about dyadic interactions, where professional actors are required perform scripted scenes that elicit specific emotions. Each video contains single dyadic dialogue, segmented into utterances. It contains 7433 utterances in total. The inputs X are utterances of video, speech, and text transcriptions. The labels Y consist of 6 emotions: Happy, Sad, Neutral, Angry, Excited, and Frustrated. Domains d consist of 11 emotion shift during conversations: No-Shift, Rare-Shift, and 9 common emotion shifts including Happy-Neutral, Happy-Angry, Happy-Excited, Sad-Neutral, Sad-Frustrated, Neutral-Angry, -xcital, Frustraaed, and
Preprocessing This section details the preprocessing steps taken bring the IEMOCAP dataset from its form to its final form used in WOODS. For each utterance, we extract multimodal features (audio, visual
52

---

Published in Transactions on Machine Learning Research (08/2023)
and text) following the same approach Hazarika al. (2018) and Majumder et al. (0019 To get our text embedding, we use a simple CNN with one convolutional layer followed by max-pooling (Poria et al. 2016 To extract high dimensional audio vectors, we use openSMILL (Eyben et al. 2010). These vectors comprise features like loudness, Mel-spectra, MFCC, pitch, etc. We use a 3D-CNN to capture vidco emboddings (Tran jet al. 2015). This embedding contains information for detecting emotional expressions like a smile or frown. We use concatenation of the unimodal features as a fusion method.
Domain information We consider utterances that have the same label as the previous utterance spoken by the same speaker as a no-shift domain. We consider emotion-shifts that appear in less than 20 utterances as rare-shift domain, namely, Happy-Angry, Excited-Angry Frustrated- -apppy, Sad-Excited, and Sad-Angry. We consider the remaining 9 emotion shifts as common ones and create a separate domain for each of them. For brevity, we call these domains common-shift in gencral. The ratios for the rare emotion-shift domain are 1/6, 1 1/6, and 2/3 for training, validation, and test respectively. For the remaining domains, dialogues are randomly chosen to achieve the ratios of 0.7, 0.1, and 0.2 for the size of training, validation, and test respectively.
Settrrate--xxciteee
Table 48 details the proportion of utterances and dialogues in the training, validation, and test sets across domains.
Table 48: Domain proportions of utterances and dialogues in the training, validation and test sets of TEMOCAP dataset.
Training
Validation
Test
# of utterances in rare-shift domain
22
19
61
# of utterances no-shift domain total # of utterance in common-shift domains
3785
369
957
1297
196
527
total # of utterances
5298
589
1546
total # of dialogues
108
12
31
Architecture choice For this dataset, we use a DialogueRNN model as defined in work from Majumder et al. (2019). We chose this model because it is well recognized by the ERM community. It also has an effective mechanisms to model context by tracking individual speaker states throughout the conversation for cmotion classification. The implementation available at https: // gigggtthbbb... cwwp://elaareelllaacoooeatttttoa master Daster/DialogueRN
C.11.2 Detailed results
Unbalanced results has been reported in prior work (Koh et al. 2021 that OOD generalization algorithms such as IRM outperforms ERM on subpopulation shift datasets. However, it is unclear whether the improvements originates from the nature OOD generalization algorithms to upsample minority domains when computing the empirical risk because the algorithm is performing well. In this work, we create Unbalanced dataset of the subpopulation shift dataset which is agnostic of the domain definition during training. This allows us to compare the gain in performance obtained by upsampling the minority domain when minimizing the empirical risk. We show those results in Table 49
Table 49: Results for the IEMOCAPUnbalaneed dataset
Average validation
Worst-domain validation
Objective
Average
Worst
Objective
Average
Worst
ERM
70.53 (0.05) 58.24 (1.41)
ERM
70.01 (0.77)
56.76 (1.24)
53

---

Published in Transactions on Machine Learning Research (08/2023)
We show the detailed benchmark results of the adapted OOD generalization algorithms in Table 50 Each line is obtained by training on all domains of the dataset and reporting the average and worst domain performance of the chosen model, more details in Appendix
Benchmark results
Table 50: OOD generalization algorithms performance the IEMOCAP dataset.
Average validation
Worst-domain validation
Objective
Average
Worst
Objective
Average
Worst.
ERM
69.12 (0.36)
57.75 (1.85)
ERM
69.85 (0.03)
56.33 (2.76)
IRM
68.73 (0.24)
55.93 (1.20)
IRM
70.21 (0.31)
58.95 (1.13)
VREx
70.12 (0.51)
59.45 (1.43)
VREx
69.64 (0.44)
57.66 (3.13)
GroupDRO
69.21 (0.75)
56.11 (1.19)
GroupDRO
70.08 (0.86)
58.79 (1.00)
IB-ERM
68.79 (0.08)
59.93 (0.55)
IB-ERM
70.04 (0.42)
58.81 (1.50)
SD
68.62 (0.22)
58.04 (0.39)
SD
68.75 (0.28)
56.14 (1.24)
C.11.3 Credits and license
This dataset was adapted from the work of Bulut et al. 2008 as made available by the Speech Analysis and Interpretation Laboratory (SAIL) at the University of Southern California (USC). This dataset is licensed htm
under the license availabel at htttps:/ssat //sail, usc. www.uuu....eeeeeeeeo. adwp/eenccap/eeocapppoco
D Further details on adapation of OOD generalization algorithms
D.1 General adaptation of 0OD generalization algorithms to time series
The problem formulation in Section 2.2 applies only sequence of same length S, and prediction times Sp across samples. However, for several dataset and tasks, this does not hold up. Take example the IEMOCAP dataset, conversations can vary in length and prediction times across samples. In this section, we provide a general formulation that accounts these changes.
Data samples consist of the input time series obscrvation X [Xilees; where St is the set of time steps for sample and the set of labels Y [Yiless, where S S is the set of labeled time steps for sample i.
Empirical risk For the empirical risk of domain d, we average the risk across the set of labeled time steps of sample i belonging to domain d: Sy..
R²
Σ
1
Σ (((()))))))))
S
(XD
LEST
where nd the number of samples from domain in the dataset D.
Penalty value function IR-ERM and SD penalize representation and logits during prediction, we follow Equation and define the penalty below.
P(f)
Σ
Σ P(). tesd.
(4)
nd
D.2 OOD generalization algorithm definition
IRM performs a constrained empirical risk minimization such that the optimal classifier of represen- tations is the across the domains. It. does by penalizing function of empirical risk across
54

---

Published in Transactions on Machine Learning Research (08/2023)
domains. We adapt IRM by using the empirical risk from Equation (3
1 Σ P()) -
IV WVll==10 NI
(IRM)
D
where [d] is the number of domains.
VREx penalizes the variance of risk across domains We adapt VREx by using the empirical risk from Equation (3):
P(f Var (
(VREx)
where Var the variance taken across domains.
GroupDRO performs importance weighting of the domains when calculating the empirical risk. We adapt the domain weighting parameter 9d using the empirical risk from Equation
qd EDED L()
(GroupDRO)
where q/ is the domain weights from the previous iteration.
IB-ERM penalizes the variance of representation within domains. Consider representation map +b (that transforms inputs X as ((X)) and a lincar classifier w such that our predictor f is defined as w- 4. We define the IB-ERM penalty as:
P()
1 d
Σ (arx DAED
(IBEERM)
where Var the variance is taken across samples of domain.
SD penalizes the squared 12 norm of the logits of the predictor f:
P(f) nd
Σ
Σ 1f(x))11
(SD)
Sp
(X.Y
BESS
where nd is the number of samples from domain the dataset D.
E Measuring the impact of distribution shifts
We use the generalization gap to empirically measure the impact of the distribution shifts on the performance of models. It measures the drop performance between data drawn In-Distribution (ID) and Out-of- Distribution (OOD), where the former is independent and identically distributed (i.i.d)) to the training distribution and the later is not. However, the generalization gap be misleading measure as it does not intrinsically indicate allainable performance gains. We show an example of unattainable performance gains later in this section. In this work, we do our best to measure an achievable performance gap for our dataset, ie., an upper bound to the achievable performance on unseen domains. In this section we give details on the generalization gaps described in Table are obtained for domain generalization and subpopulation shift datasets.
E.1 Generalization gap for domain generalization
Given a set of training domains traii and a test domain Deest we measure the OOD performance by training a model on the training domains Draain with ERM and measure the performance of this model on the test domain test The ID performance can be measured in multiple ways. Koh 2021) provides multiple definitions for it:
Same restriction with respect the training distribution is implied, see Section 2.1
55

---

Published in Transactions on Machine Learning Research (08/2023)
Train-to-train Performance of model on Dtrain when trained D
Mixod-to-test Performance of a model on Diett when trained on a mixture of D and Diest
Test-to-test Performance of a model on Deett when trained on Diest
We use the mixed-to-test measure for ID performance, because test-to-test and train-to-train can lead to erroncous mcasures leading to an inflated gencralization gap that unattainable in reality. To illustrate this problem, consider the generalization gap obtained with the train-to-train ID performance on the Spurious-Fourer dataset.
Example E.1 (Unattainable performance gap). Performing ERM on the training domains ( 90%) will lead to model relying on the spurious features to make predictions, as they are stronger predictor of the label (85%) than the invariant features (75%). Thus, the model will achieve 85% accuracy data sampled ID to domains d E (80%, 90%), but only achieve 10% accuracy on the test domain D10% Comparing the ID and OOD performance would lead to a generalization gap of 75%. However, this gap is misleading as a model could never achieve 85% accuracy on the test domain because the strongest invariant predictor can only achieve 75%. A similar case can be made for the test-to-test measure of ID performance, where the generalization gap lead to an unattainable performance on the test. domain.
Instead, consider the generalization gap obtained with the mixed-to-test ID performance for the same dataset.
Example E.2 (Attainable performance gap). Performing ERM on the training domains D lae (10% 80%, 90%) will lead to predictor that relies the invariant features, as they are stronger predictor of the label (75%) than the spurious features (60%). Therefore, the ID performance will be 75%, and the OOD performance will be 10,, leading to a generalization gap of 65%. This gap is much more significant measure of the upper bound of the performance than the original definition.
To summarize, we compute the generalization gap for domain generalization datasets as follows. Given a set of training domains Darain and a test domain Diesr we first measure the 0OD performance by training a model on the training domains Draain with ERM and measure the performance of this model on the test domain Drest Second, we mcasure the ID performance by training model on all domains D = Uuaittrin,test)) Dd and evaluate the model on the test domain Diest The generalization gap for that test domain is then defined the difference between the ID and O0D performance. That process can then be repeated for all domains in the dataset and we average the performance.
E.2 Generalization gap for subpopulation shifts
In subpopulation shift datasets, measure the OOD performance the worst domain performance, and the ID performance as the train-to-train performance, i.e., the average domain performance. We recognize that this measure is not a perfect because of similar arguments made in Section E.1 e.g., one domain might be much more difficult that the others and thus might be impossible to achiove the level of average performance on it. However, we argue that it is reasonable to consider the gap between the average domain performance and the worst domain performance as attainable. As a sanity check, one could verify that the average performance is achievable for a domain d by doing a test-to-test style measure. We leave this to future work.
To summarize, we compute the generalization gap for subpopulation shifts datasets follows. Given set of training domains traiin we mcasure the ID performance by training model on all domains, and measure the average performance across domains. We define the OOD performance as the performance of the model on the worst domain in Dtrain, The generalization gap for that dataset is then defined as the different between the ID and OOD performance.
F Evaluation framework workflow
In this section of the appendix, we detail the methodology employed to evaluate the performance of OOD generalization algorithms on our datasets. We follow the workflow used by Gulrajani & Lopez-Paz (2020) in their DomainBed testbed and adapt the framework to time series tasks.
56

---

Published in Transactions on Machine Learning Research (08/2023)
F.1 Reported performance
We detail in this section the performance measure used for the different dalasets in the WOODS benchmark.
Synthetic challenge datascts Spurious-Fourier, TCMNIST-Souree and TCMNIST-Time were formulated to address specific OOD generalization challenges in time series; thus we only investigate the training and testing domain configuration of interest, i.e., Dtrain = D Iae (80%, 90%) and Diest = 100% With this domain
configuration, we perform a hyperparameter sweep, the model selection (see Section and report the performance of the chosen model on the 10% domain.
Real-world domain generalization datasets We report the performance of an OOD generalization algorithm with domain vrossvaaldation measure follows. For every domain in dataset, perform hyperparameter sweep with that domain held out from training. After this hyperparameter search, we perform the model selection associated with the dataset (sce Section G and report the performance of the chosen model on the held out lest set. We then report the average performance across domains.
Real-world subpopulation shift datasets We report the performance an OOD gencralization algo- rithm with the worst domain performance. We perform a hyperparameter sweep with all domains in the training dataset Draain After this search, we perform model selection (see Section G and report the worst domain performance.
F.2 Systematic framework
Hyperparameter search All hyperparameter searches in this work use random searches (Bergstra & Bengio 2012) over the hyperparameter distribution spaces defined in Table 51 and Table 52 We train 20 models using randomly sampled hy hyperparameter configurations. We then select the best performing model of those 20 configurations using different validation sets definitions, see Appendix G
Statistically relevant We repeat each hyperparameter search three times to obtain statistically relevant results. This reduces the probability that some algorithm samples a lucky configuration of hyperparameters. All the results reported in this work are averaged over those three trials with a different seed. We also provide the estimated standard deviation of those averaged results.
Reducing bias The search range is an important topic when discussing the fairness of this evaluation strategy. Having reasonable hyperparameter distributions for sampling in the random search is essential to remaining fair between the algorithms and reducing the induced bias in the results. Defining narrow hyperparameter distribution for which one knows the algorithın performs very well on dataset or test domain leads to bias of the evaluation due to queries of the test domain through human intervention. This bias could lead to algorithms getting better results by increasing the chance of the random search finding a good value. When defining the hyperparameter range, one should define a range wide enough as to cover at least the relevant search space for this hyperparameter. In this work we use ranges that accurately reflects the range of useful hyperparameters values, see Table 52
G Model Selection
Section detailed the hyperparameter search and uncertainty estimation used in this framework. In this section, we detail the model selection strategy used in hyperparameter sweeps determine the model evaluate on the test domain.
G.1 Model selection for domain generalization
A fundamental restriction domain generalization that the training procodure does not have access to the test domains during training. As a result, the challenge OOD generalization is not only create models that generalize to the test domains but also to select the right models without. having access to the test.
57

---

Published in Transactions on Machine Learning Research (08/2023)
Table 51: Distributions of training hyperparameters for random search
Dataset
Hyperparameter Random distribution
learning rate
10 Unifformm (-..1 -4.5,-2.5)
batch size
2
Spurious-Fouries
class balance
True
learning rate
10Uniform-115. (-..6omm(-11,-2.33))
batch size
2 2nnform((39)
TCMNIST-Source
class balance
True
learning rate
10
batch size class balance
2nniform(3,9)
TCMNIST-Time
True
learning rate
10Unform(55,-3)
batch size
2nnforr((3.4
CAP
class balance
True
learning rate
10Uniformm-55-- -5,.3))
batch size
2nnifor((3.4)
SEDFx
class balance
True
learning rate
10 10Unffom(-55-3)
batch size
2 2niform(3.55
PCL
class balance
True
learning rate
Uniiformm (-5,33
batch size
2 2niform(3.44
LSA64
class balance
True
learning rate
Uniformm -4,-2)
batch size
2nnifor((3,4)
HHAR
class balance
True
learning rate
Uninfoomm( (-5-form(-5,33)) -5,-3) 2 2niform(3.5)
batch size
PedCount
class balance
True
learning rate
Uniform) (-5,33)
batch size
2 2niferm(3,5)
AusElec
class balance
True
learning rate
10 10Unifomm 5,-33
batch size
2 2niform(11.4)
IEMOCAP
class balance
True
domains. Many model sclection strategics were proposed (Gulrajani & Lopez-Paz. 2020 Ye ct al.
2021b
Koh el al. 2021 the simplest of which Truin-domain validation.
Train-domain validation We split the training domains into training and validation sets. The training split of the training domain is used to train the model. We choose the model that gets the best average validation performance across training domains. We report the performance of the chosen model on the testing domains.
However, tackling both problems of creating and finding invariant models the same time might be a very difficult research endeavor. Instead, we can first start by narrowing the scope and only focus on creating invariant models. For this purpose, relax the fundamental restriction and allow the queries of the test.
58

---

Published in Transactions on Machine Learning Research (08/2023)
Table 52: Distributions algorithm hyperparameters for random search
Dataset
Hyperparameter
Random distribution
penalty weight
10 (-,5)
Invariant Risk Minimization
annealing iterations
Uniform (0, 2000)
penalty weight
10 (QUnifor((-1))
Variational REx
anmealing iterations ")
Uniform (0, 2000)
GroupDRO
Uniformm( (13nform(-33111)
IB-ERM
penalty weight
10 Uniforrmm( (-3,0)
Spectral Decoupling
penalty weight
10 Uniformm( (-5.-1)
CAD
X
Choice( 10-3, 10-2, 10-1 ,1 10 102))
temperature
Choice([0.05,0..
CondCAD
Choice(11-~,110- 10-3, 10-0- 10-
1,,00,,,110 10 10
temperature
Choice Choice(0.00,0001)
Transfer
X
10 (-211)
8
Uniform 00.1orm(0.1.330 0.1, 10 Uniform 4.5.-2.5)
adv lr
adv steps
Choice (
CCDG
a temperature
Uniform (0, 1)
Uniform (0, 1)
Diversify
\
Uniform (0, 1)
X2
Uniform 1)
domain to obtain some signal on the absolute performance of an algorithm. Although quorying the test domain can never be considered a valid model selection stralegy practical scenarios, the results can be very insightful when evaluating the behavior of an algorithm. Gulrajani & Lopez-Paz Paz (2020) formulated Test-domain validation that queries the test domains to perform model selection.
Test-domain validation We split the test domains into testing and validation sets. Models are trained for a fixed number training stops on the training domains. We choose the model with the best performance on the validation set the lest domains. However, we ouly consider the final checkpoint of the model after a fixed number of steps, effectively disallowing early stopping. We report the performance of the chosen model on the testing set of test domains.
Test-domain validation has proven to be very useful measure of performance for algorithms synthetic datasots driven primarily by correlation shift (Ye et al. 2021b), e.g., CMNIST. In such datasets, simple spurious features highly correlated with the label create shorteuts in the data that model leverage to minimize the empirical risk quickly (c.g., cow or camel classification problem). As a result, these shortcuts lead to very high training domain performance and very low test domain performance early training. Consequently, any model selection criteria that rely on performance on data drawn i.i.d. to the training distribution is a poor way investigate the performance of an algorithm because there is bias of model selection towards early training correlation. Thus, by disallowing early stopping, obtain insightful measure to investigate the absolute performance of an algorithm.
On the oer hand, Test-domain validation is ill-equipped to provide meaningful measures of performance with other kinds datasets. For example, Test-domain validation not an insightful measure of performance when dealing with wal-world datasots. The reason that we often do not know beforchand the number of training steps required for a given set of hyperparameters such that model will finish the learning of the task. Therefore, we train models past the point of overfitting and pick the model with the highest validation
59

---

Published in Transactions on Machine Learning Research (08/2023)
performance. This renders the last checkpoint in training suboptimal for generalization performance, both ID and OOD, and leads to uninformative measure of the generalization performance.
We introduce a more pragmatic model selection method that queries the test domain for real-world datasets to resolve this problem: Oracle train-domain validation.
Oracle train-domain validation We split the training domains into training and ID validation splits. We also split the test domains into testing and OOD validation splits. For every model training run, we choose the early stopped model that performs best the ID validation split. Among all early stopped model of the sweep, then choose the model that performs the best in the OOD validation split. Notice that this model selection method has the same number of queries of the test domain as the test-domain validation, i.c., one query per training run.
light of the discussion of this section, we use two different sets of model selection methods for the two different types datasets WOODS: Synthetic challenge and real-world datasets. We use train-domain validation and test-domain validation for our synthetic challenge datasets driven by correlation shift. We train-domain validation and oracle train-domain validation for real-world datascts which are likely driven by other kinds of shifts.
G.2 Model selection for subpopulation shifts
Model selection in subpopulation shift dataset is a much simpler endeavor because access to domains is not. restricted. We define the two model selection strategies for our real-world datasets follows.
Average domain validation We split all domains into training, validation and testing splits. The training splits of that dataset is used to train the model. We choose the model that gets the best average validation performance on all domains. We report the worst testing split performance of the chosen model.
Worst-domain validation We split all domains into training, validation and testing splits. The training splits of that dataset used to train the model. We choose the model that gets the best worst domain validation performance. We report the worst lesting split. performance of the chosen model.
60


Reviews:
[{"review_id": "CbeQaDwSFu", "reviewer": "Reviewer_QoQM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nI agree with the authors that there has been limited exposure for generalization under distribution shift for time-series-based tasks. Introducing there datasets are useful.\n\n\nCons:\n1. My main concern is regarding the quality of the distribution shifts, especially on real-world datasets.\nIn Table 1, we observe that the effect of distribution shift (i.e., the gap between in-domain and OOD performance) is not significant (except for the LSA64 dataset), even for ERM. Further, the performance improves as we apply more sophisticated techniques (Table 2: Train-domain validation). \nFinally, as we apply early stopping based on the validation set, the performance significantly improves, particularly for synthetic datasets.\n\n2. Please provide a more detailed summary table for the descriptions of datasets such as the number of instances, shape of inputs etc.\n \nAlso, I believe that such work of introducing new datasets, without technical novelty, should be submitted to a more dataset-focused track.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a set of 10 datasets on time series. It includes 7 real-world datasets and 3 synthetically generated datasets. The authors also report on the existing OOD generalization methods to demonstrate the effect of distribution shifts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the above comments.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IPhl6l1r3F", "reviewer": "Reviewer_7KNQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n1. The topic is very necessary and interesting: OOD in time series remains an important and unsolved problem.\n2. There are a lot of efforts in building such a complex and various benchmark, with benchmarking results and discussion.\n\n### Weakness\n\n1. The definition of subpopulation shift is missing in Section 2, but the results are just shown in experimental sections. You should add the definition in problem formulation.\n2. The diversity of the datasets is not enough: there is only one forecasting dataset and the remaining nine datasets are all for classification. While time series classification is common and popular, the forecasting task is more unique in this domain. Therefore, I would encourage authors to add more forecasting datasets such as weather forecasting.\n3. Authors only benchmarked existing general OOD algorithms while ignoring the advance of algorithms that are designed specifically for time series in OOD settings. I can provide several references (see below) and authors are encouraged to add more in their revision.\n4. There should be more analysis towards the OOD generalization problem in time series rather than simply benchmarking algorithms. For instance, what makes it harder for time series in OOD setting? What can future research focus on?\n5. Finally, the experiment section needs more details. For instance, what is the backbone of the experiments? As can be seen, there lacks a unique backbone in time series, not like ViT/ResNet in CV and BERT in NLP. So, the backbone matters in the results. Authors should clearly state the backbones in the paper and analyze the performance of different backbones. For instance, Transformers are known to work well in time series, so how about its performance in OOD settings?\n\nReference on time series OOD algorithms:\n\n[1] Ragab, Mohamed, et al. \"Conditional Contrastive Domain Generalization for Fault Diagnosis.\" IEEE Transactions on Instrumentation and Measurement 71 (2022): 1-12.\n\n[2] Lu et al. \"Out-of-distribution Representation Learning for Time Series Classification\". ICLR 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed WOODS, a benchmark suite for out-of-distribution generalization research on time series datasets. WOODS consists of ten synthetic and real-world datasets in both classification and forecasting tasks. The authors first gave a formal definition of OOD generalization and then explained why OOD in time series is different and harder than traditional static setting. Next, they described the construction of each dataset. After that, authors used DomainBed suite to perform experiments on these datasets. The main conclusion is that OOD in time series remains a challenging problem and several OOD approaches do not significantly outperform ERM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add definition to subpopulation shift.\n2. Add more forecasting datasets.\n3. Add more time series OOD algorithms.\n4. Add more discussions as said in weakness section.\n5. Add more backbones.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tLoEY3tn0T", "reviewer": "Reviewer_k3SH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The problem addressed by this work is interesting and important. In particular, the paper explores the underexplored area of OOD generalization in time series tasks, which is crucial for various applications such as computational medicine, natural sciences and finance.\n2. The WOODS benchmark is solid and contain data from diverse data modalities and a wide range of critical problems, offering researchers a versatile tool for evaluating algorithms.\n3. The authors examine several important OOD baselines on the time series data, providing valuable insights and highlighting challenges in time series tasks.\n\n\nWeaknesses:\n1. The paper does not explore the potential benefits of large-scale pretraining for time series tasks, leaving it an open question for future research. For instance, in WILD 2.0, they include the results with foundation models.\n2. It is interesting to see some baselines such as IRM perform worse than ERM even in synthetic and toy setting, as it is specifically designed to address the spurious correlation. Is there any plausible explanation to this?\n\nOverall, the paper does not have significant weaknesses and the contribution is solid.\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the underexplored area of Out-of-Distribution (OOD) generalization in time series tasks. The authors present WOODS, a benchmark consisting of 10 time series datasets, covering a wide range of data modalities, such as videos, brain recordings, and smart device sensory signals. They develop a systematic framework for evaluating time series datasets and algorithms and adapt existing OOD generalization algorithms for time series tasks. Extensive experiments show that there is a significant room for improvement in empirical risk minimization and OOD generalization algorithms on these datasets, highlighting the unique challenges posed by time series tasks.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While the paper is already making valuable contributions to the field, exploring large-scale pretraining could potentially enhance its impact. In particular, it would be interesting to investigate the impact of large-scale pretraining on time series tasks to determine if it can help address OOD generalization challenges. Comparing pretrained and non-pretrained models would provide valuable insights. In general, the paper is already in a good shape.\n", "broader_impact_concerns": "The paper does not have serious ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers identified a number of issues, which authors have addressed in the revision, including: adding new forecasting dataset, adding 2 new baseline methods, discussions of future work, clearly stated definitions of some terms, and other presentation issues.

One reviewer still had a concern "in Table 3 (Oracle train-domain validation), baseline models often significantly closed the performance gap between ID and OOD settings. However, I agree that one single model did not always achieve the best performance. Also, except LSA64 dataset, the performance gap between ID and OOD remains quite low." Responding, authors noted that: 1) the absolute values of the generalization gap are within the range considered as failures in the field; 2) the oracle performance listed Table 3 "cheats" by using the test data to select the best model, and thus should not be considered as the current capabilities, but rather an upper-bound.  

The AE agrees with the authors, and also notes that benchmarks are necessary to ensure measurable progress in the field. As performance on a benchmark becomes saturated newer or larger benchmarks will be introduced. 

The benchmark provides a common platform for evaluating OOD methods on a variety of time-series data, making an important contribution to field to ensure measurable progress for future research. Thus, the AE also recommends a "featured certification".
Decision: Accept (poster)