Towards Understanding Evolving Patterns in Sequential Data
Authors: QIUHAO Zeng
Long-Kai Huang
Qi CHEN
Charles Ling
Boyu Wang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Time series
sequential data
autoregressive tasks
evolving domain generalization
temporal domain generalization
Primary area: other

Abstract: In many machine learning tasks, data is inherently sequential. Most existing algorithms learn from sequential data in an auto-regressive manner, which predicts the next unseen data point based on the observed sequence, implicitly assuming the presence of an \emph{evolving pattern} embedded in the data that can be leveraged. However, identifying and assessing evolving patterns in learning tasks often relies on subjective judgments rooted in the prior knowledge of human experts, lacking a standardized quantitative measure. Furthermore, such measures enable us to determine the suitability of employing sequential models effectively and make informed decisions on the temporal order of time series data, and feature/data selection processes. To address this issue, we introduce the Evolving Rate (EvoRate), which quantitatively approximates the intensity of evolving patterns in the data with Mutual Information. Furthermore, in some temporal data with neural mutual information estimations, we only have snapshots at different timestamps, lacking correspondence, which hinders EvoRate estimation. To tackle this challenge, we propose EvoRate$_\mathcal{W}$, aiming to establish correspondence with optimal transport for estimating the first-order EvoRate. Experiments on synthetic and real-world datasets including images and tabular data validate the efficacy of our EvoRate.

Towards Understanding Evolving Patterns in Sequential Data
Qiuhao Zeng Western University
Long-Kai Huang Tencent Al Lab
q2eng538uwo.co
hlongkaie@mail..o com
Qi Chen Laval University
Charles Ling" Western University
Boyu Wang"
Western University
qi. chen.eenccco 1@ulaval.ca ca
charles. weteesliing www.glinnngw..oommmmm
bwang@osd. uwo ca
Abstract
In many machine learning tasks, data is inherently sequential. Most existing al- gorithms learn from sequential data in an auto-regressive manner, which predicts the unseen data point based on the observed sequence, implicitly assuming the presence of an evolving pattern embedded in the data that can be leveraged. However, identifying and assessing evolving patterns learning tasks heavily relies on human expertise, and lacks a standardized quantitative measure. In this paper, we show that such a measure enables us to determine the suitability of employing sequential models, measure the temporal order of time series data, and conduct feature/data selections, which can be beneficial to a variety of learning tasks: series forecastings. classification tasks with temporal distribution shift, video predictions, etc. Specifically, we introduce which quantifies the evolving patterns in the data by approximating mutual infor- mation between the next data point and the observed sequence. To address cases where the correspondence between data points at different timestamps is absent, we develop EVORATEW. a simple and efficient implementation that leverages optimal transport to construct the correspondence and estimate the first-order EvORAT Experiments on synthetic and real-world datasets including images and tabular data validate the efficacy of our EVORATE method.
EVOLVING RATE (EVORATE),
Introduction
Sequential data is ubiquitous across various machine learning tasks, including multivariate time series [33] 38 video streams in computer vision [18 52 54), textual data natural language
processing 34 and state-action trajectories in reinforcement learning [5] 45 [6]. Learning
with sequential data usually involves predicting future data points, fostering the development of
auto-regressive techniques that learn to forecast the subsequent unseen entries in a sequence. Despite the progress this field, one fundamental challenge persists: the identification of underlying evolving patterns often depends heavily on the subjective interpretations and prior knowledge of human experts. This reliance on subjective judgment lacks a robust quantitative method to assess the evolving patterns over the high-dimensional data in deep learning. For example, when designing recommendation system, certain products such as clothing are highly dependent on temporal factors (c.g., seasons, fashion trends), while others, like computers, are more influenced by individual customer preferences. Therefore, it is critical to identify and quantify the underlying evolving patterns for different products and integrate this information into the algorithmic design.
"Correspondigg authors: Boyu Wang, Charles X. Ling.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Specifically, the following questions are essential but unresolved yet in literature: i) How can the existence of evolving patterns in data sequences be determined? Determining the existence of evolving patterns in data is a critical task. It is possible that the data points of sequence are entirely independent and no evolving patterns exist. For instance, consider the scenario of a person repeatedly tossing coin. In this case, historical information does not influence the outcome of the next toss. ii) Can one determine the historical span that significantly influences the current time point? For example, how do we determine the order (the optimal number of past observations) of an autoregressive model in a principled way? iii) How can we determine if the collected features are sufficient to reveal evolving patterns? For instance, to achieve better weather forecasting, how can one determine the essential features, such as altitude, humidity, and geographic location, for gathering a comprehensive set of information for forecasting?
In this work, we address these questions through a unified framework by introducing EVORATE (EvolVinG RATH), a novel approach designed to quantify the evolving patterns of data sequences. EVORATE leverages mutual information as a measure of the existence of the evolving patterns in the data. Notably, while there is a rich history of mutual information estimation in the existing literature 8 I 27 I0 29], existing works ignore the underlying temporal dependency between the data points, and therefore are not well-designed for sequential data. EVORATE tackles this issue by estimating mutual information in an autoregressive manner when learning the compressed embedding from the observed sequence, thereby addressing the aforementioned questions: i) it can serve as an indicator to show that learning a sequential model is not feasible to learn the provided sequential dataset. ii) EVORATE can provide quantitative measure of the temporal dependency of sequence, allowing us to control the trade-off between computational complexity and learning performance. iii) EVORATE can also guide in selecting the most informative features for model training for sequential data.
However, EVORATE is difficult to estimate when dealing with temporal data characterized by snapshots captured at disparate timestamps without clear correspondence between them [30 48 42]. as we do not track the same data point over different timestamps and thus lack access to its corresponding sample. This scenario hinders the estimation of EvoRate, due to the absence of the correspondence. To mitigate this issue, we propose enhanced version of our methodology. EVORATEW, which is specifically designed to establish correspondence among data points across different timestamps utilizing optimal transport within the Wasserstein distance metric, thereby facilitating the estimation of the first-order EVORATE. In all, the benefits of EVORATE to be highlighted include:
EVORATE enables quantitatively measuring the evolving patterns existing in high-dimensional sequential data by utilizing the neural mutual information estimator. Furthermore, it can be applied assess temporal order and conduct feature selections in sequential data.
We further proposed EVORATEW to leverage optimal transport to build the correspondence
between snapshots at the different timestamps, and hence allow the MI approximations.
We motivate through analysis the use of mutual information as indicators of evolving patterns and show optimal transport can mitigate the without correspondence issue.
Synthetic and real-world datasets verify that EVORATE can be a good indicator for evolving patterns, supporting our claim of its benefits. We also design an EDG algorithm based on the insight of EvoR ATEW and verify its performance. The codes are available on GitHub: https://githae
2 Related Works
Sequential Data The analysis and processing of sequential data is driven by diverse applications
ranging from video predictions time series forecasting [25 50 9 16 33]. Pioneering works such as Long Short-Term Memory (LSTM) 25 networks have established foundational principles for handling long-range dependencies in sequence data. Building on this, the Transformer [50] introduced revolutionary approach through self-attention mechanisms, enhancing flexibility in handling sequence dependencies. The versatility of Transformers has been further demonstrated in models such as GPT-3 9 and BERT [16]. Beyond text, sequential data analysis in machine learning also extends to time-series forecasting 3 Moreover, the application of Graph Neural Networks in capturing dependencies in irregular sequences underscores the breadth of methodologies exploring

---

the complexities of sequential data 77.. However, a qualitative method for measuring the intensity of evolving patterns remains lacking in the literature.
Mutual Information (MI) Estimation has become a pivotal tool in machine learning 39 g 27
10 29
enabling insights into dependencies that extend beyond traditional correlation measures. In feature selection, MI offers data-driven approach to identify relevant features without strong assumptions about data distributions
(39), Mutual Information Neural Estimation (MINE) B applies deep learning to estimate MI in high-dimensional settings, providing a new methodology for analyzing neural network training dynamics. MI's application variational inference, especially in the training of variational autoencoders ( (VAEs) [. In reinforcement learning, MI has been used to enhance exploration strategies by quantifying information gain [27]. MI also improves the performance of generative udversarial networks (GANs) [TO]. Furthermore, in unsupervised and semi-supervised learning, MI maximization has been shown to effectively leverage unlabeled data However, none of them employ MI as an indicator for evolving patterns of sequential data.
Optimal Transport (OT) has emerged as a powerful framework in machine learning 3 31
41l. offering a principled approach to compare probability distributions. Optimal transport theory has been leveraged for applications ranging from domain adaptation to generative modeling (ST). Recent advances include the integration of OT with deep learning architectures; Wasserstein GAN (WGAN) utilizes the Wasserstein distance improve the stability of training GANs 3. Furthermore, optimal transport has been applied effectively in NLP [1.. The computational aspect of OT has also seen significant developments, Sinkhorn [13 as a scalable method approximates transport plans efficiently, More recently, researchers have explored the differential properties of transport plans in dynamic environments [4] EvoR ATE employs OT to recover the correspondence between two consecutive timestamps, facilitating approximations of mutual information.
Patterns estimation for sequential data has only one related work the literature ForeCA (24). which proposes a similar concept, "forecastibility" which measures the uncertainty of the entropy of the spectral density. However, ForeCA has two drawbacks. Firstly, ForeCA can not be used in deep learning as an unacceptable huge computational consumption for real-world high- -igh--imensional data (audio, videos, etc.). In contrast, EVORATE shows the prediction power by relying on mutual information, which tells the ability to predict another variable based on known observed variables. Secondly, while temporal patterns can include trends, cycles, irregular fluctuations, and more complex behaviors, ForeCA can only detect cycled patterns. Instead, EvoR ATE relies on the neural mutual information estimator, which known as a good measurement for various patterns as a result of the strong fitting power of neural nets 8
37 46).
3 Preliminary
3.1 Variational mutual information estimation
The mutual information between two random variables X and Y is defined as the KL divergence DKL. between their joint distribution and the product of their marginal distributions:
= =xxx)))Dx<PP(x,,,,, (DX(((XXXXXXXX))))))))
(1)
where we aim to estimate this using samples from P(X, Y); in some cases, the density of the marginals such as P(X) may be known. wide range of variational methods are designed to
estimate variational mutual information g
46 36 37). We then use the below estimator to
estimate mutual information:
i(x; E Px.. [m v)) log E P()) (em(a.s))
(2)
where X the random variable, x is a realization of X (as the case with Y and y), P is the
empirical distribution associated with a dataset of i...d. samples, and m(r y) is a critic function to
quantify the similarity between X and Y, usually realized by neural network [8] 46 36 37
We show that MI is highly related to the evolving patterns of the sequential data in Section 4.2
3

---

3.2 Optimal transport
rich class of divergences between probability distributions is induced by the optimal transport (OT)
problem [5T]. Kantorovich's 's formulation of the problem is given by
inf W. (Px), P(Y)) TEI(((X))(()
Px [c(X YX),
(3)
where c(x, y) X' x y - R is any measurable cost function and (X), P(Y)) )) is the set of all the joint distributions т (XX, whose marginals are P(X) and P(Y) respectively. The Wasserstein distance We is then the "cost" of the optimal transport plan.
4 Measure evolving patterns via MI
4.1
EvoRate
Consider sequence z (=t)]= collection of sequential data points from time to T, where each 2, E DR& denotes a state observation the discrete time step with total T steps. practice, the sequence zi can represent time series data, video, textual, audio, or any other ordered data stream.
We propose the use of the mutual information (MI) between the next observation and historical data over the past k steps / (2i Z to measure the evolving pattern within time window of length In the literature, the mutual information is empirically estimated through equation 2 which involves learning the critic function m [8 I 46 37). However, one critical issue with existing works is that they ignore the temporal dependency of the data, and therefore the critic function m can have a high bias for sequential data (shown in Figure Ta Tb).
To take the temporal dependency into account when estimating I (Zi Ze+1), instead of learning the critic function m, we propose learning the autoregressive function f. which summarizes the historical information embedded in Zi and measuring its distance to Zz+1 via the squared error
metric. Specifically, introduce EvoRate estimate the empirical sequential MI i(z]- Zz+1)
by defining m : R RR I. m(x y) (llf((()))))....)) g.xx)) ( in equation 2
EvoRate: i(zi- kt1; Z++1) sups.g B, P and (1 .. o(x)) g(=++1)ll
- log E, _ ~ PiZ, I 2.).
(4)
where 9 RDD + Rd is an encoder. By selecting a different d. we can make trade-off between computational cost and MI estimation accuracy. With d < D, EVORATE is more computationally efficient method for approximating sequential MI than learning an autoregressive model in the original data space. However, due the Data-processing inequality [12. this results in lower MI estimates. As is employed as an identity function, MI is estimated in the original space, thereby enhancing estimation correctness at the expense of increased computational consumption.
4.2 Discussion
In this section, we justify the validity of EVORATE as a metric of evolving patterns through the lens of k-th order autoregression. Specifically, define the Maximum likelihood estimation (MLE) loss as Liit = -EP(2 .-)11) log Q( ( where Q is the probability distribution learned by the autoregressive model F trained with supervised loss (MLE, MSE) sequential data. Note that the MLE loss can also be viewed as the expected risk of autoregressive prediction tasks [44).
The following proposition establishes the connection between the expected risk of a k-th order autoregression task and the mutual information
Z++1):
Proposition 1. Let H denote the entropy. For autoregression tasks, the expected MLE loss satisfy:
Lamte = DKL(((ZeH)) (2 1Z, (Ze++)) - T(7++1; 2 2
(5)
(8) Model related
(6) Dana related
proof of the proposition is provided in Appendix A learning predictive model for autoregression task from an cittmmattioo-thoretti perspective:
Proposition I provides novel insights into

---

1. The expected risk can be decomposed into two orthogonal factors, where (i) measures the distance between the learned distribution Q and true distribution P. and therefore is determined by the predictive model F. (ii) quantifies the inherent temporal dependency of the sequence. Notably, is independent of F.
2. More importantly, when Z is a discrete variable, due to the nature of mutual information, ((4411Zi S H (Zz+1) and (i) attains a minimum of zero when the observed sequence - encapsulates all the information of Z++1- Conversely, Proposition| 1 reveals that even F can properly learn the true probability P (i.e., (i) is small), its expected risk remains high when there is no temporal dependency that can be leveraged (i.e., I is small).
Consequently, EvoRate, as an empirical estimate of T 7Z++1; 7 can play an important role in indicating the success of learning from sequential data and therefore is adopted quantify the evolving pattern in this work.
Subsequently, we demonstrate that MSE loss defined Lmae E
2++11 can be interpreted as a variant of MLE loss, hence MI can be applied to a wide range of sequential data tasks that utilize MSE loss.
Proposition Assume that the predicted conditional probabiiity density Q learned by the predic-
tion model follows (ZZ++1 [Z, (Z+x+1) = 1( distribution with mean F(Z and identity covariance matrix 1 Lp. Then, the following holds
where N-) denotes Gaussian
Limle
fmme + const,
(6)
where Lonse is the MSE loss and const. a constant term.
5 Measure evolving patterns without correspondences
5.1 Estimate joint distribution
In many real-world applications, instead of processing many data point observations at different timestamps as data sequences, one needs to handle a data set at each timestamp: {z<,],= collected from multiple timestamps
(1,... T} 48 42. where is the sample index and n4 is the number of samples collected at timestamp L. The distribution P( Ze) associated with these data sets evolves over time 1. E R. For example, consider a supervised learning problem involving medical data 21,1 = (zz,5; st,s) collected from multiple patients at different ages [42 6) In this scenario, we do not track the same patient across different ages, resulting in lack of correspondence between timestamps and our objective extends to characterizing the evolving patterns of (2 1 across these discrete timestamps. However, EVORATE proposed subsection 4.1 cannot be applied to this context due to the absence of the correspondences.
Estimating the mutual information from two data sets requires the pairwise correspondences between the sample of two data sets, which are assumed given in existing works [8 27 10 29]. The correspondence between Z, and Z++1 reflects their joint distribution of Le and Z141 co-occur. This structured relationship indicates the interdependence of Ze and Ze+1. which the joint distribution quantifies. Since the absence of the correspondence (i.c., an object observed at time / is not at time i + 1), we can not access the joint probability distribution of the past states and the next state. To tackle this issue, we estimate the joint distribution through the optimal transport plan of the Wasserstein Distance. Specifically, we define the distance loss according to a joint distribution measurement π
encapsulates how the values
(7) where g is fixed from updated gradients computed from Cwy Empirically, allowing g to update during model training leads the undesirable outcome of all representations collapsing into single point as a result of minimizing the Wasserstein distance loss. To avoid this and preserve maximal information within the representations, g is trained separately using auto-encoder architecture with a reconstruction MSE loss.
fT (==llf(((z))) - g(
Then, we compute the optimal transport plan π to approximate the real joint distribution
nt (7z, Z++1) =
arg mig
fi, Vt e (1,...,--1),
(8)
т E (L((())).P(2..)))
and f is updated in an alternating optimization manner with fixed n" to minimize ( .f). In practice, the following implementation is used: 7² := arg
(C. 7).. where C is
5

---

the cost matrix with Ct,, = (z,)) 90 When the original dimension D is low, can be an identity function for precise MI estimation. Conversely, when D high, directly learning from Ze to Z+++1 requires more accurate information and precise correspondence. This is because must be a considerably more complex and larger model to facilitate mapping from one high-dimensional space to another. As result, the hypothesis space F for expands, requiring more information to ensure the model converges to an optimal state. The absence of correspondence therefore presents a challenge as it leads to information- iintuattionnnsuuficientt situation and it becomes more suitable to set smaller representation dimension d.
It is noted that when the correspondences between two consecutive timestamps exist, they can be inferred by minimizing the Wasserstein distance. When such correspondences do not exist, one can still establish correspondences by identifying proxy of 2 in the succeeding timestamp that exhibits similar dynamics and shares the latent evolving patterns.
5.2 EVORATEW
We hence use 7" (Ze, 1x-1) to estimate joint distribution and then obtain the following estimator with n (Ze, Z(+1)
EvoRatew
/ (( (2s.2e-1) -=](((=))))- ((+++1)ll
log P00. -11f((())))- -t
(9)
Here can regarded to set to compared to equation indicating that EVORATEw focuses on the first-order evolving patterns. It is possible to extend this approach to estimate higher-order k-order sequences by iteratively leveraging outcomes from first-order through k-order sequential modeling.
5.3
Discussion
The following assumption argues that there exists an optimal function that precisely captures the underlying dynamics of evolving data.
Assumption 1. (Realization) In machine learning prediction tasks, there exists a function f* F + where the conditional distribution of /2+1 given Ze satisfies
7x+1 N( 77)), o T) = 7(7++1)7.)
(10)
The following lemma demonstrates that when f reaches the optimal predictive model f", the estimated optimal transport plan equals the real joint distribution. In this context, we consider g to be the identity function. Lemma 1.
Let P(Z,, Zit1) be the ground truth joint distribution. If attains f", then
T" (Ze, Ze+1) P(z,, Ze+1)
(11)
Below, give an illustrative example. As T >> 1, the function f will converge to f by minimizing ( demonstrates that for a dynamic system without correspondences, the number of timestamps must be greater than 1 to learn the optimal autoregressive model effectively.
Example Consider data collected from multiple time steps where each sample a vector Z, R.D. Specifically, the initial data points at the first timestamp is modeled as a Gaussian variable Z ~ N(p,, E,). The temporal evolution of the data is governed by a transition function
2t11
(Z,) A bb, (I,...,
and each Z, follows Gaussian distribution Z, Ni,, E,) where A E DRDx D' E Solving the optimizing problem , E {l,.... T - }} can lead to the solutions reaching optimal mapping f with t > (Experiment results shown in Figure Ic Id
6
Experiment
6.1
Multivariate Gaussians with tractable MI
Sequential data with known correspondence We sample data sequences (2 TT}.
2T p 1 /1- p²c, with correlation E [-1,1], 1),, Z, (),,

---

170
120
180
1808 2000 30000 1803 SRO
38.9 3800 30XX 4000
1000 20000 38009 4000 5000
10K6 2000 3900 0859 5000
(a)
(b)
(c)
(d)
Figure 1: (a-b) Performance of (a) EVORATE / (b) concat and separate critic on mutual information estimation on sequential data with correspondence. (c-d) Performance of EvoR ATE on mutual information estimation on two consecutive time steps without correspondence, where 9 is (c) identity function / (d) neural nets.
13. Given the correlation coefficient, p and dimensionality D 128, we can compute the ground truth MI as EvoRate Z ZT) = (2) Im(1 - p²). The optimal MI estimation can be achieved when sequential model f equals the ground truth model f Avg, where Avg(.) is an average operation. In Figure la and [b] we increase P over training steps to show the estimator behavior depends on the true mutual information. Additionally, we experiment with two forms of architecture: separable and joint. Separable architectures independently map the representations of history states /Z and the future state Zr to an embedding space with neural nets Φ1 and 22 separably, and then take the inner product, i.e. $1 (f(Zz (z in [37]. Joint critics concatenate each f(z," ) Zr pair before feeding it into the network, i.c (([ Zr]) as in []. In this experiment, is set to an identity function, and the sequential model f is set to an LSTM [5)) All networks are fully-connected networks with ReLU activations. Figure Ta shows the estimated mutual information by EvORATE over the number of iterations, and square error metric can let f converge to f such that the EVORATEw converges ground truth mutual information. Figure 1b verifies that the square error metric has a lower bias compared to trainable concat critic and separable critic.
T1...,7-1},
[Tb]
Sequential data without known correspondence We sample data sequence { e (T....,T-- 1}. R++1 = p(A'z 2₁ + 6*) + VI - pPe, where A' E DDD is a rotation matrix, 8" "E _R is translation vector, correlation of p e [-1, .]. ~ (N, T), and Z, ~ (O I). Given the correlation coefficient p and dimensionality D 128, we can compute the ground truth MI value EvoRale(Ze: 1++1) - (D/2) In(1 - p²). The optimal MI estimation can be achieved when actually
sequential model f equals the ground truth model f A'z very difficult to estimate mutual information without correspondence. As a result, the estimations by joint and separable critic do not converge and fail in the case without correspondence, which further shows the square error metric shows better performance than the trainable neural nets critic. In Figure Tc ld. g being an identity function estimates a higher value than g being a neural-nets encoder. It is noted that EvoRATEw is the only method able to estimate the mutual information without the correspondence between timestamps, achieving a reasonable performance estimate MI.
In this experiment,
6.2 Sequential data's order approximation and feature selection
Order Approximation We sample data with 5-order (k = 5), and dimensionality D = 5, which means Ze+1 is determined by Z 4- More specifically, the data generated by the dynamic function b", where in this experiment, vec(-) is vectorized operation, A E 25525 and b* E 1R5 We vary & E {1, 3, 5, 12, 24) measure the EVORATE between Z and Ze+1- Figure 2a shows that A: 5 has the maximal EvoR value. In another experiment, the time series forecasting task is used to verify the effectiveness of EVORATE. Time series forecasting performance is evaluated with the sMAPE metric [35], measured as the mean absolute error scaled by the magnitude of the predictions and target. The performance shown in Table [ is the SOTA method (53) and they set the model with order & = 45. The order is set as A: E (TO, 25, 45, 90, 180, 270}. Although 270 achieves the highest EvoRate, the difference between k = 270 and k = 90 only 0.03, and the performances over average (AVG) sMAPE have the same prediction error. For 44-Weekll,
Zi+1 A (ec
7

---

1.0
0.50
0.5
0.25
025
MM matta
-025
sum fest-2
sun, Satted
-050
sum 1est
compoee.
-0.75
tum Snste5
tun 2osted
compo-6..5
compt-6.5
50de
10000
15000
20200
5002
10200
15000
20300
2000
6100
6000
8920
10000
(b)
(c)
Figure 2: (a) k-order EVORATE estimation. (b) EVORATE estimation on a different number of features. (c) EVORATE estimation of the video prediction tasks with different comuption rate.
k 90 achieve good performance. Although EVORATE is slightly higher for k = 270 than k = 90, it sacrifices three times more computation consumptions compared only +00.3 EvORATE gain the model time complexity is (k)). Forecastability (ForeCA) fails in this experiment, as shown in Table since the longer time series shows smaller forecastability but achieves smaller sMAPE and better performance. Longer sequence can have more evolving patterns in different frequencies combined and result in smaller forecastability, but it may be more easily predictable once the patterns are learned by the model. Therefore, we conclude that the entropy used by ForeCA is not good indicator of the capability of predictions while MI used by EvOR is. addition, randomness a critical factor for capability of the predictions the sequential data. Since one of the evolving patterns is learned by sequential models, the performance only relies on the randomness of the data, which can be regarded as unwanted noises or unobserved factors.
EVORATE shows order
Table 1: Time series forecasting (TSF) tasks: M4-Weekly The values of EVORATE and time series forecasting performance below are experiments on dataset M4 Weekly. Here, short, medium, long, Avg stands short- horizon sMAPE, medium-horizoon sMAPE, ong-horizon sMAPE, and the whole average sMAPE.
ORDER:K
SHORT
MEDIUM
LONG
AVG
EVORATE
FORECA
8.28
10.13
11.44
10.06
1.98
0.50
25
5.78
9.82
10.85
8.97
2.07
0.39
45
5.69
8.80
8.52
7.74
2.11
0.33
90
5.48
5.92
7.22
6.28
2.55
0.27
5.40
6.41
7.39
6.47
2.56
0.22
270
5.47
6.39
6.84
6.28
2.58
0.19
Feature Selections For autoregressive tasks, poor predictions may due to the lack of the features. Some features may be redundant and some may be unrelated predictions. Others may be related to the task but are not put as input fed into the prediction model. The synthetic data has dimensions, where the first 3 are useful, the fourth is redundant and the fifth is unrelated (Details Appendix (B4]. Figure 2b shows the EvORATE of the data sequence with the first features. The results show that EVORATE achieves the highest value with the first three features. ii) the first four features containing one redundant feature sees a minor performance drop, and iii) using all five features sees a larger drop.
6.3 EvoRate as a criterion for existence of evolving patterns
In some problems, data is sampled independently from the history observations 44 2] In this
case, we suggest directly learning a model using ERM 44 for i.i.d (independent and identically distributed) or IRM 2 for data sampled independently but with distribution shifts. In many machine learning applications, data predicted in an autoregressive manner by training sequential models (50] 9 [6, 33). Whether to ERM/IRM sequential models directly depends on the existence of the evolving patterns. Therefore, we take EVORATE as the criterion for the existence of evolving patterns.
Multivariate time series In Table 2 EvoR can achieve better estimates of the evolving patterns compared to ForeCA, where stronger evolving patterns indicate smaller regression errors using

---

sequential models. Specifically, for M4-Monthly and M4-daily, ForeCA shows equal values but EVORATE shows higher values for M4-Daily, consistent with experimental results which M4-Daily achieves lower sMAPE.
Table 2: Time series forecasting (TSF) tasks: The estimated mutual information for the sequential data for different datasets. RMSE (Crypto, Player Traj.V/MAPE (M4-Monthly. M4 M4-WWekkyy M4-Daily) is the performance of one SOTA TSF method [8)
CRYPTO
PLAYER TRAJ.
M4- MONTHLY
M4- WEEKLY
M4- DAILY
RMSE/SMAPE 6.91 + 0.01
1.16 + 0.01
11.93
7.25
2.99
FORECA
0.35
0.49
0.44
0.43
0.44
EVORATE 2.80
4.67
1.58
2.25
2.26
Evolving Domain Generalization (EDG) follows setting in Section 5.1 where the corresoon- dence is intractable and we aim to learn the evolving patterns to predict yt., conditioned on input Tt, for every sample 2,, (Ir,, V/.s 42 [7]. In Table 3 show the performance of EVORATEW
and SOTA performance for invariant learning 2 and evolving learning [42 57 [9). Although,
performance of the evolving representation learning only depends the existence of the evolving patterns (shown by values of EVORATEWE use sequential models. For example, PORTRAITS hahs the lowest EVORATEN, ATEW 0.25 and show the smallest improvement 3.7% of the performance of evolving learning than invariant learning, and RGaussian has the highest EVORATEw 1.58 and show the largest improvement of the performance as 50.2%.
EvOR ATEW is still critical factor in deciding whether
Table 3: The estimated mutual information for the evolving domains for different datasets. The reported results are average accuracy of multiple target domains.
RGAUSSIAN
CIRCLE
SINE
RMNIST
PORTRAITS
CALTRAN
POWERSUPPLY
INVARIANT (ACC:%)
47.5
51.3
63.2
39.0
85.4
64.1
70.8
EVOLVING (Ace:%)
97.7
73.8
71.4
46.4
89.
70.6
75.7
ACC ACC.vv (%)
50.2
22.5
8.2
7.4
3.7
6.5
4.9
EVORATEW
1.58
0.58
0.54
0.95
0.25
0.28
0.46
6.4 Control randomness to corrupt evolving patterns
Video prediction aims to predict future video frames from the current ones. In this exper- the KITTI iment, we evaluate EVORATE driving videos dataset [20). which contains 28
with a resolution of 375x 1242. 24 videos in KITTI dataset are used for training. We ver- ify the performance of EVORATE by shuffling the index of the sequential data with certain corrupt probability, and this randomness will decrease the evolving patterns (Figure3) 3).
the Park
Figure 2c shows that by increasing the corrup- tion rate to the video sequence, EVORATE ex- hibits a lower value. This is consistent with our intuition, which is that the continuous video stream shows higher patterns compared the disordered video clips.
Figure 3: Illustration of corrupting the video's evolving
patterns by randomly swapping the frame.
6.5 Algorithms improve performance on EDG tasks
From the intuition that Lw can estimate the joint distribution, we apply this loss to learn a transition model based on the estimated joint distribution between two consecutive domains. Table shows this is an efficient method and achieves improved performance on EDG tasks. Our method shows an 11.7% higher average accuracy than the second-best baselines.

---

Table 4: The comparison of the classification accuracy (%) between our and baseline methods across the
synthetic and real-world datasets. The reported results are the average accuracy of the multiple target domains.
ALGORITHM MIXUP [55
IRM 2
CORAL (47
DIVA (28
LSSAE 43
DRAIN 6
OUR METHOD
RMNIST
44,9
39.0
44.5
42.7
46.4
43.8
48.5
RGAUSSIAN
55.4
47.5
53.0
56.6
48.7
61.0
91.2
POWERSUPPLY
70.8
70.8
71.0
70.8
71.0
71.3
AVG
$7.0
52.4
56.2
56.7
55.4
58.6
70.3
7 Conclusion
In this work, we propose EVORATE to qualitatively estimate the evolving patterns for the data sequences and the data snapshots from multiple consecutive timestamps without correspondences. We show the square error metric can be both a better critic for mutual information estimation, and well-designed loss to help the optimal transport plan converge to the real joint distribution and the sequential model converge the latent dynamic governing function. EVORATE reflects the complex patterns for high-dimensional data and more computationally efficient than directly evaluating performance of sequential data predictions. Experiments show EvORATE is effective measure for evolving patterns and has the potential for many applications in the machine learning area.
10

---

Acknowledgements
We appreciate constructive feedback from anonymous reviewers and meta-reviewers. This work is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), Discovery Grants program.
References
[1] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2016.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint 1rXiv:197.0028820 2019.
[3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. International conference on machine learning, pages 214-223. PMLR, 2017.
[4] Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting se- quential data using consistent koopman autoencoders. In International Conference on Machine Learning, pages 475-485. PMLR, 2020.
[5] Adrià Puigdomenech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517, 2020.
[6] Guangji Bai, Chen Ling, and Liang Zhao. Temporal domain generalization with drift-aware dy- namic neural networks. The Eleventh International Conference on Learning Representations, 2023.
[7] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez. Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 2018.
[8] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 531-540. PMLR, 10-15 Jul 2018
[9] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
[10] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances neural information processing systems, 29, 2016.
[11] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pages 1779-1788. PMLR, 2020.
[12] Thomas M Cover. Flements of information theory. John Wiley Sons, 1999.
[13] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[14] Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The uer time series archive. IEEE/CAA Journal of Automatica Sinica, 666): 2019.
11

---

[15] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 2916):141-14,, 2012.
[16] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 04000, 2018.
Ming-Wei
[17] Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 4171-4186, 2019.
[18) Chelsea Finn, lan Goodfellow, and Sergey Levine. Unsupervised learning for physical in- teraction through video prediction. In Advances in Neural Information Processing Systems, volume 29, pages 64-72, 2016.
[19] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boishunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021.
[20] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11): 2013.
[21] Matthias Gelbrich. On a formula for the 12 wasserstein metric between measures euclidean and hilbert spaces, Mathematische Nachrichten, 147(1):185-203, 1990.
[22] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain gen- eralization for object recognition with multi-task autoencoders. Proceedings of the IEEE international conference computer vision, pages 2551-2559, 2015.
[23] Shiry Ginosar, Kate Rakelly, Sarah Sachs, Brian Yin, and Alexei Efros. A century portraits: A visual historical record of american high school yearbooks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 1-7, 2015.
[24] Georg Goerg. Forecastable component analysis. In International conference machine learning, pages 64-72. PMLR, 2013.
[25] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 173 1997.
[26] Judy Hoffman, Trevor Darrell, and Kate Saenko. Continuous manifold based adaptation for evolving visual domains. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 867-874, 2014.
[27] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016
[28] Maximilian Ilse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pages 322-348. PMLR, 2020.
[29] Xu Ji, Joao Henriques, and Andrea Vedaldi. Invariant information clustering for unsuper- vised image classification and segmentation. In Proceedings of the IEEF/CVF international conference on computer vision, pages
2019.
[30] Takeshi Koshizuka and Issei Sato. Neural lagrangian schrödinger bridge: Diffusion modeling for population dynamics. The Eleventh International Conference on Learning Representations, 2022.
12

---

[31] Matt Kusner, Yu Sun. Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International conference on machine learning, pages 957-966. PMMR, 2015.
[32] Aodong Li, Alex Boyd, Padhraic Smyth, and Stephan Mandt. Detecting and adapting to irregular distribution shifts in bayesian online learning. Advances in neural information processing systems, 44:6816-6828 2021.
[33] Bryan Lim, Sercan O Ank, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable holti-h time series forecasting. In International Conference on Machine Learning pages 6315-6326, 2021.
[34] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. In Proceedings of the Association for Computational Linguistics (ACL), 2020.
[35] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54- 74, 2020.
[36] David McAllester and Karl Stratos. Formal limitations on the measurement of mutual infor- mation. In International Conference on Artificial Intelligence and Statistics, pages 875-884. PMLR, 2020.
[37] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1807.033803 2018
[38] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
[39] Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based mutual information criteria of max-dependency. max-relevance, and min-redundancy. IEEE Transactions on pattern analysis and machine intelligence, 27(6):1226-12333 2005.
[40] Ali Pesaranghader and Herna L Viktor. Fast hoeffding drift detection method for evolving data streams. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part II 16, pages 111. Springer, 2016.
[41] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ Machine Learning. 1165-6)3355-07, 2019.
[42] Tiexin Qin, Shiqi Wang, and Haoliang Li. Generalizing to evolving domains with latent structure-aware sequential autoencoder. In International Conference on Machine Learning, pages 18062-18082. PMLR, 2022.
[43] Tiexin Qin, Shiqi Wang, and Haoliang Li. Generalizing to evolving domains with latent structure-aware sequential autoencoder. In International Conference on Machine Learning, pages 18062-18082. PMLR, 2022.
[44] Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Llorenę Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Advances in Neural Information Processing Systems, volume 31, 2018.
[45] David Silver, Satinder Singh, Doina Precup. and Richard S Sutton. Reward is enough. In Artificial Intelligence, volume 299, page 103535. Elsevier, 2021.
[46] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual informa- tion estimators. In International Conference on Learning Representations, 2019.
[47] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 443-450 Springer, 2016.
13

---

[48] Alexander Tong, Jessie Huang, Guy Wolf, David Van Dijk, and Smita Krishnaswamy. Trajecto- rynet: A dynamic optimal transport network for modeling cellular dynamics. In International conference on machine learning, pages 9526-9536. PMLR, 2020.
[49] Vladimir Naumovich Vapnik, Vlamimir Vapnik, al. Statistical learning theory. 1998.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in newral information processing systems, 30, 2017.
[51] Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
[52] Ruben Villegas, Jimei Yang. Seunghoon Hong, Xumin Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In International Conference on Learning Representations, 2017.
[53] Rui Wang, Yihe Dong, Sercan Ô Arik, and Rose Yu. Koopman neural forecaster for time series with temporal distribution shifts. arXiv preprint arXiv:2210.03675, 2022.
[54] Yunbo Wang. Zhifeng Jiang, Xingjian Wang, Junyuan Gao, and Yilun Xiong. Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 5123-5132, 2018.
[55] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint (xXi::201.000677 2020.
[56] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model- offline policy optimization. In Advances in Neural Information Processing Systems, volume 33, pages 14129-14142, 2020.
[57] Qiuhao Zeng, Changjian Shui, Long-Kee Kai Huang, Peng Liu, Xi Chen, Charles Ling, and Boyu Wang. Latent trajectory learning for limited timestamps under distribution shift over time. In The Twelfth International Conference on Learning Representations, 2024.
(58) Qiuhao Zeng, Wei Wang, Fan Zhou, Charles Ling, and Boyu Wang. Foresee what you will learn: Data augmentation for domain generalization in non-stationary environment. Proceedings of the AAAI Conference on Artificial Intelligence, 37(9) 1147:1114--11155,, Jun. 2023.
[59] Qiuhao Zeng, Wei Wang, Fan Zhou, Gezheng Xu, Ruizhi Pu, Changjian Shui, Christian Gagne, Shichun Yang, Charles X Ling, and Boyu Wang. Generalizing across temporal domains with koopman operators. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 16651 2024.
14

---

A Proof of Theories
Proposition 1. Assume that the model's probability density follows Q(1++177
N(Ze+i p(1; (1t-k41).pp)
fille fims + const
(12)
Proof.
D log(2n) Lmse + const
Liie Ep Q = [1F(E) 2411l
(13)
2
Proposition 2. For autoregression tasks, the expected risk satisfy:
Lille (1 -++x+1) H(z+1) + DKI [ Zz QZrr1) (Z)
(14)
Proof.
Line
-E,+1 4--++ ~ P P
241) log Q(Z++1 [[
(15)
--E,t] -- P 2441) log Q(Z++) ((1))
P(Z Z k441)
E1 4/ ~ P B(41) log ((z+11 (Z
(16)
k++
P(Z (2;
=Dku(P(Zo+1Zi (Z) ((1))) ((++1)
E E,441 P(7 (11mm) log
p(Z(+))
- -B,t,_, - 411........................... 2411) log (()41)
(17)
Dxu((P(2441/iis4t)))) - x+))
1(2+11 Zi-
x+1) H(z++))
(18)
8-mixing is measure of the degree of dependence between random variables in a sequence over time, which is closely related to MI and furthermore upper bounded by MI: Remark E N, B-mixing coefficients defined in below satisfy:
8(s) sup Bzz
Per PffalltT - sup ( 221(Zztsi
(19)
where ||
|| is the maximum total variation distance.
Proof.
B(s) sup Ez
[Pts( (- - P.PslITV
< sup Ez:
(20)
suup
/Ex (2
sup 21 22 : Z'
oo)
(21)
where the first inequality follows Pinsker's inequality; the second inequality follows Jensen's Inequal- ity.
Lemma 1. P(X.Y) Y) the real underlying distribution, and (X, the optimal transport plan
that satisfies both margins comply with P(X) and P(Y). We first define:
Ix(X;Y) / p(x,)) v) log
p(x.v) dx dy
(22)
p(z)p(y)
(X:Y)) = / T(z, v) log p(x))(()
TT v) dr dy
(23)
15

---

Then, we have
-1, - HM)())
(24)
Therefore, we can get Ip(Z; Zz+1) - (21; Zx+1) = (2, -H((zi
Proof.
-(x;))-((x:1) p(z, v) log p(r,y) dx dy / x(xx)) log p(z)p(y)
[(x,y) dr dy (25)
p(z)p(v)
= = p(x, v) pla, y)de - T T(æ, v) log (x, y) dy -
f p(a, u) log pl )r dy
/ p(a, u) logrly) da dy / x ((x,))) log logp(s))da dy + / It ( log log ply) do dy
(26)
= / p(z, y) log (a. y) dy - - π(Ā y) log (2, y) dy - / p(x) p(z dr dy
/ p(y) log ply) dy + / p(x) log())o))) dy + / p(u) log ply)dz dy
(27)
Hx, - H)
(28)
(X(Y) Hp(X(Y) = HY H2(YIX)
(29)
26 to 27 is due to p(x, y) and (Ix y) have the same margins p(x) and p(y) and we integral over x or
y first.
Lemma 2. Let Piz,, Zi+1) be the ground truth joint distribution. If f attains f*, then
n' (Ze, Ze+1) = P(
(30)
Proof. To finish the proof, we first assume 2441 ~ T (Ze, Ze+1), 2441 ~ T TZe, Zc+1) and <~ NN00 o²I), then,
inf
(w((T,)=
inf = ((((Z)) P(Z. 1))
E B( )~m fl -24 2
" E M(P(Z)) P(Z. 1))
inf
E Be 1(2 -c) - 1)12
= P(2
/ [1 2241113 dx ( Zx+1)
inf
(31)
a I(P(2)) (((2)
226 26" 202((16 (24+1- f
- -241) (Ze, Zet1) + / ||el12 dr dz Zo-1)
If |1 z++1 112 dm (Ze, Zet1)
inf E I(P(Z.). P(Z(11))
(32)
/ 26 26T (zi+1 f (2x) + E) dr dr Zc+1)
(33)
+ / Ilel2s dn (Ze, Zori)
If [IEice Z242112 dr (2e, Z(41)
inf E I(P(Z)))
(34)
- 1 26 20 (21+1- ff((z)) da (Ze, Zetz) - / [1e|13 dilld, ((t1))
Since 1I 1441-11(a0
/ [1é441 -1+1112 (Ze, Ze+1)
inf fw( t E (()(())) E (2),)))
inf
+ const
To achieve infimum of [t)( 7/41 Z441 should satisfy and hence " *Z P(Z++1 (Z)) and x² (Zi+1, Zi) = P(Z Zi) with feasible solution.
16

---

Example We consider data collected from multiple time steps where each sample is vector Z, RD. Specifically, the initial data points at the first timestamp modeled Gaussian variable Z1 ~ N(m, E1). The temporal evolution of the data is governed by a transition function:
1 (Z,) A ã5 + ,", E {I....,T).
and each 7z follows a Gaussian distribution, Z, ~ M Ex), where A E RDxD, b' E RD. Solving the optimizing problem (n f), E {1, [1,....: T 1} lead to the solutions reaching optimal mapping f with > 1.
Proof. Assume the linear transition function has parameters (A, as 2++1 f(Zz) Az b, can be inferred that Z++1 ~ (Ap, + b, AE)A".
For each pair of data from two consecutive timestamp data, the Wasserstein distance loss can be expressed as follows, according to [2
Wasserstein Distance loss = inf Lw(() = 111e -App-1+ + b)1
Tr |E + E, - -(xx AE 12"
Then, have
inf Cw(() = 11ke - (Ap4-1 + b]]l +Tr
AΣ,-1A" +2x
2TT(x; AΣ AE
= -1lue - (Au-1 + b)11 TT
AΣ +Dx
(Tr 1(A=)1))
=Ilme (A14-1 + b)1b3 Tr
AE +5x - -2(1" ((E,)))),_))
-lle (Ap4-1 b)lb3 TT
AE +I1 12Tr(A -ATT(A'
-1ee (A 6 +Tr
AΣ +20 -
|| (Am4- b)112 Tr (ATE,A - -
1| - (Aut11 + b)113 + +1AA E.A Σ.-IlFF
where || - || is Frobenius norm. inf, (t) = 0, and the infimum is attained when Av-1 b Ft A'E,-AA E, -2,=.......
(35)
We dealing with system where the matrix A and b together comprise X (n + 1) unknown variables. At each time step f, the system is described by n x n quadratic equations and n linear equations. Typically, these quadratic equations yield two possible sets of solutions. To refine our estimates and converge towards the optimal parameters (A*,, b²), employing a large number time steps (t > 1) allows us formulate overdetermined system of equations.
B Datasets
B.1 Multivariate Gaussians: sequential data with known correspondence
We sample data sequence {2 (2)) 2r Ed' T-1 = VI- Pee, where correlation
of p [-1, 1), c A/(0, I), Z₁ ~ (),, (1. f1....,T 17. Given the correlation coefficient p
and dimensionality D 128, we can compute the ground truth MI value EvoRate (74)" 11; 7T) = -(D/2) )²))
17

---

B.2 Multivariate Gaussians: sequential data without known correspondence
We sample data sequence { {l,...,I' - 1), 2++1 p( + b*) A" E RDx is rotation matrix, 6" E DR is translation vector, correlation of p [-1,1], e ~ (0, and Zı ~ ~N( I). Given the correlation coefficient p and dimensionality = 128, we can compute the ground truth MI value EvoRate(Zr: Ze+1) (D/2) In(1 (n).
-I-p²c, where
B.3 Synthetic data for order approximation
We sample data with 5-order (k 5), and dimensionality D = 5, which means Z,+1 is determined by Zi More specifically, the data generated by the dynamic function Ze+1 = vecec((2 (Zi- +b. where in this experiment, vec(.) is a vectorized operation, A" 52525 and b E R$. In this experiment, we set {1,3, 5, 12, 24) measure the EVORATE between Z Z and Z
B.4 Synthetic data for feature selection
In this experiment, We sample data with 5-order (k 5), and dimensionality D 5. Only first three features are decided by the past states: Zi+1|: 3] = A veve Zi-al:: 3)) + b", where in this experiment, vec(.) is vectorized operation. A E R 15, and 6* E R². The fourth feature is linear combination of the first three dimension features, as redundant feature. The fifth feature is purely noise following the normal distribution.
B.5 Time-series forecastings
M4 [35 contains 10000 highly nonstationary univariate time series with different frequencies from hourly to yearly and different categories from financials to demographies. The forecasting horizon varies across different frequencies.
Crypto This multivariate dataset contains 8 features on historical trades, such open and close prices, for 14 cryptocurrencies. The original challenge is to predict 3-step ahead 15-minute relative future returns. Since focus long-term forecasting, we train all models make 15-step predictions of 15-minute relative future returns. We use the original training set from the competition and do 19%-10%-109 rrining-validaaiion-est split.
Player Trajectory 32 contains basketball player movement trajectories from NBA games in 2016.
We randomly sample 300 player trajectories for training and validation and 50 trajectories for testing. All models are trained to yield 30-step ahead predictions
B.6 Evolving domain generalization
Rotated Gaussian (58) consists 30 domains generated by the same Gaussian distribution, but the decision boundary rotates from 0 to 338" with interval of 12". We split the domains into source domains (1-22 domains), intermediate domains (22-25 domains), and target domains (26-30 domains). The intermediate domains utilized as the validation set.
Circle 40 contains evolving 30 domains where the instance are sampled from 30 2D Gaussian distributions. The label is assigned using a half-circle curve the decision boundary. (15 source domains, validation domains, and target domains)
Sine In Sine 40 each data owns two attributes (2). The label is assigned using sine curve as the decision boundary. We rearrange this dataset by extending 24 evolving domains. Each domain covers 21 the period of the sinusoid. (12 source domains, 4 validation domains, and 8 target domains)
Rotated MNIST (RMNIST) 22 is an adaptation of the popular MNIST digit dataset 15 composed of MNIST digits of various rotations. The task to classify digit from 0 to given image of the digit. We follow 43] and extend it to 19 evolving domains via applying the rotations with degree of (0', 15 30 ...., 180") in order. (10 source domains, validation domains, and target domains).
Portraits 23 is real-world dataset that comprises photographs of American high school seniors collected over period of 108 years (1905-2013) across 26 states. The objective is to accurately
18

---

classify the gender for each photograph. The dataset is divided into 34 domains based on fixed interval over time. (19 source domains, validation domains, and 10 target domains)
Caltran [26] consists of real-worr world images captured by fixed traffic camera deployed in an inter- updated at 3-minute intervals each with resolution 320 320. We divide it into 34 domains by time. The task of Caltran is to classify scenes to identify the presence of or more vehicles in or approaching the intersection. The challenge mainly raise from the continually evolving domain shift as changes include time, illumination, weather, etc. (19 source domains, validation domains, and 10 target domains)
section time. Frames
PowerSupply [14] dataset designed for the task time-section prediction of current power supply based on hourly records obtained from an Italian electricity company. The dataset consists of 30 domains formed according to days. Each data point is assigned a binary class label indicating whether the current power supply belongs the morning the afternoon. Domain shifts may arise due to variations in season, weather, price, or the differences between working days and weekends. (15 source domains, 5 validation domains, and 10 target domains)
C Full experiments on EDG tasks
In this section, we present complete experimental results to validate the efficacy of our proposed evolving domain generalization task.
Table RMNIST. We show the results on each target domain by domain index.
ALGORITHM
130"
140
150
160º
170"
180
AVG
MIXUP
61.3
0.7
47.4
1
0.8
39.1
38.3
40.5
0.8
42.8
1
0.9
44.9
IRM 2
47.7
0.9
0.9
38.5
0.7
34.1
0.7
35.7
0.8
37.8
0.8
0.8
40.3
1.8
39.0
CORAL 47
58.8
1..80099
0.9
46.2
0.8
18.9
38.5
0.8
41.310.
4
44.5
DIVA 28
58.3
0.8
37.6
0.7
3..1
40.1
42.7
LSSAE
43
0.8
51.6
4.8
3.7
4.8
40.4
46.4
DRAIN
6
59.5 0.8
45.4
0.8
40.2
37,2
0.7
39.6
0.8
41.0
43.8
OUR METHOD
65.5 0.6
55.9
+
0.8
+
0.8
41.8
0.9
10.1
40.3 + 0.8
48.5
Table 6: Rotated Gaussian. We show the results on each target domain by domain index.
ALGORITHM
26
27
28
29
30
AVG
MIXUP
56.2
1.5
63.4
1 3.0
56.8 + 1.4
49.4 +
1.5
41.4
+ 2.0
55.4
IRM
56.8
1.9
55.8
3.1
51.8
2.3
41.6
1.6
31.4
+ 2.1
47.5
CORAL
54.8 1.6
54.0
53.8
52.0
50.6
+ 1.6
53.0
DIVA
1.5
5.9
5.7
59.2
55.6
13.6113 1.5
56.6
LSSAE
50.6 0.9
5..
4.4
48.4
50.4220
48.7
DRAIN
73.2
2.9
1.7
2.4
53.2
45.0
+ 1.2
61.0
OUR METHOD
98.0
- 0.6
94.6 0.9
98.0
9.94
73.2
+ 0.8
91.2
Table 7: PowerSupply. show results on
target domain by domain index.
ALGORITIM
21
22
23
24
25
26
27
28
29
30
AYG
MIxUP
69.5
64.3
70.1
69.2
1.3
68.1
65.0
IRM
69.5
69.5
68.
64.1
87
76.5
70.0
1.5
69.1
68.2
65.0
70.8
CORAL
69.921
69.7
68.9
1.4
64.6
1.4
86
10
76.3
1.3
70.0
69.5
68.5 1
65.7
71.0
DIVA
69.7 1.4
69.5
1.3
682
1.4
63.9
1.5
87.5
14
76.5
1.3
69.9
69.1 -
1.5
68.1 1.3
64.1 -
70.7
LSSAE
70.0 1.4
69.8
09.0
65.4
+
70.1
69.0
66.3
69.8
68.9
OUR METHOD
69.4 1:
69.3
68.2
64.1
86.2
1.
75.8
1.4
70.3
1
70.8
1.4
70.0
68.6
1.0
71.3
19

---

D Computational Cost
All experiments are carried out on 498G memory, x AMD Milan 7413 @ 2.65 GHz 128M cache L3, and 2 x NVidia A 100SXM4 (40 GB memory). The algorithmm's 's computational cost is the cost of OT (using package [9] O () and the cost of estimation of MI ((2d)), where the batch size in an iteration, and d is the representation dimension. In all, the total computational cost is O(B'd + B (not counting the encoder g and decoder h).
E Algorithms Training Procedures
Algorithm EVORATE: Data is sampled in sequential manner with correspondence
for each training iteration do
2: Sample [= } from p(z]-1). where B is the batch size per training iteration
3: Compute EvoRate of i-th sample at timestamp according to Eq (4), k <
EvoRater, := .(2..)...........).. 9(
log
4: Update / and by maximize B(T EtEK 1= EvoRate,,j
5: end for
Algorithm 2 EVORATEw: Data is sampled from different timestamps but without correspondence
1: for each training iteration do
2: Sample {= T from p(zf=)). where B is the batch size per training iteration 3: Compute the optimal transport plan I², where (w f) defined in Eq (7)
x" (Z,, Z441)=
arg min (T))),)) P(Z+41)
(w,,,s), VE (L...,,--11
4: Compute EvoRatew of i-th sample at timestamp / according to Eq (9), especially (2r,5, #441,)) is sampled from n
B Σ
(EvoRatew)ss -1](((())))
( log B 5=1,
5: Update / by maximize B(T-I) = 1== (EvoRatew)s.
6: end for
F Comparisons with Traditional time-series statistic indicators
There are traditional statistic indicators for time series, but they have significant limitations: a) System sensitivity, measured by Lyapunov Exponents (LEs) [2] measures how sensitive dynamic system to initial conditions, b) Trend is the slope of a linear regression fitted to sequential data, and c) Scasonality, measured by the ACF test [3]. assesses linear correlations between observations at different time lags. These methods are not designed to measure evolving patterns and struggle to handle high-dimensional data. Each method measures only one aspect of evolving patterns: system sensitivity, trend, or seasonality. Together, they determine the overall evolving patterns. We present a comparison of these metrics with our method below:
In the above table, a larger EvoRate consistently indicates a smaller potential prediction error (RMSE//MAPE) for the dataset. In contrast, LEs, trend, and seasonality show little impact on the prediction errors. Another significant drawback these methods is their inability to be directly applied to high-dimensional data, such images, vidcos, and NLP datasets.
20

---

Table Comparison of Different Metrics Across Various Datasets
Crypto
Player Traj.
M4-Monthly
M4- M4-Weekly
M4-Daily
RMSE/SMAPE
6.91
1.16
11.93
7.25
2.99
L.Es
0.026
0.052
0.011
0.013
0.020
Trend
0.02
0.01
0,48
0.13
0.05
Seasonality
0.00%
0.00%
66.34%
0.00%
0.00%
EvoRate
2.80
4.67
1.58
2.25
2.26
G
Limitations
Due to computational resource limitations, have not included experiments involving Natural Language Processing (NLP) tasks on Large Language Models (LLMs) our study. These models typically require extensive processing power and substantial memory, which exceed our current hardware capabilities. Additionally, the high costs associated with running these models make them impractical for our budget. Instead, focused alternative datasets and models that align with our available resources. We believe that our chosen datasets still provide valuable insights while remaining within our operational constraints. Future work could explore LLMs as computational capacity expands.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims are in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: see section G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification: see section A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See the experiment section and dataset section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[Yes]
Justification:
See dataset section and supplementary files of codes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.. cc/
pub i quides Codeeubmies ss casionoollii icy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. cc/
wwwblic/guideee / //odiddee/Cdeeeeeeee Cooeeeooeeeeeeeeee SudeSuumiiee wwwmisssiiii ssssnooiiccyyyycco for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See the experiment section and dataset section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: see the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See computational cost section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Bttgghtttosee./rrr....o/oo/////a
Question:
Answer: [Yes]
Justification:
We preserve the anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work is not an application, but foundational research on machine learning. Hence, there is no negative social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All related works mentioned in related works.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetttett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets documented in the experiment and dataset sections.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "ngEKyHDdsq", "reviewer": "Reviewer_8iST", "summary": "This paper aims to identify the evolving pattern in sequential data. In addition, given evolving pattern may present in the sequential data, this paper would like to introduce a technique that can identify the best temporal order and features for learning the sequential data.\nTo address that, this paper proposed an indicator Evolving Rate (EvoRate), by borrowing the ideas from existing works in Mutual Information (MI) Estimation. The effectiveness of EvoRate is tested by experiments on multivariate time series forecasting, video forecasting and evolving Domain Generalization (EDG) tasks.", "questions": "Apart from comparing to existing works in MI, do you also consider studies working on evolving patterns but not using MI? How to use the results of accurate identification of evolving patterns? This paper shows the results of the estimated EvoRate compared to MI results. However, it is less clear to me what the real benefit of introducing MI into solving the evolving pattern issue.", "limitations": "This paper shows some interesting ideas and results, but the motivation is not very clear to me. The proposed method is an edited version of MI. However, I am not clear why it is beneficial to introduce MI to address evolving patterns compared to existing solutions.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The research question discussed is interesting.", "weaknesses": "This paper measures evolving patterns by an improved MI indicator. Some discussions and experiments are conducted to evaluate the advantages of EvoRate over existing MIs. However, the question discussed in this paper is \"How to determine the existence of evolving patterns in data sequences\" which does not limit the technical solutions to MI-based one. Therefore, I am questioning the contribution of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- W1 & Limitation (motivation is not very clear) - **Our motivations and contributions**: Our work's contribution is three-fold: \n  - It is theoretically motivated by the use of MI to estimate evolving patterns.\n  - It applies a specially designed similarity critic that considers the autoregressive manner. \n  - For cases where data is sampled at each timestamp and lacks correspondence, we propose methods to approximate the absent correspondence between timestamps with optimal transport (OT), thus enabling EvoRate estimation. \n\n- W1 & Q1 (benefit of introducing MI) - **Why we use MI**: We justify our choice of using MI in Section 4.2 and Proposition 1. Specifically, we show that **a larger MI leads to a smaller expected maximum likelihood estimation (MLE) loss, which indicates better autoregression performance**. As shown in Eq (5), the expected risk can be decomposed into two terms: the first term is related to the prediction model, and the second term, $H(Z_{t+1}) - I(Z_{t+1};\\textbf{Z}^t_{t-k+1})$, is only related to the data. Since the inequality $H(Z_{t+1}) \\ge I(Z_{t+1}; \\textbf{Z}^t_{t-k+1})$ always holds, minimizing the prediction risk in autoregressions requires MI to be as close as possible to $H(Z_{t+1})$. The insight behind achieving equality is that we can predict the future state $Z_{t+1}$ if we have complete information about the historical states $\\textbf{Z}_{t-k+1}^t$. Consequently, we argue that MI serves as a direct and reliable indicator of evolving patterns. Based on this conclusion, our experiments aim to show that EvoRate can effectively approximate MI in sequential data.\n\n- W1 & Q1 (comparing to existing works) - **Advantages over the existing work**: The problem of evolving patterns in sequential data is under-studied, and to the best of our knowledge, the only related work is ForeCA [1]. However, ForeCA relies on the spectral entropy of data and assumes linear dependency between historical information and future states. Consequently, it can only detect the cyclical pattern and cannot serve as an indicator of complex evolving patterns. Moreover, ForeCA cannot be applied to high-dimensional data, limiting its use for many real-world datasets. EvoRate addresses these issues by MI-guided efficient deep representation learning, and its effectiveness is justified in Table 1 of our paper. More discussions of the benefits of EvoRate over ForeCA are also presented in lines 106-116 of our paper. We also include three statistic indicators for time series in global response.\n\n- Q1 (how to use results of identification of evolving patterns): There are various applications: \n    - a) Model selection (sequential vs. static): By establishing an empirical threshold for EvoRate, we can determine the appropriate model for predictions. If EvoRate exceeds the threshold, a sequential model is recommended. Conversely, if EvoRate is below the threshold, indicating a lack of temporal patterns (e.g., coin tossing), a static model should be used. For example, based on the results in Table 3, we should apply a static model for the Portraits and Caltran datasets. \n    - b) Temporal order estimation: EvoRate can be used to estimate the temporal order in the data. From Table 1, we suggest building a temporal model with a temporal order of 90 can achieve optimal performance, as EvoRate values for orders 180 and 270 are very close to that of 90. \n    - c) Temporal feature selections: EvoRate can be used for feature selection for temporal prediction tasks, allowing for efficient and explainable model training by excluding redundant or irrelevant features, as shown in Figure 2-b.\n    - d) Evolving domain generalization (EDG): EvoRate naturally acts as a regularizer to enhance the performance of learning dynamics in sequential data, as verified in EDG tasks in Table 4.\n    - e) We further experiment with EvoRate on the NLP dataset Associative Recall (AR) of zoology [4]. The results indicate that a low EvoRate for the NLP dataset implies either a low prediction potential or that the dataset alone cannot be used to train a good predictive model. EvoRate is estimated between the historical sequence and its query's recall.\n\n        |Corruption Ratio|0   |0.25|0.5|0.75|1|\n        |-|-|-|-|-|-|\n        |EvoRate |2.93| 1.52|1.28|1.18|1.04|\n\n        The Corruption ratio represents the probability of shuffling the data sequence. A high ratio leads to a degradation of evolving patterns, resulting in a lower EvoRate. This is consistent with the results shown in the above table. Notably, the test accuracy of the AR dataset is normally 100%, which aligns with its high EvoRate of 2.93.\n\n- Limitation (EvoRate is edited MI): There are two major differences between EvoRate and existing MI estimation methods (e.g., [5-6]): 1) Existing MI estimation methods are designed for static data, and their concatenated and separable critic functions directly compute the similarity between embeddings in two different domains. In contrast, EvoRate leverages an autoregressive critic function to efficiently account for the structure of sequential data by mapping historical states to the next state and computing similarity in the domain of the next state. As a result, EvoRate underestimates the ground truth MI values of sequential data, as shown in Figure 1-b. 2) Existing MI estimation methods do not account for sequential data without correspondences, and hence cannot directly estimate MI in such cases. EvoRate$_W$ mitigates this issue by building correspondence with optimal transport (OT). However, even with OT, existing MI estimation methods still fail to estimate MI for sequential data due to limitations in their critic functions, as shown in Figure 1.\n\n[1] Forecastable component analysis\n\n[2] Lyapunov exponents\n\n[3] Testing stationarity in time series\n\n[4] Measuring and Improving Recall in Efficient Language Models\n\n[5] Mutual information neural estimation\n\n[6] A contrastive log-ratio upper bound of mutual information", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BAFcaHpz4Q", "reviewer": "Reviewer_rYZr", "summary": "This paper introduces EvoRate, a novel metric designed to quantify the evolving patterns in sequential data. The authors propose leveraging mutual information (MI) to measure the temporal dependencies between data points in a sequence. The paper addresses a significant challenge in machine learning: identifying and quantifying evolving patterns in high-dimensional sequential data. EvoRate is further extended to EvoRate$_W$, which uses optimal transport to establish correspondences between data points at different timestamps, facilitating MI estimation even when direct correspondences are absent. The proposed methods are validated through experiments on both synthetic and real-world datasets, demonstrating their effectiveness in various applications such as time-series forecasting, classification with temporal distribution shifts, and video prediction.", "questions": "See the weaknesses above.", "limitations": "This paper is about fundamental machine learning research and is unlikely to have any potential negative social impact.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. It is true that most researchers apply sequential models to sequential data without considering whether there are evolving patterns. Therefore, it is critical to delve deeper into understanding these evolving patterns in sequential data. The introduction of EvoRate and its extension, EvoRate$_W$, addresses a significant gap in the literature by providing a quantitative measure for evolving patterns in sequential data. This is a novel and impactful contribution.\n\n2. The paper provides analysis and examples to clarify why EvoRate can measure evolving patterns, particularly through Proposition 1. This proposition reveals that mutual information (MI) is an intrinsic property of sequential data, reflecting the evolving patterns within the data. Detailed explanations of how MI quantifies temporal dependencies and the use of optimal transport for handling data without direct correspondences are included, enhancing the understanding of this measure.\n\n3. Figure 1 (a, b) illustrates the designed MI critic function used in Equation 4, which better measures MI by considering the data structure of sequential data. More importantly, Figure 1 (c, d) demonstrates the effectiveness of EvoRate$_W$. Even in the absence of correspondence between time points, EvoRate$_W$ can still approximate the ground truth MI, showcasing its robustness and practical utility.", "weaknesses": "1. The authors refer to EDG as their algorithmic contribution in Line 70, but the detailed setting and definition of the EDG problem are lacking. A clear introduction and definition are necessary to understand the context and setup of EDG.\n\n2. There is some confusion regarding the term \"learning performance\" mentioned in Line 53. It is unclear whether this refers to the prediction tasks of sequential data or the process of training the model to measure mutual information (MI). Additionally, how EvoRate manages the trade-off between computational complexity and learning performance needs clarification. Since the mutual information measurement model also requires training, as is standard in variational MI estimators [1], why is it necessary to measure MI to reflect evolving patterns rather than directly relying on the predictive performance of sequential models (e.g., LSTM, Attention, SSM), which also require training?\n\n3. It would be beneficial to include a pseudocode table to facilitate a clearer understanding of the algorithms for the readers.\n\n4. Although the authors state they do not have experiments on NLP tasks, it would be highly recommended to apply EvoRate to NLP datasets, given that NLP is one of the most important sequential data tasks.\n\n5. The claim that the function $f$ will converge to $f^*$ in Remark 2 is unconvincing. This is because the joint distribution is not available, making it difficult to accurately learn the transition function $f$. Without access to the joint distribution, the learning process lacks the necessary information to ensure convergence to the optimal function $f^*$, especially when dealing with complex, high-dimensional real-world data.\n\n6. Synthetic experiments with known correspondences in Section 6.1, only include the average operation as the transition function. It would be more beneficial to also consider a more practical dataset.\n\n7. There are some typos that cause confusions: Table 2 should be EvoRate and Table 3 should be EvoRate$_W$.\n\n[1] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dyand Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 531–540. PMLR, 10–15 Jul 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. (EDG setting) We thank you for pointing it out and will include an illustration of the Evolving Domain Generalization (EDG) setup. A brief explanation of EDG can be found in lines 328-330. More specifically, The EDG tasks setup involves using the training dataset $D_S = \\\\{ \\\\{ x_{t,i}, y_{i,t} \\\\}^N_{i=1} \\\\}^M_{t=1}$, where $x_{i,t}$ and $y_{i,t}$ represent the data and its label of the $i$-th sample at time $t$ respectively and $N$ is the sample size, to learn a function that can predict the class of samples from future timestamps, which are then evaluated on the test dataset $D_T = \\\\{ \\\\{ x_{i,M+t}, y_{i,M+t} \\\\}^N_{i=1}\\\\}^L_{t=1}$.\n\n\n2. - (a: learning performance) Thanks for pointing it out. The term \"learning performance\" refers to the performance of approximating MI using the trained EvoRate, which is parameterized by the learned autoregressive function $f$ and the encoder function $g$.\n\n    - (b: trade-off between complexity and performance) When the original dimension of the data $D$ equals the dimension of the encoded data's embedding $d$, it provides the most precise MI estimation, comparable to the computational cost of directly training a deep autoregressive model. On the other hand, when $d \\ll D$, the estimated precision is lower but more computationally efficient. Specifically, if we directly load a pretrained model for $g$, we only need to train $f$, which is typically only composed of multi-layer perceptrons (MLPs). \n\n    - (c: why measure MI instead of relying on the predictive performance) As pointed out in (b), setting $d \\ll D$ makes EvoRate efficient in estimating MI compared to training a deep autoregressive model in the original high dimension, especially for high-resolution videos. The key is to estimate MI in the latent encoding space. We can trade off some precision of EvoRate due to the Data Processing Inequality [1] for computational cost efficiency. We provide additional experimental results with the KITTI dataset:\n\n        |     Corruption Ratio           |0                       |0.05|0.1|0.25|0.5                         |\n        |-------------------------|-------------------------------|-----------------------------|-------------------------------|-----------------------------|----------------------------|\n        |Finetune $g$ | 2.56|1.52|0.93|0.37|0.05\n        |Pretrained&fixed $g$ |2.54| 1.55| 0.92|0.39|0.10\n\n        In the above table, the Corruption ratio represents the probability of shuffling the data sequence, where a high ratio leads to a degradation of evolving patterns.\n\n\n3. (pseudocode) We have included the pseudocode in the global response PDF and will add it to the revised version of the paper.\n\n4. (NLP tasks) We further experiment with EvoRate on the NLP dataset Associative Recall (AR) in zoology [4], and the results are shown below. Due to time limit, we choose this simple synthetic dataset for our experiments. EvoRate is estimated between the historical sequence and its query's recall. A low EvoRate for the NLP dataset implies either a low prediction potential or that this dataset alone cannot be used to train a good predictive model. Notably, the test accuracy of AR is typically 100%, which is consistent with the high EvoRate of 2.93, indicating strong predictive performance..\n\n\n    |     Corruption Ratio           |0                       |0.25|0.5|0.75|1                         |\n    |----------------|-------------------------------|-----------------------------|-------------------------------|-----------------------------|----------------------------|\n    |EvoRate |2.93| 1.52|1.28|1.18|1.04|\n\n5. ($f$ will converge to $f^*$) In Lemma 1, we show that with our defined autoregressive cost function, if $f$ attains $f^*$, the estimated joint distribution $\\pi^*$ will converge to the real distribution. Furthermore, it is a common practice to estimate the joint distribution with optimal transport to build the correspondence, as demonstrated in [2,3]. However, proving the convergence of $f$ to $f^*$ is challenging, and we will try to address this in future research.\n\n6. (Synthetic experiments) For more complex transition functions, the analytical mutual information value is intractable, as there are no methods to compute the ground truth MI value quantitatively. For datasets with correspondences, we test EvoRate on real-world time series forecasting datasets Crypto, Player Traj., M4-Monthly, M4-Weekly, M4-Daily, and video prediction KITTI dataset.\n\n7. (typos) Thank you for pointing them out, and we will revise the paper accordingly.\n\n[1] Elements of information theory. John Wiley & Sons, 1999\n\n[2] Joint distribution optimal transportation for domain adaptation, NIPS 2017\n\n[3] Optimal Transport for Domain Adaptation, TPAMI 2015\n\n[4] Zoology: Measuring and Improving Recall in Efficient Language Models, Arxiv 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GXq5ZvowJz", "reviewer": "Reviewer_oZmm", "summary": "The article introduces a groundbreaking technique for measuring changes in sequential data, marking a notable advancement in the realm of machine learning. The authors bring forth the Evolving Rate (EvoRate) and its advanced iteration, EvoRate$_\\mathcal{W}$, as effective tools for evaluating the temporal dynamics within datasets.", "questions": "-I find the concept of leveraging Mutual Information (MI) to evaluate the evolution of patterns to be logical, yet I harbor reservations about whether MI can truly equate to the changes observed in sequential data. A deeper exploration of this connection would be highly beneficial.\n\n-The ForeCA method's results are counterintuitive, as they oppose the prediction performance outlined in Table 1, where one would expect the inclusion of more historical data to enhance the predictive capabilities. Can the authors elucidate why EvoRate, an estimator of MI, is deemed superior for capturing evolving patterns over ForeCA, which is based on entropy estimation?\n\n-I am curious to know if the EvoRate model, once trained, can be directly applied to new datasets without the need for retraining. If this is possible, it would make EvoRate a more adaptable and computationally efficient tool that could be easily integrated into various systems.", "limitations": "There's no limitation relating to social impact.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Tackles a real-world issue in machine learning, with possible uses in predicting time-series data, categorization, and other sectors.\n- Bridges a knowledge gap in the current comprehension and measurement of patterns that change over time in sequential data.\n- The creation of EvoRate and EvoRate$_\\mathcal{W}$ represents a creative leap forward, offering a numerical assessment of evolving patterns applicable to a range of learning tasks. The utilization of optimal transport (OT) to resolve the issue of non-correspondence in temporal data is especially innovative.", "weaknesses": "- It may not fully address the scalability of the proposed methods as dimensionality grows. The authors assert that the dimensionality reduction of the original data can lead to a decrease in computational time when calculating Mutual Information (MI). However, the process of training an auto-encoder itself demands considerable time and resources. Is this factored into their considerations?\n- The scope of the experiments might be limited, with a notable absence of extensive real-world datasets\n- The true MI values for high-dimensional data, such as in video and time series contexts, are often infeasible to obtain. How do the authors ensure that EvoRate can provide an accurate measure of MI for such high-dimensional data, especially given this significant limitation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Responses to Weakness:\n\n1. (scalability & training an auto-encoder) We have added experiments on the scalability of the method with different dimensions in the encoding space using the Video Prediction dataset KITTI.\n\n    Encoding Dim|128| 256 |512| 1024 \n    |-------------------------|-------------------------------|-----------------------------|-------------------------------|-----------------------------|\n    |EvoRate|2.19| 2.37|2.47| 2.56|\n\n    From the above table, a smaller dimension in the encoding space leads to a lower EvoRate due to greater information loss. However, the performance remains acceptable even with a very low dimension of 128.\n\n    Training an auto-encoder model from scratch can be computationally expensive. However, we can use a pretrained backbone, significantly reducing computational costs by directly learning the dynamic function $f$ from the encoded embeddings. \n\n    |     Corruption Ratio           |0                       |0.05|0.1|0.25|0.5                         |\n    |-------------------------|-------------------------------|-----------------------------|-------------------------------|-----------------------------|----------------------------|\n    |Finetune $g$ | 2.56|1.52|0.93|0.37|0.05\n    |Pretrained&fixed $g$ |2.54| 1.55| 0.92|0.39|0.10\n\n    The Corruption Ratio represents the probability of shuffling the sequence to disrupt the evolving patterns. A higher Corruption Ratio should result in a lower EvoRate. We find that with a fixed pretrained encoding function $g$, the estimated EvoRate closely approximates the result of training $g$ with EvoRate on the video dataset KITTI. The pretrained model used is ResNet-18 from torchvision, pretrained on ImageNet-1K.\n\n\n2. (Absence of real-world datasets) We have utilized real-world datasets: five time-series forecasting datasets presented in Table 2, three EDG datasets (Portraits, Caltran, and Powersupply) in Table 3, and one video prediction dataset (KITTI) in Figure 2-c. Additionally, we included the experimental result of EvoRate on the NLP dataset Associative Recall from zoology [3]. A low EvoRate for the NLP dataset implies either a low prediction potential or that this dataset alone cannot be used to train a good predictive model. Notably, the test accuracy of AR is typically 100%, which is consistent with the high EvoRate of 2.93.\n\n    |     Corruption Ratio           |0                       |0.25|0.5|0.75|1                         |\n    |----------------|-------------------------------|-----------------------------|-------------------------------|-----------------------------|----------------------------|\n    |EvoRate |2.93| 1.52|1.28|1.18|1.04|\n\n\n3. (True MI are not able to obtain) We acknowledge that for real-world high-dimensional data, MI is often intractable, as highlighted in the literature [1]. However, our experimental results demonstrate that we can achieve a reasonable estimation of MI, as illustrated in Figure 1. Similar to the approach of ForeCA, we use an information-theory-based estimator as an indicator of evolving patterns. While ForeCA uses entropy, we use a mutual information estimator.\n\n\n\nResponses to Questions:\n\n1. (MI to evaluate evolution of patterns) Using MI to estimate the relationship between pairs of variables is a fundamental problem in science and engineering. We propose to leverage MI to quantify the strength of the evolving patterns, which reveals that the evolving patterns is strongly related to the latent temporal dependency. We justify our choice of using MI to measure evolving patterns in Section 4.2 and Proposition 1. Specifically, MI is an intrinsic property of the data that directly influences the expected error of the maximum likelihood estimation loss. Therefore, we claim that MI serves as a direct and reliable indicator of evolving patterns.\n\n2. (Why EvoRate better than ForeCA) This limitation is due to ForeCA's inability to detect complex patterns, as illustrated in lines 112-114 of our paper. While temporal patterns can encompass trends, cycles, irregular fluctuations, and more complex behaviors, ForeCA is restricted to detecting only simple linear cyclical patterns. In his paper [2], he assumes that the data from future time steps is a linear transformation of the historical data. However, our methods can leverage MI-guided efficient deep representation learning to describe complex and non-linear evolving patterns.\n\n3. (apply to new datasets without finetuning) We acknowledge that the current EvoRate model can utilize a fixed pre-trained encoder $g$ to project data into a lower dimension to reduce computational complexity. However, the autoregressive function $f$ in Eq (4) and (9) still requires fine-tuning for each dataset due to the distinct evolving patterns present in different datasets. Nonetheless, training a larger EvoRate model with more datasets could potentially result in a more robust model capable of testing evolving patterns across various datasets. This is a promising direction for our future research.\n\n[1] Estimating mutual information. Physical Review 2004\n\n[2] Forecastable component analysis. ICML 2013\n\n[3] Zoology: Measuring and Improving Recall in Efficient Language Models, Arxiv 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h3d2e6kmai", "reviewer": "Reviewer_V8vA", "summary": "This paper proposes a novel metric, Evolving Rate, using mutual information to measure the existence of the evolving patterns in sequential data. For the scenarios in which data samples across disparate time steps are not aligned, the paper proposes to build the correspondence between snapshots using optimal transport, and thus develop the corresponding EvoRate_w, to measure evolving patterns without correspondences. The final experiments conducted on multiple tasks using several datasets, demonstrate the superior performance of the proposed EvoRate and EvoRate using OT to align data across time steps, compared to SOTA baselines.", "questions": "How to choose the cost function in using optimal transport to build correspondence for data samples across time steps? Can you provide more experimental details?", "limitations": "No, the authors do not discuss the potential negative societal impact of their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significance:\nEvolving patterns are important for us to use sequential models to observed time series data. This paper develop the evolving rate (EvoRate), to quantify the evolving patterns. In particular, this EvoRate can be to assess temporal order and to conduct feature selections in sequential observations.\n\nClarity:\nThe paper is well motivated! Most of the technical parts of the work are clearly presented!\n\nQuality:\nOverall, the technical contribution of the work look sound! \nThe theorectical aspect of the evolving rate is valuable in modeling sequential data!", "weaknesses": "The experimental details about the cost function of the optimal transport in aligning data across time steps, the parameter and the neural architecture, are not provided. These details could be crucial for the others to use the methods!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Responses to Weakness\n1. (Cost function) The cost function of Optimal Transport (OT) is defined in Eq (7). In the literature [1-2], OT typically uses a distance metric between two samples, each sampled from different marginal distributions, as the cost function. For instance, $z^s$ and $z^t$ represent data from source and target domains in domain adaptation tasks [1-2]. In our approach, we account for the dynamic characteristics of sequential data. Specifically, we assume the presence of an autoregressive function $f$ that governs the evolving patterns within the data. Consequently, we design the cost function as the distance between the output of $f$ with historical samples as input $f(\\\\textbf{z}^t_{t-k+1})$, and the subsequent sample $z_{t+1}$. In our implementation, OT aims to find the matching between a batch of samples $\\\\{z_{t,i}\\\\}^B_{i=1}$ and $\\\\{z_{t+1,i}\\\\}^B_{i=1}$ in a training iteration across time steps.\n\n2.  (parameter and architecture) Thanks for pointing it out. We will add this detail in the revision. The autoregressive model $f$ is typically a 2 or 3-layer LSTM, with input size, hidden size, and output size set to 128, 256, or 512, depending on the architecture of the encoder $g$. For tabular data, $g$ is a 2 or 3-layer MLP, with the input size corresponding to the data size, and the hidden and output sizes set to either 128 or 512. For image data, $f$ is a ResNet-12 with a hidden size of 512.\n\n\nQ: (choose the cost function) We do not select the cost function; rather, we use a cost function that is consistent with the critic function as an autoregressive form in mutual information estimation, which leverages the structure of sequential data.\n\n[1] Optimal Transport for Domain Adaptation, TPAMI 2015\n\n[2] Joint distribution optimal transportation for domain adaptation, NIPS 2017", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their valuable comments on our work.

We have received four reviews with ratings of 3, 6, 7, and 7.

We are pleased that the reviewers have good impressions of our work, including:

- Addressing an interesting and important problem (8iST, rYZr, V8vA);
- Presenting a well-motivated paper with the proposed method EvoRate being useful for a variety of applications (rYZr, V8vA, oZmm);
- Theoretical justification for choosing mutual information (MI) as the indicator of evolving patterns in sequential data (rYZr, V8vA);
- Generally clear presentation (V8vA).

During the rebuttal period, we have provided detailed responses to all comments and questions point by point. Specifically,

- We further clarified why we chose Mutual Information as the metric to measure evolving patterns (to 8iST & oZmm).
- We further clarified how we trade off computational cost with the performance of EvoRate by selecting the dimension of the encoding space and added experiments to address this with different dimensions of the encoding space (to rYZr & oZmm).
- We added an NLP dataset to evaluate EvoRate, demonstrating its capability in real-world applications (to rYZr & oZmm).
- We added experiments with a fixed pretrained encoder to show that EvoRate can be very efficient by only training the autoregressive model $f$ (to oZmm).

In the attached PDF, we provide the pseudocode for our proposed algorithms, a detailed comparison of three traditional time-series statistical indicators with EvoRate, and an introduction to the newly added NLP dataset.

Lastly, we would like to thank all the reviewers for their time once again. Could you please check our response and confirm if you have any further questions? **We are looking forward to your post-rebuttal feedback!**

Meta-review: The paper examines the underlying assumptions made by autoregressive time series models that data streams are evolving, by introducing a method EvoRate that seeks to detect whether a data stream is evolving or not by looking at mutual information across the stream. They introduce a method EvoRateW to detect EvoRate using optimal transport.

After the author response period, the paper has scores of 6, 8, 8 and 6. The reviewers agreed that the method has potential for real world applicability, with one reviewer particularly impressed by the applicability to NLP. The reviews also agreed that there is significant novelty in the methodology. 

I recommend this paper for acceptance, with the suggested changes being included.

Minor comment - I noticed that citation [30] has formatting errors - I would recommend a thorough proof-reading before .camera ready submission.
Decision: Accept (spotlight)