Towards Understanding Evolving Patterns in Sequential Data

Authors: qiuhao  zeng
long-kai  huang
qi  chen
charles  ling
boyu  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 08:50:06

Abstract: In many machine learning tasks, data is inherently sequential. Most existing algorithms learn from sequential data in an auto-regressive manner, which predicts the next unseen data point based on the observed sequence, implicitly assuming the presence of an \emph{evolving pattern} embedded in the data that can be leveraged. However, identifying and assessing evolving patterns in learning tasks often relies on subjective judgments rooted in the prior knowledge of human experts, lacking a standardized quantitative measure. Furthermore, such measures enable us to determine the suitability of employing sequential models effectively and make informed decisions on the temporal order of time series data, and feature/data selection processes. To address this issue, we introduce the Evolving Rate (EvoRate), which quantitatively approximates the intensity of evolving patterns in the data with Mutual Information. Furthermore, in some temporal data with neural mutual information estimations, we only have snapshots at different timestamps, lacking correspondence, which hinders EvoRate estimation. To tackle this challenge, we propose EvoRate$_\mathcal{W}$, aiming to establish correspondence with optimal transport for estimating the first-order EvoRate. Experiments on synthetic and real-world datasets including images and tabular data validate the efficacy of our EvoRate.

Main paper: 
 introduction: 
 Sequential data is ubiquitous across various machine learning tasks, including multivariate time series [33, 38, 44] , video streams in computer vision [18, 52, 54] , textual data in natural language processing [9, 17, 34] , and state-action trajectories in reinforcement learning [5, 45, 56] . Learning with sequential data usually involves predicting future data points, fostering the development of auto-regressive techniques that learn to forecast the subsequent unseen entries in a sequence. Despite the progress in this field, one fundamental challenge persists: the identification of underlying evolving patterns often depends heavily on the subjective interpretations and prior knowledge of human experts. This reliance on subjective judgment lacks a robust quantitative method to assess the evolving patterns over the high-dimensional data in deep learning. For example, when designing a recommendation system, certain products such as clothing are highly dependent on temporal factors (e.g., seasons, fashion trends), while others, like computers, are more influenced by individual customer preferences. Therefore, it is critical to identify and quantify the underlying evolving patterns for different products and integrate this information into the algorithmic design. Specifically, the following questions are essential but unresolved yet in literature: i) How can the existence of evolving patterns in data sequences be determined? Determining the existence of evolving patterns in data is a critical task. It is possible that the data points of a sequence are entirely independent and no evolving patterns exist. For instance, consider the scenario of a person repeatedly tossing a coin. In this case, historical information does not influence the outcome of the next toss. ii) Can one determine the historical span that significantly influences the current time point? For example, how do we determine the order (the optimal number of past observations) of an autoregressive model in a principled way? iii) How can we determine if the collected features are sufficient to reveal evolving patterns? For instance, to achieve better weather forecasting, how can one determine the essential features, such as altitude, humidity, and geographic location, for gathering a comprehensive set of information for forecasting.In this work, we address these questions through a unified framework by introducing EVORATE (EVOLVING RATE), a novel approach designed to quantify the evolving patterns of data sequences. EVORATE leverages mutual information as a measure of the existence of the evolving patterns in the data. Notably, while there is a rich history of mutual information estimation in the existing literature [8, 1, 27, 10, 29] , existing works ignore the underlying temporal dependency between the data points, and therefore are not well-designed for sequential data. EVORATE tackles this issue by estimating mutual information in an autoregressive manner when learning the compressed embedding from the observed sequence, thereby addressing the aforementioned questions: i) it can serve as an indicator to show that learning a sequential model is not feasible to learn the provided sequential dataset. ii) EVORATE can provide a quantitative measure of the temporal dependency of a sequence, allowing us to control the trade-off between computational complexity and learning performance. iii) EVORATE can also guide us in selecting the most informative features for model training for sequential data.However, EVORATE is difficult to estimate when dealing with temporal data characterized by snapshots captured at disparate timestamps without clear correspondence between them [30, 48, 42] , as we do not track the same data point over different timestamps and thus lack access to its corresponding sample. This scenario hinders the estimation of EvoRate, due to the absence of the correspondence. To mitigate this issue, we propose an enhanced version of our methodology, EVORATE W , which is specifically designed to establish correspondence among data points across different timestamps utilizing optimal transport within the Wasserstein distance metric, thereby facilitating the estimation of the first-order EVORATE. In all, the benefits of EVORATE to be highlighted include:• EVORATE enables quantitatively measuring the evolving patterns existing in high-dimensional sequential data by utilizing the neural mutual information estimator. Furthermore, it can be applied to assess temporal order and conduct feature selections in sequential data. • We further proposed EVORATE W to leverage optimal transport to build the correspondence between snapshots at the different timestamps, and hence allow the MI approximations. • We motivate through analysis the use of mutual information as indicators of evolving patterns and show optimal transport can mitigate the without correspondence issue. • Synthetic and real-world datasets verify that EVORATE can be a good indicator for evolving patterns, supporting our claim of its benefits. We also design an EDG algorithm based on the insight of EVORATE W and verify its performance.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Claims are in the abstract and introduction sections.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Sequential Data The analysis and processing of sequential data is driven by diverse applications ranging from video predictions to time series forecasting [25, 50, 9, 16, 33] . Pioneering works such as Long Short-Term Memory (LSTM) [25] networks have established foundational principles for handling long-range dependencies in sequence data. Building on this, the Transformer [50] introduced a revolutionary approach through self-attention mechanisms, enhancing flexibility in handling sequence dependencies. The versatility of Transformers has been further demonstrated in models such as GPT-3 [9] and BERT [16] . Beyond text, sequential data analysis in machine learning also extends to time-series forecasting [33] . Moreover, the application of Graph Neural Networks in capturing dependencies in irregular sequences underscores the breadth of methodologies exploring the complexities of sequential data [7] . However, a qualitative method for measuring the intensity of evolving patterns remains lacking in the literature.Mutual Information (MI) Estimation has become a pivotal tool in machine learning [39, 8, 1, 27, 10, 29] , enabling insights into dependencies that extend beyond traditional correlation measures.In feature selection, MI offers a data-driven approach to identify relevant features without strong assumptions about data distributions [39] . Mutual Information Neural Estimation (MINE) [8] applies deep learning to estimate MI in high-dimensional settings, providing a new methodology for analyzing neural network training dynamics. MI's application in variational inference, especially in the training of variational autoencoders (VAEs) [1] . In reinforcement learning, MI has been used to enhance exploration strategies by quantifying information gain [27] . MI also improves the performance of generative adversarial networks (GANs) [10] . Furthermore, in unsupervised and semi-supervised learning, MI maximization has been shown to effectively leverage unlabeled data [29] . However, none of them employ MI as an indicator for evolving patterns of sequential data.Optimal Transport (OT) has emerged as a powerful framework in machine learning [51, 3, 31, 13, 41] , offering a principled approach to compare probability distributions. Optimal transport theory has been leveraged for applications ranging from domain adaptation to generative modeling [51] . Recent advances include the integration of OT with deep learning architectures; Wasserstein GAN (WGAN) utilizes the Wasserstein distance to improve the stability of training GANs [3] . Furthermore, optimal transport has been applied effectively in NLP [31] . The computational aspect of OT has also seen significant developments, Sinkhorn [13] as a scalable method approximates transport plans efficiently.More recently, researchers have explored the differential properties of transport plans in dynamic environments [41] . EVORATE employs OT to recover the correspondence between two consecutive timestamps, facilitating approximations of mutual information.Patterns estimation for sequential data has only one related work in the literature ForeCA [24] , which proposes a similar concept, "forecastibility", which measures the uncertainty of the entropy of the spectral density. However, ForeCA has two drawbacks. Firstly, ForeCA can not be used in deep learning as an unacceptable huge computational consumption for real-world high-dimensional data (audio, videos, etc.). In contrast, EVORATE shows the prediction power by relying on mutual information, which tells the ability to predict another variable based on known observed variables. Secondly, while temporal patterns can include trends, cycles, irregular fluctuations, and more complex behaviors, ForeCA can only detect cycled patterns. Instead, EVORATE relies on the neural mutual information estimator, which is known as a good measurement for various patterns as a result of the strong fitting power of neural nets [8, 11, 37, 46] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: see section G.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
variational mutual information estimation: 
 The mutual information between two random variables X and Y is defined as the KL divergence D KL between their joint distribution and the product of their marginal distributions:EQUATIONwhere we aim to estimate this using samples from P (X, Y ); in some cases, the density of the marginals such as P (X) may be known. A wide range of variational methods are designed to estimate variational mutual information [8, 11, 46, 36, 37] . We then use the below estimator to estimate mutual information:EQUATIONwhere X is the random variable, x is a realization of X (as is the case with Y and y), and m(x, y) is a critic function to quantify the similarity between X and Y , usually realized by a neural network [8, 11, 46, 36, 37] . We show that MI is highly related to the evolving patterns of the sequential data in Section 4.2.A rich class of divergences between probability distributions is induced by the optimal transport (OT) problem [51] . Kantorovich's formulation of the problem is given byEQUATIONwhere c(x, y) : X × Y → R + is any measurable cost function and Π(P (X), P (Y )) is the set of all the joint distributions π(X, Y ) whose marginals are P (X) and P (Y ) respectively. The Wasserstein distance W c is then the "cost" of the optimal transport plan.4 Measure evolving patterns via MIQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: See computational cost section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
evorate: 
 Consider a sequence z T 1 : {z t } T t=1 , a collection of sequential data points from time 1 to T , where each z t ∈ R D denotes a state or observation at the discrete time step t with total T steps. In practice, the sequence z T 1 can represent time series data, video, textual, audio, or any other ordered data stream. We propose the use of the mutual information (MI) between the next observation and historical data over the past k steps I(Z t t-k+1 ; Z t+1 ) to measure the evolving pattern within a time window of length k. In the literature, the mutual information is empirically estimated through equation 2, which involves learning the critic function m [8, 11, 46, 36, 37] . However, one critical issue with existing works is that they ignore the temporal dependency of the data, and therefore the critic function m can have a high bias for sequential data (shown in Figure 1a, 1b ).To take the temporal dependency into account when estimating I(Z t t-k+1 ; Z t+1 ), instead of learning the critic function m, we propose learning the autoregressive function f , which summarizes the historical information embedded in Z t t-k+1 , and measuring its distance to Z t+1 via the squared error metric. Specifically, we introduce EvoRate to estimate the empirical sequential MI Î(Z t t-k+1 ; Z t+1 ) by defining m : R k×D × R D → R, m(x k 1 , y) = -∥f (g(x 1 ), .., g(x k )) -g(y)∥ 2 2 in equation 2:EQUATIONwhere g : R D → R d is an encoder. By selecting a different d, we can make a trade-off between computational cost and MI estimation accuracy. With d ≪ D, EVORATE is a more computationally efficient method for approximating sequential MI than learning an autoregressive model in the original data space. However, due to the Data-processing inequality [12] , this results in lower MI estimates. As g is employed as an identity function, MI is estimated in the original space, thereby enhancing estimation correctness at the expense of increased computational consumption.In this section, we justify the validity of EVORATE as a metric of evolving patterns through the lens of a k-th order autoregression. Specifically, we define the Maximum likelihood estimation (MLE) loss asL mle = -E P (Zt+1,Z t t-k+1 ) log Q(Z t+1 |Z t t-k+1), where Q is the probability distribution learned by the autoregressive model F trained with a supervised loss (MLE, MSE) on sequential data. Note that the MLE loss can also be viewed as the expected risk of autoregressive prediction tasks [44] .The following proposition establishes the connection between the expected risk of a k-th order autoregression task and the mutual information I(Z t t-k+1 ; Z t+1 ): Proposition 1. Let H denote the entropy. For autoregression tasks, the expected MLE loss satisfy:EQUATIONA proof of the proposition is provided in Appendix A. Proposition 1 provides novel insights into learning a predictive model for an autoregression task from an information-theoretic perspective:1. The expected risk can be decomposed into two orthogonal factors, where (i) measures the distance between the learned distribution Q and true distribution P , and therefore is determined by the predictive model F . (ii) quantifies the inherent temporal dependency of the sequence. Notably, it is independent of F . 2. More importantly, due to the nature of mutual information, I(Z t+1 ; Z t t-k+1 ) ≤ H(Z t+1 ) and (i) attains a minimum of zero when the observed sequence Z t t-k+1 encapsulates all the information of Z t+1 . Conversely, Proposition 1 reveals that even if F can properly learn the true probability P (i.e., (i) is small), its expected risk remains high when there is no temporal dependency that can be leveraged (i.e., I is small).Consequently, EvoRate, as an empirical estimate of I(Z t+1 ; Z t t-k+1 ), can play an important role in indicating the success of learning from sequential data and therefore is adopted to quantify the evolving pattern in this work.Subsequently, we demonstrate that MSE loss defined asL mse = E P (Zt+1,Z t t-k+1 ) ||F (z t t-k+1 ) - z t+1 || 22 can be interpreted as a variant of MLE loss, hence MI can be applied to a wide range of sequential data tasks that utilize MSE loss. Proposition 2. Assume that the predicted conditional probability density Q learned by the prediction model followsQ(Z t+1 |Z t t-k+1 ) = N (Z t+1 |F (Z t t-k+1 ), I D ), where N (•) denotes a Gaussian distribution with mean F (Z t t-k+1 ) and identity covariance matrix I D . Then, the following holdsEQUATIONwhere L mse is the MSE loss and const is a constant term.5 Measure evolving patterns without correspondences 
estimate joint distribution: 
 In many real-world applications, instead of processing many data point observations at different timestamps as data sequences, one needs to handle a data set at each timestamp: {z t,i } nt i=1 collected from multiple timestamps t = {1, . . . , T } [48, 42] , where i is the sample index and n t is the number of samples collected at timestamp t. The distribution P (Z t ) associated with these data sets evolves over time t ∈ R. For example, consider a supervised learning problem involving medical data z t,i = (x t,i , y t,i ) collected from multiple patients at different ages [42, 6] . In this scenario, we do not track the same patient across different ages, resulting in a lack of correspondence between timestamps and our objective extends to characterizing the evolving patterns of {Z t } T t=1 across these discrete timestamps. However, EVORATE proposed in subsection 4.1 cannot be applied to this context due to the absence of the correspondences.Estimating the mutual information from two data sets requires the pairwise correspondences between the sample of two data sets, which are assumed as given in existing works [8, 1, 27, 10, 29] . The correspondence between Z t and Z t+1 reflects their joint distribution as it encapsulates how the values of Z t and Z t+1 co-occur. This structured relationship indicates the interdependence of Z t and Z t+1 , which the joint distribution quantifies. Since the absence of the correspondence (i.e., an object observed at time t is not at time t + 1), we can not access the joint probability distribution of the past states and the next state. To tackle this issue, we estimate the joint distribution through the optimal transport plan of the Wasserstein Distance. Specifically, we define the distance loss according to a joint distribution measurement πL t W (π, f ) = E (z t ,z t+1 )∼π ||f (g(zt)) -g(zt+1)|| 2 2 (7)where g is fixed from updated gradients computed from L t W . Empirically, allowing g to update during model training leads to the undesirable outcome of all representations collapsing into a single point as a result of minimizing the Wasserstein distance loss. To avoid this and preserve maximal information within the representations, g is trained separately using an auto-encoder architecture with a reconstruction MSE loss.Then, we compute the optimal transport plan π * to approximate the real joint distributionEQUATIONand f is updated in an alternating optimization manner with fixed π * to minimize L t W (π * , f ). When the original dimension D is low, g can be an identity function for precise MI estimation. Conversely, when D is high, directly learning f from Z t to Z t+1 requires more accurate information and precise correspondence. This is because f must be a considerably more complex and larger model to facilitate mapping from one high-dimensional space to another. As a result, the hypothesis space F for f expands, requiring more information to ensure the model converges to an optimal state. The absence of correspondence therefore presents a challenge as it leads to an information-insufficient situation and it becomes more suitable to set a smaller representation dimension d.It is noted that when the correspondences between two consecutive timestamps exist, they can be inferred by minimizing the Wasserstein distance. When such correspondences do not exist, one can still establish correspondences by identifying a proxy of z t,i in the succeeding timestamp that exhibits similar dynamics and shares the latent evolving patterns.We hence use π * (Z t , Z t+1 ) to estimate joint distribution P , and then obtain the following estimator with π * (Z t , Z t+1 )EvoRateW = E (z t ,z t+1 )∼π * (Z t ,Z t+1 ) -||f (g(zt))) -g(zt+1)|| 2 2 -log E z t ∼P (Z t ),z t+1 ∼P (Z t+1 ) e -||f (g(z t ))-g(z t+1 )|| 2 2 (9)Here k can be regarded to set to 1 compared to equation 4, indicating that EVORATE W focuses on the first-order evolving patterns. It is possible to extend this approach to estimate higher-order k-order sequences by iteratively leveraging outcomes from first-order through to k-order sequential modeling.The following remark argues that there exists an optimal function that precisely captures the underlying dynamics of evolving data. Remark 1. (Realization) In machine learning prediction tasks, there exists a function f * ∈ F : Z → Z where the conditional distribution of Z t+1 given Z t satisfiesEQUATIONThe following lemma demonstrates that when f reaches the optimal predictive model f * , the estimated optimal transport plan equals the real joint distribution. In this context, we consider g to be the identity function.EQUATIONBelow, we give an illustrative example. As T ≫ 1, the function f will converge to f * by minimizing L t W (π * , f ). It demonstrates that for a dynamic system without correspondences, the number of timestamps must be greater than 1 to learn the optimal autoregressive model effectively.Example Consider data collected from multiple time steps where each sample is a vector Z t ∈ R D . Specifically, the initial data points at the first timestamp is modeled as a Gaussian variable Z 1 ∼ N (µ 1 , Σ 1 ). The temporal evolution of the data is governed by a transition function z t+1 = f * (Z t ) = A * z t + b * , t ∈ {1, . . . , T } and each Z t follows a Gaussian distribution Z t ∼ N (µ t , Σ t ) where A * ∈ R D×D , b * ∈ R D . Solving the optimizing problem L t W (π * , f ), t ∈ {1, . . . , T -1} cant } T t=1 , t ∈ {1, . . . , T }, z T = ρ T -1 t=1 zt T -1 + 1 -ρ 2 ϵ, with correlation ρ ∈ [-1, 1], ϵ ∼ N (0, I), Z t ∼ N (0, I), t ∈ {1, . . . , T -1}.Given the correlation coefficient ρ and dimensionality D = 128, we can compute the ground truth MI as EvoRate(Z T -11 ; Z T ) = -(D/2) ln(1 -ρ 2 ). The optimal MI estimation can be achieved when sequential model f equals the ground truth model f * = Avg, where Avg(•) is an average operation. In Figure 1a and 1b, we increase ρ over training steps to show the estimator behavior depends on the true mutual information. Additionally, we experiment with two forms of architecture: separable and joint. Separable architectures independently map the representations of history states f (Z T -1) and the future state Z T to an embedding space with neural nets ϕ 1 and ϕ 2 separably, and then take the inner product, i.e.EQUATION)) T ϕ 2 (Z T ) as in [37] . Joint critics concatenate each f (Z T -1), Z T pair before feeding it into the network, i.e. ϕ([f (Z T -1 1 ); Z T ]) as in [8] . In this experiment, g is set to an identity function, and the sequential model f is set to an LSTM [25] . All networks are fully-connected networks with ReLU activations. Figure 1a shows the estimated mutual information by EVORATE over the number of iterations, and square error metric can let f converge to f * such that the EVORATE W converges to ground truth mutual information. Figure 1b verifies that the square error metric has a lower bias compared to trainable concat critic and separable critic.t } T t=1 , t ∈ {1, . . . , T -1}, z t+1 = ρ(A * z t + b * ) + 1 -ρ 2 ϵ, where A * ∈ R D×D is a rotation matrix, b * ∈ R D is a translation vector, correlation of ρ ∈ [-1, 1], ϵ ∼ N (0, I), and Z 1 ∼ N (0, I). Given the correlation coefficient ρ and dimensionality D = 128, we can compute the ground truth MI value EvoRate(Z t ; Z t+1 ) = -(D/2) ln(1 -ρ 2 ). The optimal MI estimation can be achieved when sequential model f equals the ground truth model f * = A * z t + b * . In this experiment, it is actually very difficult to estimate mutual information without correspondence. As a result, the estimations by joint and separable critic do not converge and fail in the case without correspondence, which further shows the square error metric shows better performance than the trainable neural nets critic. In Figure 1c ,1d, g being an identity function estimates a higher value than g being a neural-nets encoder. It is noted that EVORATE W is the only method able to estimate the mutual information without the correspondence between timestamps, achieving a reasonable performance to estimate MI. 
sequential data's order approximation and feature selection: 
 Order Approximation We sample data with 5-order (k = 5), and dimensionality D = 5, which means Z t+1 is determined by Z t t-4 . More specifically, the data is generated by the dynamic function Z t+1 = A * vec(Z t t-4 ) + b * , where in this experiment, vec(•) is a vectorized operation, A * ∈ R 5×25 and b * ∈ R 5 . We vary k ∈ {1, 3, 5, 12, 24} to measure the EVORATE between Z t t-k+1 and Z t+1 . Figure 2a shows that k = 5 has the maximal EVORATE value. In another experiment, the time series forecasting task is used to verify the effectiveness of EVORATE. Time series forecasting performance is evaluated with the sMAPE metric [35] , measured as the mean absolute error scaled by the magnitude of the predictions and target. The performance shown in Table 1 is the SOTA method [53] and they set the model with order k = 45. The order is set as k ∈ {10, 25, 45, 90, 180, 270}. Although k = 270 achieves the highest EvoRate, the difference between k = 270 and k = 90 is only 0.03, and the performances over average (AVG) sMAPE have the same prediction error. For M4-Weekly, EVORATE shows order set as k = 90 can achieve a good performance. Although EVORATE is slightly higher for k = 270 than k = 90, it sacrifices three times more computation consumptions compared to only a +0.03 EVORATE gain if the model time complexity is O(k). Forecastability (ForeCA) fails in this experiment, as shown in Table 1 , since the longer time series shows smaller forecastability but it achieves smaller sMAPE and a better performance. Longer sequence can have more evolving patterns in different frequencies combined and result in a smaller forecastability, but it may be more easily predictable once the patterns are learned by the model. Therefore, we conclude that the entropy used by ForeCA is not a good indicator of the capability of predictions while MI used by EVORATE is. In addition, randomness is a critical factor for the capability of the predictions of the sequential data. Since one of the evolving patterns is learned by sequential models, the performance only relies on the randomness of the data, which can be regarded as unwanted noises or unobserved factors. Feature Selections For autoregressive tasks, poor predictions may due to the lack of the features. Some features may be redundant and some may be unrelated to predictions. Others may be related to the task but are not put as input fed into the prediction model. The synthetic data has 5 dimensions, where the first 3 are useful, the fourth is redundant and the fifth is unrelated (Details in Appendix B.4).Figure 2b shows the EVORATE of the data sequence with the first n features. The results show that i) EVORATE achieves the highest value with the first three features, ii) the first four features containing one redundant feature sees a minor performance drop, and iii) using all five features sees a larger drop.In some problems, data is sampled independently from the history observations [49, 2] . In this case, we suggest directly learning a model using ERM [49] for i.i.d (independent and identically distributed) or IRM [2] for data sampled independently but with distribution shifts. In many machine learning applications, data is predicted in an autoregressive manner by training sequential models [50, 9, 16, 33] . Whether to use ERM/IRM or sequential models directly depends on the existence of the evolving patterns. Therefore, we take EVORATE as the criterion for the existence of evolving patterns.In Table 2 , EVORATE can achieve better estimates of the evolving patterns compared to ForeCA, where stronger evolving patterns indicate smaller regression errors using sequential models. Specifically, for M4-Monthly and M4-daily, ForeCA shows equal values but EVORATE shows higher values for M4-Daily, consistent with experimental results in which M4-Daily achieves lower sMAPE.Evolving Domain Generalization (EDG) follows our setting in Section 5.1, where the correspondence is intractable and we aim to learn the evolving patterns to predict y t,i conditioned on input x t,i for every sample z t,i = {x t,i , y t,i } [42, 57] . In Table 3 , we show the performance of EVORATE W and SOTA performance for invariant learning [2] and evolving learning [42, 57, 59] . Although, performance of the evolving representation learning not only depends on the existence of the evolving Video prediction aims to predict future video frames from the current ones. In this experiment, we evaluate EVORATE on the KITTI dataset [20] , which contains 28 driving videos with a resolution of 375×1242. 24 videos in KITTI dataset are used for training. We verify the performance of EVORATE by shuffling the index of the sequential data with a certain corrupt probability, and this randomness will decrease the evolving patterns (Figure 3 ).Figure 2c shows that by increasing the corruption rate to the video sequence, EVORATE exhibits a lower value. This is consistent with our intuition, which is that the continuous video stream shows higher patterns compared to the disordered video clips.From the intuition that L W can estimate the joint distribution, we apply this loss to learn a transition model based on the estimated joint distribution between two consecutive domains. Table 5 shows this is an efficient method and achieves improved performance on EDG tasks. Our method shows an 11.7% higher average accuracy than the second-best baselines.  
conclusion: 
 In this work, we propose EVORATE to qualitatively estimate the evolving patterns for the data sequences and the data snapshots from multiple consecutive timestamps without correspondences. We show the square error metric can be both a better critic for mutual information estimation, and a well-designed loss to help the optimal transport plan converge to the real joint distribution and the sequential model converge to the latent dynamic governing function. EVORATE reflects the complex patterns for high-dimensional data and is more computationally efficient than directly evaluating the performance of sequential data predictions. Experiments show EVORATE is an effective measure for evolving patterns and has the potential for many applications in the machine learning area. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: This work is not an application, but foundational research on machine learning. Hence, there is no negative social impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All related works are mentioned in related works.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Proposition 1. Assume that the model's probability density followsEQUATIONProof.EQUATIONProposition 2. For autoregression tasks, the expected risk satisfy:EQUATIONProof.EQUATION= -E z t+1 t-k+1 ∼P (Z t-k+1 ,...,Z t+1 ) logEQUATION= DKL(PEQUATION= DKL(PEQUATIONβ-mixing is a measure of the degree of dependence between random variables in a sequence over time, which is closely related to MI and furthermore upper bounded by MI: Remark 1. s ∈ N, β-mixing coefficients defined in below satisfy:EQUATIONwhere || • || T V is the maximum total variation distance.Proof.EQUATIONEQUATIONwhere the first inequality follows Pinsker's inequality; the second inequality follows Jensen's Inequality.Lemma 1. P (X, Y ) is the real underlying distribution, and π(X, Y ) is the optimal transport plan that satisfies both margins comply with P (X) and P (Y ). We first define: I P (X; Y ) = p(x,Then, we haveEQUATIONTherefore, we can getI P (Z t ; Z t+1 ) -I π (Z t ; Z t+1 ) = H π (Z t ; Z t+1 ) -H P (Z t ; Z t+1 ).Proof. I p (X; Y ) -I π (X; Y ) = p(x,EQUATION)26 to 27 is due to p(x, y) and π(x, y) have the same margins p(x) and p(y) and we integral over x or y first.Lemma 2. Let P (Z t , Z t+1 ) be the ground truth joint distribution. IfEQUATIONProof. To finish the proof, we first assumeEQUATIONEQUATIONEQUATIONEQUATION)-2ϵ T (z ′ t+1 -f * (z t )) dπ(Z t , Z t+1 ) -||ϵ|| 2 2 dπ(Z t , Z t+1 ) . Since ϵ ⊥ ⊥ z ′ t+1 -f * (z t ) inf π ∈ Π(P (Zt), P (Zt+1)) L t W (π, f ) = inf π ∈ Π(P (Zt), P (Zt+1)) ||z ′ t+1 -z t+1 || 2 2 dπ(Z t , Z t+1 ) + constTo achieve infimum of L t W (π, f ), Z ′ t+1 = Z t+1 should satisfy and hence π * (Z t+1 |Z t ) = P (Z t+1 |Z t ) and π * (Z t+1 , Z t ) = P (Z t+1 , Z t ) with a feasible solution.Example We consider data collected from multiple time steps where each sample is a vector Z t ∈ R D . Specifically, the initial data points at the first timestamp is modeled as a Gaussian variable Z 1 ∼ N (µ 1 , Σ 1 ). The temporal evolution of the data is governed by a transition function:z t+1 = f * (Z t ) = A * z t + b * , t ∈ {1, . . . , T }, and each Z t follows a Gaussian distribution, Z t ∼ N (µ t , Σ t ), where A * ∈ R D×D , b * ∈ R D .Solving the optimizing problem L t W (π, f ), t ∈ {1, . . . , T -1} can lead to the solutions reaching optimal mapping f * with t ≫ 1.(A, b) as z t+1 = f (Z t ) = Az t + b, it can be inferred that Z t+1 ∼ N (Aµ t + b, AΣ t A T ).For each pair of data from two consecutive timestamp data, the Wasserstein distance loss can be expressed as follows, according to [21] Wasserstein Distance loss = infπ L W (t) = ||µ t -(Aµ t-1 + b)|| 2 2 + T r AΣ t-1 A T + Σ t -2 Σ 1 2 t AΣ t-1 A T Σ 1 2 t 1 2Then, we can haveinf π L W (t) = ||µ t -(Aµ t-1 + b)|| 2 2 + T r AΣ t-1 A T + Σ t -2T r Σ 1 2 t AΣ t-1 A T Σ 1 2 t 1 2 = ||µ t -(Aµ t-1 + b)|| 2 2 + T r AΣ t-1 A T + Σ t -2T r A T Σ t AΣ t-1 1 2 = ||µ t -(Aµ t-1 + b)|| 2 2 + T r AΣ t-1 A T + Σ t -2 A T Σ t AΣ t-1 1 2 = ||µ t -(Aµ t-1 + b)|| 2 2 + T r AΣ t-1 A T + Σ t -2T r A T Σ t AΣ t-1 1 2 = ||µ t -(Aµ t-1 + b)|| 2 2 + T r AΣ t-1 A T + Σ t -2 A T Σ t AΣ t-1 1 2 = ||µ t -(Aµ t-1 + b)|| 2 2 + T r (A T Σ t A -Σ t-1 )(A T Σ t A -Σ t-1 ) T = ||µ t -(Aµ t-1 + b)|| 2 2 + ||A T Σ t A -Σ t-1 || F where || • || F is Frobenius norm. inf π L W (t) = 0,and the infimum is attained whenEQUATIONWe are dealing with a system where the matrix A and b together comprise n × (n + 1) unknown variables. At each time step t, the system is described by n × n quadratic equations and n linear equations. Typically, these quadratic equations yield two possible sets of solutions. To refine our estimates and converge towards the optimal parameters (A * , b * ), employing a large number of time steps (t ≫ 1) allows us to formulate an overdetermined system of equations.We sample data sequence {z t } T t=1 , t ∈ {1, . . . , T }, z T = ρ T -1 t=1 zt T -1 + 1 -ρ 2 ϵ,; Z T ) = -(D/2) ln(1 -ρ 2 ).We sample data sequence {z t } T t=1 , t ∈ {1, . . . , T -1},z t+1 = ρ(A * z t + b * ) + 1 -ρ 2 ϵ, where A * ∈ R D×D is a rotation matrix, b * ∈ R D is a translation vector, correlation of ρ ∈ [-1, 1],ϵ ∼ N (0, I), and Z 1 ∼ N (0, I). Given the correlation coefficient ρ and dimensionality D = 128, we can compute the ground truth MI value EvoRate(Z t ; Z t+1 ) = -(D/2) ln(1 -ρ 2 ).We sample data with 5-order (k = 5), and dimensionality D = 5, which means Z t+1 is determined by Z t t-4 . More specifically, the data is generated by the dynamic function Z t+1 = A * vec(Z t t-4 ) + b * , where in this experiment, vec(•) is a vectorized operation, A * ∈ R 5×25 , and b * ∈ R 5 . In this experiment, we set k to {1, 3, 5, 12, 24} to measure the EVORATE between Z t t-k+1 and Z t+1 . Crypto [4] This multivariate dataset contains 8 features on historical trades, such as open and close prices, for 14 cryptocurrencies. The original challenge is to predict 3-step ahead 15-minute relative future returns. Since we focus on long-term forecasting, we train all models to make 15-step predictions of 15-minute relative future returns. We use the original training set from the competition and do an 80%-10%-10% training-validation-test split.Player Trajectory [32] contains basketball player movement trajectories from NBA games in 2016. We randomly sample 300 player trajectories for training and validation and 50 trajectories for testing. All models are trained to yield 30-step ahead predictionsRotated Gaussian [58] consists of 30 domains generated by the same Gaussian distribution, but the decision boundary rotates from 0 • to 338 • with an interval of 12 • . We split the domains into source domains (1-22 domains), intermediate domains (22-25 domains) , and target domains (26-30 domains). The intermediate domains are utilized as the validation set.Circle [40] contains evolving 30 domains where the instance are sampled from 30 2D Gaussian distributions. The label is assigned using a half-circle curve as the decision boundary. (15 source domains, 5 validation domains, and 10 target domains) Sine In Sine [40] each data owns two attributes (x 1 , x 2 ). The label is assigned using a sine curve as the decision boundary. We rearrange this dataset by extending it to 24 evolving domains. Each domain covers 1 24 the period of the sinusoid. (12 source domains, 4 validation domains, and 8 target domains)Rotated MNIST (RMNIST) [22] is an adaptation of the popular MNIST digit dataset [15] , composed of MNIST digits of various rotations. The task is to classify a digit from 0 to 9 given an image of the digit. We follow [43] and extend it to 19 evolving domains via applying the rotations with degree of {0 • , 15 • , 30 • , . . . , 180 • } in order. (10 source domains, 3 validation domains, and 6 target domains).Portraits [23] is a real-world dataset that comprises photographs of American high school seniors collected over a period of 108 years (1905-2013) across 26 states. The objective is to accurately classify the gender for each photograph. The dataset is divided into 34 domains based on a fixed interval over time. (19 source domains, 5 validation domains, and 10 target domains)Caltran [26] consists of real-world images captured by a fixed traffic camera deployed in an intersection over time. Frames were updated at 3-minute intervals each with a resolution 320 × 320. We divide it into 34 domains by time. The task of Caltran is to classify scenes to identify the presence of one or more vehicles in or approaching the intersection. The challenge mainly raise from the continually evolving domain shift as changes include time, illumination, weather, etc. (19 source domains, 5 validation domains, and 10 target domains)PowerSupply [14] is a dataset designed for the task of time-section prediction of current power supply based on hourly records obtained from an Italian electricity company. The dataset consists of 30 domains formed according to days. Each data point is assigned a binary class label indicating whether the current power supply belongs to the morning or the afternoon. Domain shifts may arise due to variations in season, weather, price, or the differences between working days and weekends. (15 source domains, 5 validation domains, and 10 target domains)In this section, we present complete experimental results to validate the efficacy of our proposed evolving domain generalization task. All experiments are carried out on 498G memory, 2 x AMD Milan 7413 @ 2.65 GHz 128M cache L3, and 2 x NVidia A100SXM4 (40 GB memory). The algorithm's computational cost is the cost of OT (using package [19] ) O(B 3 ) and the cost of estimation of MI O(B 2 d), where B is the batch size in an iteration, and d is the representation dimension. In all, the total computational cost is O(B 2 d + B 3 ) (not counting the encoder g and decoder h). Compute EvoRate of i-th sample at timestamp t according to Eq (4), k < t:EvoRatet,i := -||f (g(z t-k+1,i ), . . . , g(zt,i)) -g(zt+1,i)|| 2 2 -log 1 B B j=1e -||f (g(z t-k+1,i ),...,g(z t,i ))-g(z t+1,j )|| 2 Compute the optimal transport plan π * , where L t W (π, f ) defined in Eq (7)π * (Zt, Zt+1) = arg min π ∈ Π(P (Zt), P (Zt+1))L t W (π, f ), ∀t ∈ {1, . . . , T -1}, 4:Compute EvoRate W of i-th sample at timestamp t according to Eq (9) , especially (z t,i , z t+1,i ) is sampled from π * :(EvoRateW )t,i : = -||f (g(zt,i)) -g(zt+1,i)|| 2 2 -log 1 B B j=1e -||f (g(z t,i ))-g(z t+1,j )|| 2 2 5:Update f by maximize1 B(T -1) T -1 t=1 B i=1 (EvoRate W ) t,i 6: end forThere are traditional statistic indicators for time series, but they have significant limitations: a) System sensitivity, measured by Lyapunov Exponents (LEs) [2] measures how sensitive a dynamic system is to initial conditions, b) Trend is the slope of a linear regression fitted to sequential data, and c) Seasonality, measured by the ACF test [3] , assesses linear correlations between observations at different time lags. These methods are not designed to measure evolving patterns and struggle to handle high-dimensional data. Each method measures only one aspect of evolving patterns: system sensitivity, trend, or seasonality. Together, they determine the overall evolving patterns. We present a comparison of these metrics with our method below:In the above table, a larger EvoRate consistently indicates a smaller potential prediction error (RMSE/sMAPE) for the dataset. In contrast, LEs, trend, and seasonality show little impact on the prediction errors. Another significant drawback of these methods is their inability to be directly applied to high-dimensional data, such as images, videos, and NLP datasets. Due to computational resource limitations, we have not included experiments involving Natural Language Processing (NLP) tasks on Large Language Models (LLMs) in our study. These models typically require extensive processing power and substantial memory, which exceed our current hardware capabilities. Additionally, the high costs associated with running these models make them impractical for our budget. Instead, we focused on alternative datasets and models that align with our available resources. We believe that our chosen datasets still provide valuable insights while remaining within our operational constraints. Future work could explore LLMs as our computational capacity expands.NeurIPS Paper ChecklistJustification: We preserve the anonymity.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper examines the underlying assumptions made by autoregressive time series models that data streams are evolving, by introducing a method EvoRate that seeks to detect whether a data stream is evolving or not by looking at mutual information across the stream. They introduce a method EvoRateW to detect EvoRate using optimal transport.

After the author response period, the paper has scores of 6, 8, 8 and 6. The reviewers agreed that the method has potential for real world applicability, with one reviewer particularly impressed by the applicability to NLP. The reviews also agreed that there is significant novelty in the methodology. 

I recommend this paper for acceptance, with the suggested changes being included.

Minor comment - I noticed that citation [30] has formatting errors - I would recommend a thorough proof-reading before .camera ready submission.

Reviews: review: paper_summary: paper_summary: This paper aims to identify the evolving pattern in sequential data. In addition, given evolving pattern may present in the sequential data, this paper would like to introduce a technique that can identify the best temporal order and features for learning the sequential data.
To address that, this paper proposed an indicator Evolving Rate (EvoRate), by borrowing the ideas from existing works in Mutual Information (MI) Estimation. The effectiveness of EvoRate is tested by experiments on multivariate time series forecasting, video forecasting and evolving Domain Generalization (EDG) tasks.
strength_weakness: strength_weakness: This paper measures evolving patterns by an improved MI indicator. Some discussions and experiments are conducted to evaluate the advantages of EvoRate over existing MIs. However, the question discussed in this paper is "How to determine the existence of evolving patterns in data sequences" which does not limit the technical solutions to MI-based one. Therefore, I am questioning the contribution of this paper.
questions: questions: Apart from comparing to existing works in MI, do you also consider studies working on evolving patterns but not using MI? How to use the results of accurate identification of evolving patterns? This paper shows the results of the estimated EvoRate compared to MI results. However, it is less clear to me what the real benefit of introducing MI into solving the evolving pattern issue.
limitations: limitations: This paper shows some interesting ideas and results, but the motivation is not very clear to me. The proposed method is an edited version of MI. However, I am not clear why it is beneficial to introduce MI to address evolving patterns compared to existing solutions.
paper_summary: paper_summary: This paper introduces EvoRate, a novel metric designed to quantify the evolving patterns in sequential data. The authors propose leveraging mutual information (MI) to measure the temporal dependencies between data points in a sequence. The paper addresses a significant challenge in machine learning: identifying and quantifying evolving patterns in high-dimensional sequential data. EvoRate is further extended to EvoRate$_W$, which uses optimal transport to establish correspondences between data points at different timestamps, facilitating MI estimation even when direct correspondences are absent. The proposed methods are validated through experiments on both synthetic and real-world datasets, demonstrating their effectiveness in various applications such as time-series forecasting, classification with temporal distribution shifts, and video prediction.
strength_weakness: strength_weakness: 1. The authors refer to EDG as their algorithmic contribution in Line 70, but the detailed setting and definition of the EDG problem are lacking. A clear introduction and definition are necessary to understand the context and setup of EDG.

2. There is some confusion regarding the term "learning performance" mentioned in Line 53. It is unclear whether this refers to the prediction tasks of sequential data or the process of training the model to measure mutual information (MI). Additionally, how EvoRate manages the trade-off between computational complexity and learning performance needs clarification. Since the mutual information measurement model also requires training, as is standard in variational MI estimators [1], why is it necessary to measure MI to reflect evolving patterns rather than directly relying on the predictive performance of sequential models (e.g., LSTM, Attention, SSM), which also require training?

3. It would be beneficial to include a pseudocode table to facilitate a clearer understanding of the algorithms for the readers.

4. Although the authors state they do not have experiments on NLP tasks, it would be highly recommended to apply EvoRate to NLP datasets, given that NLP is one of the most important sequential data tasks.

5. The claim that the function $f$ will converge to $f^*$ in Remark 2 is unconvincing. This is because the joint distribution is not available, making it difficult to accurately learn the transition function $f$. Without access to the joint distribution, the learning process lacks the necessary information to ensure convergence to the optimal function $f^*$, especially when dealing with complex, high-dimensional real-world data.

6. Synthetic experiments with known correspondences in Section 6.1, only include the average operation as the transition function. It would be more beneficial to also consider a more practical dataset.

7. There are some typos that cause confusions: Table 2 should be EvoRate and Table 3 should be EvoRate$_W$.

[1] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dyand Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 531–540. PMLR, 10–15 Jul 2018.
questions: questions: See the weaknesses above.
limitations: limitations: This paper is about fundamental machine learning research and is unlikely to have any potential negative social impact.
paper_summary: paper_summary: The article introduces a groundbreaking technique for measuring changes in sequential data, marking a notable advancement in the realm of machine learning. The authors bring forth the Evolving Rate (EvoRate) and its advanced iteration, EvoRate$_\mathcal{W}$, as effective tools for evaluating the temporal dynamics within datasets.
strength_weakness: strength_weakness: - It may not fully address the scalability of the proposed methods as dimensionality grows. The authors assert that the dimensionality reduction of the original data can lead to a decrease in computational time when calculating Mutual Information (MI). However, the process of training an auto-encoder itself demands considerable time and resources. Is this factored into their considerations?
- The scope of the experiments might be limited, with a notable absence of extensive real-world datasets
- The true MI values for high-dimensional data, such as in video and time series contexts, are often infeasible to obtain. How do the authors ensure that EvoRate can provide an accurate measure of MI for such high-dimensional data, especially given this significant limitation?
questions: questions: -I find the concept of leveraging Mutual Information (MI) to evaluate the evolution of patterns to be logical, yet I harbor reservations about whether MI can truly equate to the changes observed in sequential data. A deeper exploration of this connection would be highly beneficial.

-The ForeCA method's results are counterintuitive, as they oppose the prediction performance outlined in Table 1, where one would expect the inclusion of more historical data to enhance the predictive capabilities. Can the authors elucidate why EvoRate, an estimator of MI, is deemed superior for capturing evolving patterns over ForeCA, which is based on entropy estimation?

-I am curious to know if the EvoRate model, once trained, can be directly applied to new datasets without the need for retraining. If this is possible, it would make EvoRate a more adaptable and computationally efficient tool that could be easily integrated into various systems.
limitations: limitations: There's no limitation relating to social impact.
paper_summary: paper_summary: This paper proposes a novel metric, Evolving Rate, using mutual information to measure the existence of the evolving patterns in sequential data. For the scenarios in which data samples across disparate time steps are not aligned, the paper proposes to build the correspondence between snapshots using optimal transport, and thus develop the corresponding EvoRate_w, to measure evolving patterns without correspondences. The final experiments conducted on multiple tasks using several datasets, demonstrate the superior performance of the proposed EvoRate and EvoRate using OT to align data across time steps, compared to SOTA baselines.
strength_weakness: strength_weakness: The experimental details about the cost function of the optimal transport in aligning data across time steps, the parameter and the neural architecture, are not provided. These details could be crucial for the others to use the methods!
questions: questions: How to choose the cost function in using optimal transport to build correspondence for data samples across time steps? Can you provide more experimental details?
limitations: limitations: No, the authors do not discuss the potential negative societal impact of their work.

Comments: comment: Thank you for raising our score. We highly appreciate your recognition of our work.
We have identified a typo in our rebuttal:

In our response to the Limitation section, the phrase "As a result, EvoRate underestimates..." should instead refer to existing MI methods, not EvoRate.
Thanks for good explanation for my concerns
Dear Reviewer  rYZr,

We greatly appreciate your feedback and are glad that our rebuttal has resolved your concerns, resulting in an improved rating! We will incorporate your suggestions into the manuscript in future revisions.

Best regards,

Authors of 6086
Thank you for the clarifications and the new experiments. After reviewing the rebuttal, my concerns have been fully addressed. I am particularly impressed that EvoRate can be effectively applied to NLP datasets, demonstrating significant potential for various real-world applications. I have no further questions and am pleased to raise my score.
Dear Reviewer oZmm,

Thank you for your kind response and for taking the time to review our responses. We are pleased to hear that our responses were satisfactory and that you are maintaining your original assessment. Your thoughtful feedback has been invaluable in strengthening our work, and we sincerely appreciate your efforts.

Best regards,  
Authors
Thank you for the response. I've read it carefully and my questions have been addressed. I will keep my current score.