Title: How Patterns Dictate Learnability in Sequential Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Time Series

Sequential Data

Autoregressive Models

Mutual Information

Generalization Error
Abstract: Sequential data—ranging from financial time series to natural language—has driven the growing adoption of autoregressive models. However, these algorithms rely on the presence of underlying patterns in the data, and their identification often depends heavily on human expertise. Misinterpreting these patterns can lead to model misspecification, resulting in increased generalization error and degraded performance. The recently proposed $\texttt{evolving pattern (EvoRate)}$ metric addresses this by using the mutual information between the next data point and its past to guide regression order estimation and feature selection. Building on this idea, we introduce a general framework based on predictive information—the mutual information between the past and the future, $\mathbf{I}(X_{\text{past}}; X_{\text{future}})$. This quantity naturally defines an information-theoretic learning curve, which quantifies the amount of predictive information available as the observation window grows. Using this formalism, we show that the presence or absence of temporal patterns fundamentally constrains the learnability of sequential models: even an optimal predictor cannot outperform the intrinsic information limit imposed by the data. We validate our framework through experiments on synthetic data, demonstrating its ability to assess model adequacy, quantify the inherent complexity of a dataset, and reveal interpretable structure in sequential data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a theoretically grounded and general framework for understanding the learnability of sequential data using predictive information—the mutual information between the past and the future. It formalizes the concept of an information-theoretic learning curve (via the universal learning curve, Λ(k)) and connects it to fundamental limits on forecasting performance. Building on prior work like EvoRate, the authors extend the analysis by defining a minimal achievable risk, offering new diagnostic tools for model adequacy. Theoretical contributions are complemented by experiments on synthetic datasets, including autoregressive and Ising processes, validating the proposed framework’s ability to detect learnability bounds and optimal regression orders.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\t1.\tStrong Theoretical Foundations:\nThe authors rigorously establish the link between predictive information and learnability, grounding their framework in information theory and learning theory. This includes closed-form results for Markov processes, asymptotic expansions for parametric families, and connections to Bayesian learning theory (e.g., Corollary 4.4 on dimensionality estimation).\n\t2.\tConceptual Clarity:\nThe paper provides intuitive examples (e.g., lunch meal prediction) to motivate its central question—whether poor forecasting performance arises from data limitations or model inadequacies.\n\t3.\tSound Use of Information-Theoretic Tools:\nThe authors successfully extend EvoRate into a more general predictive information metric that connects to the universal learning curve. This supports robust interpretations of structure in data and connects theory with empirical forecasting tasks.\n\t4.\tSynthetic Experiments:\nThe experiments validate the theory by showing that the proposed estimator Λ(k) tracks ground-truth model orders and minimum risk accurately in controlled settings.\n\n⸻\n\nWeaknesses:\n\t1.\tEmpirical Evaluation Scope:\nThe experiments are performed entirely on synthetic datasets—autoregressive Gaussian processes and Ising spin chains. While these are useful for validation, they fall short of demonstrating the framework’s practical utility on real-world data (e.g., NLP or financial sequences), which the paper itself acknowledges as a limitation.\n\t2.\tEstimation Stability:\nThe authors note estimation variance and occasional instability of the Ipred estimators in high dimensions or near low-complexity regimes. This potentially limits the utility of the learning curve in real settings unless estimation is improved.\n\t3.\tModel Dependency of the Estimator:\nThe proposed estimator of minimal achievable risk is model-dependent, which contrasts with EvoRate’s model-free nature. While this allows diagnostic comparisons, it introduces potential confounds related to model capacity and training variability.\n\t4.\tInterpretability vs. Accessibility:\nWhile the theoretical depth is commendable, some derivations and notations—though well-supported in the appendix—could be inaccessible to readers not well-versed in information theory or statistical learning, potentially limiting broader appeal.\n\nOverall Evaluation:\n\nThe paper is clearly written, well-structured, and theoretically sound. It presents a significant conceptual advance over EvoRate by offering a quantitative and theoretically justified estimator of the fundamental limits of learnability. While the theory is solid, I cannot fully assess its novelty in the context of the most recent literature on predictive information estimation and sequential risk bounds. As someone less familiar with the deep technical nuances of the information-theoretic theory presented, I found the framework persuasive, but dense.\n\nThe empirical component appears somewhat underwhelming compared to the strength of the theoretical contribution. It may be that synthetic validation is standard in this line of work, but the lack of any real-world application or benchmark comparison makes it hard to assess its broader utility.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses two questions:\n1. What are the minimum achievable risks for any predictor intending to model sequential data?\n2. Can the error be attributed to predictor limitation or unpredictability of the sequential data?\n\nIn the context of stationary processes, the authors consider the mutual information between the past and the future with increasing window size to derive lower bounds for the minimal achievable risks and highlight patterns' impact on learnability. Theoretical results have been highlighted with empirical experiments on synthetic\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-motivated and written. The theoretical framework is well described, and several intuitions are provided, giving a good understanding of the framework's usability.\n- Applying the framework to the case of Markovian and parametric processes seems promising, as the estimation of the proper window length or parameter space remains challenging in practice.\n- Appendices are well-detailed, complementing both the theoretical section with proofs and the experimental section with detailed results and implementation, easing reproducibility.\n\nWeaknesses:\n- While not central to this paper, an illustration of the framework on real-world data would benefit the overall comprehension and paper motivation.\n- The paper could also benefit from a summary answering the two introductory questions with the framework as partially done in the theoretical and experimental parts.\n\nQUESTIONS:\n- Regarding the additional remarks on Result Interpretability, would it be possible to illustrate corollary 4.4 with a plot showing the asymptotic consistency of the dimension estimator?\n- Link to the first question: The window size is closely linked to the parameter space dimension for the autoregressive model. Assuming a large window size, would it be possible to estimate the window size from smaller windows?\n\nLIMITATIONS:\nLimitations are raised in my questions and weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces predictive information \\( I_{\\text{pred}} \\), which measures the mutual information between a length-\\(k\\) past and a length-\\(k'\\) future in sequential data. It generalizes the EvoRate metric from \\(k'=1\\) to arbitrary future block lengths. The authors connect \\( I_{\\text{pred}} \\),  with the concept of a universal learning curve \\( \\Lambda(k) \\), which quantifies the incremental benefit of extending the past window by one step. Theoretically, this allows identifying the optimal context length and estimating the parameter dimension \\(p\\) of the underlying generative process. The method is evaluated on synthetic data, including Gaussian, autoregressive, and Ising processes, to validate its theoretical and empirical utility.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is very well written and presents a clear theoretical contribution. It offers exhaustive derivations and proofs, and it generalizes EvoRate while drawing a formal connection to information theory through the universal learning curve.\n\nWeaknesses:\n- Limited empirical scope: currently all experiments are performed on synthetic data. No real-world sequential datasets are used to show practical benefit. This makes it hard for me to assess the impact of this work apart from the theoretical work and the mapping of predictive information and information theory. However, it is to note that the authors mention the lack of real-world data experiments as a limitation in the corresponding section.\n\n- Expressiveness of the experiments: Although the chosen synthetic experiments are well designed, it is unclear whether the reported benefits would persist under a wider set of settings, for example, more \\((k,k')\\) combinations in the first experiment, additional \\(p\\) values in the second, or cross-validation splits in the third.\n\n- Reproducibility: for the last experiment using MLP and LSTM the exact model parameter counts are not provided. Including these numbers would make the results easier to reproduce and interpret.\n\nQUESTIONS:\nQuestions:\n\n1. The Gaussian-process study reports results only for the fixed pair \\((k = 5,\\; k' = 10)\\).  How was this setting chosen, and do your conclusions hold for other \\((k,k')\\) combinations?\n\n2. What does the critical zone stand for in the learning-curve plot (Figure 2)?\n\n3. Could you provide the correct reference for the definition of consistency in line 569?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBuilding on the classical works of Bialek and Tishby [1], and Crutchfield and Feldman [2], this paper addresses the following questions related to sequential modeling: \n\n1)\tWhat is the minimum achievable risk for a predictor modeling sequential data?\n2)\tIf a model performs poorly, is it due limitations of the model, or due to the inherent unpredictability of the data? \n\nEffective sequence modeling requires (1) predictable patterns in the data, and (2) a model capable of exploiting those patterns. Determining whether a sequence contains predictable patterns is a crucial first step. Predictive information $I_{\\text{pred}}(k, k')$ (mutual information between the past $k$ observations and the future $k'$ observations) and universal learning curve $\\Lambda(k)$ (the discrete derivative of predictive information as $k' \\to \\infty$) are classical quantities that measure the learnability and complexity of a sequence.\n\nThe paper begins by revisiting basic concepts from information theory and defining predictive information and the universal learning curve under the assumptions of stationarity and finite entropy. The authors then compute these quantities for a $m$-order Markov chain and a parametric process where the dimensionality of the parameter space is $p$. Similar calculations can also be found in [1, 2]. \n\nNext, the authors bound the minimal achievable risk by the true $k$-th order forecasting risk minus $\\Lambda(k)$. They also present an empirical version of this bound involving the empirical $k$-th order forecasting risk. Based on this bound, if the model's loss fails to improve beyond a certain context length $k$ while $\\Lambda(k) > 0$, it indicates that the current model complexity is insufficient to capture the learnable patterns in the sequence. This, I believe, is the key contribution of the paper, as it addresses questions (1) and (2) above.\n\nFinally, the authors provide some experimental results demonstrating the estimation of $I_{\\text{pred}}(k, k')$ on a Gaussian process, the ability of the estimated $\\Lambda(k)$ to identify the correct order of an autoregressive process, and the estimation of the minimum achievable forecasting risk of an Ising spin sequence. \n\n**References**\n\n[1] William Bialek and Naftali Tishby. Predictive information. arXiv preprint cond-mat/9902341, 352 1999.\n\n[2] James P Crutchfield and David P Feldman. Regularities unseen, randomness observed: Levels of entropy convergence. Chaos: An Interdisciplinary Journal of Nonlinear Science, 13(1): 25–54, 2003.\n\nSTRENGTHS AND WEAKNESSES:\nI enjoyed reading this paper. The ideas are communicated clearly and effectively. However, I believe it lacks a bit of originality, as tools such as predictive information (or excess entropy) and the universal learning curve (or entropy gain) are well-established concepts in the field. That said, the paper makes a novel and interesting contribution by connecting these quantities to the minimum achievable risk, which adds valuable insight to sequential learning and modeling.\n\nQUESTIONS:\nI'm listing some minor questions and suggestions below. \n\n1. In line 32, you cite an example about the performance discrepancy related to GluonTs despite recent innovations. Can you elaborate this example by proving more context?\n\n2. In line 139, you use $\\ln$ and later in Eq. 6, you use $\\log$. Also, in line 139, $\\mathcal{F}_k$ is a set of functions, and in line 186, $\\mathcal{F}$ is the Fisher information matrix. Better to be consistent with the notation. \n\n3. In Figure 1, you have not indicated what the strength of the color means. \n\n4. Prospective Learning [1] introduces a framework in which data is generated by a stochastic process, and the learner is expected to produce a sequence of hypotheses that achieves low risk on future observations, given the data observed up to the present. I believe some of the results from your paper could be used to extend or enrich this framework. Related ideas have also been explored in [2, 3]. It would be valuable to position your work in the context of these studies in the related work section.\n\n**References**\n\n[1] De Silva, A., Ramesh, R., Yang, R., Yu, S., Vogelstein, J. T., & Chaudhari, P. (2024). Prospective Learning: Learning for a Dynamic Future. arXiv preprint arXiv:2411.00109.\n\n[2] Dawid, A. P., & Tewari, A. (2020). On learnability under general stochastic processes. arXiv preprint arXiv:2005.07605.\n\n[3] Hanneke, S. (2021). Learning whenever learning is possible: Universal learning under general stochastic processes. Journal of Machine Learning Research, 22(130), 1-116.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}