Title: Universal Sequence Preconditioning
Venue: NeurIPS 2025
Accepted: yes
Keywords: sequence prediction

time series

pre-conditioning
Abstract: We study the problem of preconditioning in the setting of sequential prediction. From the theoretical lens of linear dynamical systems, we show that applying a convolution to the input sequence translates to applying a polynomial to the unknown transition matrix in the hidden space. With this insight, we develop a novel preconditioning method that convolves the input sequence with the coefficients of the Chebyshev or Legendre polynomials. We formally prove that this improves the regret of a wide family of prediction methods. 

We proceed to apply this preconditioning technique to the method of spectral filtering. This gives the first sublinear regret bound that is also hidden-dimension free (up to logarithmic factors) even when the hidden transition matrix is asymmetric.

From rigorous experiments on synthetic data we show that our simple preconditioning method generalizes to both 1) settings where the data is \emph{not} from a linear dynamical system and 2) a broad range of learning algorithms, including recurrent neural networks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to solve sequential prediction via preconditioning. Based on Chebyshev polynomials, the authors propose a universal sequence preconditioning methodology that be applied as a convolutional preconditioned regression and convolutional preconditioned spectral filtering. The authors prove the convergence rate for a particular linear dynamic system, and experiment the methods on various settings, including linear dynamic system, non-linear dynamic system, and deep RNN.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The authors prove the explicit convergence rate.\n2. The authors give intuition behind the proof and the trade-off with respect to parameters (such as the degree of polynomials and the threshold).\n3. The algorithms are tested empirically.\n\nWeakness:\n1. There is unclear implicit dependence: Theorem 2.1 does not seem to be dependent on the degree of the polynomial $n$, which is surprising considered Algorithm 2 heavily depends on the choice of $n$. I would suspect that there is an implicit dependence on $n$ hidden somewhere, for example perhaps like the scaling $n=\\log(T)$ on line 114? If the result is completely independent of $n$, then the authors should explain why that is the case.\n2. Slow convergence rate: The convergence rate in Theorem 2.1 is $T^{-2/13}$ (up to log factors), which is quite slow. It is unclear from this paper whether this is the actual empirical convergence rate, or a pessimistic bound due to proof technique.\n\nQUESTIONS:\nThe questions follow from the \"Weakness\":\n1. Could you fix all the missing references, such as \"Eq.TO DO\" and \"Algorithm 4\"?\n2. Could you clarify all the implicit dependence?\n3. Could you discuss how sharp your theoretical results are?\n4. Could you provide more analysis comparing Chebyshev and Legendre polynomials in terms of their strength and weakness in terms of theoretical guarantee and empirical performance?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors investigate which input sequence modification method would facilitate the learning process of model learning and improve its prediction accuracy, which is referred as the preconditioning problem. Inspired by the theory of linear dynamic systems, they propose a novel and universal preconditioning method that convolves the input sequence with the coefficients of the Chebyshev or Legendre polynomials. They further combine the proposed universal preconditioning method with the spectral filtering method and propose a new spectral filtering algorithm. The authors then provide comprehensive theoretic analysis on the proposed preconditioning method and shown that its regret bound is smaller than that of previous methods. Particularly, the proposed spectral filtering algorithm has a sublinear hidden-dimensional free regret bound, even if the hidden transition matrix is asymmetric. Experiments on synthetic data show that the proposed preconditioning method is applicable to diverse data types and learning algorithms.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The motivation of this work is clear and theoretic solid. The inspiration comes from the theory of linear dynamic systems, and the authors provide comprehensive theoretic analysis for the proposed preconditioning method and show that it can achieve smaller regret bound.\n- The experiments on synthetic data verify that the proposed method has universal adaptability. It can be applied on data that do not form a linear dynamic system and various learning algorithms including recurrent neural network.\n\n**Weakness**\n- The major weakness of this paper is the presentation quality. The manuscript has several typographical errors, insufficient contextual details in certain parts, and requires improvement in logical coherence. For example, sentences ''...potentially shrinking the domain'' in line 33 and ''..., is a hard problem'' in line 68 are missing a period. Besides, the derivations of some theoretical results exhibit ambiguities that require clarification. See Question part for more detail.\n- The experiments are conducted only on synthetic data without validation on real-world data, which makes the proposed method less convincing. It is encouraged to conduct experiments on real-world datasets.\n\nQUESTIONS:\n1. There is no explicit definition of $p _n(\\cdot)$ in the last equation presented on page 5. Its definition only appears in line 2 of algorithm 3, that is, $p _n(x) = \\sum _{i=0}^n c_i x^{n-i}$. It is encouraged to add this definition before the last equation on page 5 to enhance clarity of expression.\n2. In line 670, the correct one should be $G \\leq \\sqrt{d _{\\rm out}}$, according to the inequality between line 669 and 670.\n3. In the equation between line 664 and 665, the authors show that $\\sum _{t=1}^T \\ell _t(\\\\mathbf{Q}^t) - \\min _{\\\\mathbf{Q}^{\\star} \\in \\\\mathcal{K}} \\sum _{t=1}^T \\ell_t (\\\\mathbf{Q}^{\\star}) \\leq \\frac{3}{2} GD \\sqrt{T}$. Also, they show that $D \\leq 2 C _{\\rm domain} n^2 \\Vert \\\\mathbf{c} \\Vert_1 $ and $G \\leq \\sqrt{d _{\\rm out}}$ in line 665 and line 670, respectively. Thus, the correct one should be $\\sum _{t=1}^T \\ell _t(\\\\mathbf{Q}^t) - \\min _{\\\\mathbf{Q}^{\\star} \\in \\\\mathcal{K}} \\sum _{t=1}^T \\ell_t (\\\\mathbf{Q}^{\\star}) \\leq 3 C _{\\rm domain} n^2 \\sqrt{d _{\\rm out}} \\Vert \\\\mathbf{c} \\Vert _1$ in line 671.\n4. In line 682, the authors show that $\\Vert \\hat{\\\\mathbf{y}} _t - \\\\mathbf{y} _t \\Vert_1 = \\Vert \\sum _{s=0}^{t-n-1} \\\\mathbf{C} p^{\\\\mathbf{c}} _n(A) \\\\mathbf{A}^s \\\\mathbf{B} \\\\mathbf{u} _{t-n-s} \\Vert _1$ holds. Then, they further show that $\\Vert \\\\mathbf{C} p^{\\\\mathbf{c}} _n(A) \\\\mathbf{A}^s \\\\mathbf{B} \\Vert _1 \\leq \\max _{\\\\lambda \\in \\\\lambda(\\\\mathbf{A})} \\vert p^{\\\\mathbf{c}} _n(\\\\lambda) \\vert \\cdot \\Vert \\\\mathbf{C} \\Vert \\Vert \\\\mathbf{B} \\Vert$ holds for any $j = 0, \\ldots, t-n-1$. Since $\\Vert \\\\mathbf{u} _j \\Vert _1 \\leq 1$, by triangleq inequality we should have \n$$\n\\begin{aligned}\n\\Vert \\sum _{s=0}^{t-n-1} \\\\mathbf{C} p^{\\\\mathbf{c}} _n(A) \\\\mathbf{A}^s \\\\mathbf{B} \\\\mathbf{u} _{t-n-s} \\Vert _1 \\leq & \\sum _{s=0}^{t-n-1} \\Vert \\\\mathbf{C} p^{\\\\mathbf{c}} _n(A) \\\\mathbf{A}^s \\\\mathbf{B} \\\\mathbf{u} _{t-n-s} \\Vert _1 \\leq \\sum _{s=0}^{t-n-1} \\Vert \\\\mathbf{C} p^{\\\\mathbf{c}} _n(A) \\\\mathbf{A}^s \\\\mathbf{B} \\Vert _1 \\Vert \\\\mathbf{u} _{t-n-s} \\Vert _1 \\leq \\sum _{s=0}^{t-n-1} \\Vert \\\\mathbf{C} p^{\\\\mathbf{c}} _n(A) \\\\mathbf{A}^s \\\\mathbf{B} \\Vert _1 \\\\\\\\\n& \\leq \\sum _{s=0}^{t-n-1} \\max _{\\\\lambda \\in \\\\lambda(\\\\mathbf{A})} \\vert p^{\\\\mathbf{c}} _n(\\\\lambda) \\vert \\cdot \\Vert \\\\mathbf{C} \\Vert \\Vert \\\\mathbf{B} \\Vert = (t-n-1) \\max _{\\\\lambda \\in \\\\lambda(\\\\mathbf{A})} \\vert p^{\\\\mathbf{c}} _n(\\\\lambda) \\vert \\cdot \\Vert \\\\mathbf{C} \\Vert \\Vert \\\\mathbf{B} \\Vert.\n\\end{aligned}\n$$\nTherefore, we should have\n$$\n\\min _{\\\\mathbf{Q}^{\\star} \\in \\\\mathcal{K}} \\sum _{t=1}^T \\ell_t (\\\\mathbf{Q}^{\\star}) \\leq \\sum _{t=1}^T \\ell _t (\\hat{\\\\mathbf{Q}}) = \\sum _{t=1}^T \\Vert \\hat{\\\\mathbf{y}} _t - \\\\mathbf{y} _t \\Vert_1 \\leq \\sum _{t=1}^T (t-n-1) \\max _{\\\\lambda \\in \\\\lambda(\\\\mathbf{A})} \\vert p^{\\\\mathbf{c}} _n(\\\\lambda) \\vert \\cdot \\Vert \\\\mathbf{C} \\Vert \\Vert \\\\mathbf{B} \\Vert.\n$$\nIt seems that it is possible to derive a slightly sharper bound, that is, $\\frac{(1+T)T}{2}$. It is also encouraged to include the above details in the paper enhance clarity of expression.\n5. In Eq. (18), the correct one seems to be $\\Vert U _{n-1}(z) \\Vert \\leq n$. If $n$ is even, we have $\\Vert U _{n-1}(z) \\Vert \\leq 2 \\sum _{j \\geq 0, j even}^n T_j (z) \\leq 2 \\cdot \\frac{n}{2} \\cdot 2 = 2n$, using Eq. (16) that $\\Vert T_j(z) \\Vert \\leq 2$ for any $j$. The case that $n$ is    add is similar.\n6. I am confused about the third line of the equation in line 918. It seems that we can only get \n$$\n\\sum _{k=0}^\\infty a _k \\vert r \\vert^{2k+1} \\min (1,(2k+1)\\vert \\theta \\vert) = \\sum _{k=0}^\\infty a _k \\vert r \\vert^{2k} \\cdot \\vert r \\vert \\min (1,(2k+1)\\vert \\theta \\vert) \\leq \\sum _{k=0}^\\infty a _k \\vert r \\vert \\min (1,2(2k+1)\\vert \\sin(\\theta) \\vert) = \\sum _{k=0}^\\infty a _k \\min (\\vert r \\vert,2(2k+1)\\vert r \\sin(\\theta) \\vert).\n$$\nCould the authors provide more explanations on this?\n7. The polynomials used in this paper include Chebyshev or Legendre polynomials, and both of them are orthogonal polynomials. Since Chebyshev or Legendre polynomials are special cases of Jacobi polynomial, is it possible to directly use the Jacobi polynomial ?\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work in line 278-380.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates preconditioning of sequences (of inputs and outputs to linear dynamical systems) for the purpose of improving prediction accuracy. The paper proposes a preconditioning method that convolves the \"input sequence\" (actually the \"output sequence\" according to Algorithm 1) with the coefficients of the Chebyshev polynomial. The paper proves that this \"universal sequence preconditioning\" can improve the regret of two popular prediction algorithms: regression (in which the predictor coefficients are updated online) and (a new variant) of spectral filtering. Numerical examples/experiments are run on synthetic data, coming from (noisy) linear dynamical systems, and nonlinear dynamical systems (including a neural network).\n\nSTRENGTHS AND WEAKNESSES:\nThe main strengths of the work are:\n- importance of problem area (investigation of preconditioning strategies for prediction)\n- mathematical rigor: I did not check all the proofs carefully, but the technical appendices were generally quite well-written and easy to follow.\n- the inclusion of numerical experiments is commendable for a paper that mostly makes theoretical contributions. \n\nIn my view, the main weaknesses of the work are:\n- Clarity of the presentation in the main paper. In contrast to the technical appendices in the supplementary material, the main manuscript is quite poorly written. For example, Line 92 refers to \"Eq: TODO\" (although this equation is given in the supplementary materials). Also, in Section 2.3 the regret of the spectral filtering algorithm is given as $\\tilde{O}(\\sqrt{T})$ whereas in the supplementary material it is given as $\\tilde{O}(T^{2/3})$.\n- Overall, the quality of the supplementary materials seems higher than that of the main submitted manuscript. (If I based my review on the main manuscript alone, I would probably recommend rejection). \n- Clarity: the paper also misses a clear problem statement. Readers familiar with the general problem setting can probably infer the overall goals and assumptions, but in my opinion the problem that the paper addresses should be precisely stated. For example, in the introduction the authors write \"The focus of this paper are not methods for sequence prediction, but rather preconditioning algorithms\", yet it appears that one of the main contributions of the paper is a more broadly applicable spectral filtering algorithm? Also, it's not immediately obvious whether the authors consider online or offline learning. For example, in Algorithm 1, line 7 makes it clear that prediction algorithms are learned offline, based on training data. However, algorithm 2 and 3 appear to update the prediction algorithm online. (Most likely the \"learning\" of the prediction algorithm in line 7 of Algorithm 1 refers to the tuning of hyper-parameters, and actual prediction algorithm is updated online -- still, it would be good to make this clear). \n- Clarity: overall, the main manuscript seems quite hastily written, e.g. \"Eqn: TODO\", sometime they refer to their algorithm as UPS instead of USP (e.g. Table 1). \n- Quality: although the core technical developments seem fairly sound, some of the statements around peripheral aspects are questionable. For example, on line 121, they claim that system identification is difficult. However, in their setting (linear systems with no noise -- which is the setting described in the supplementary material) system identification is easy (e.g. any subspace method should recover the system parameters perfectly). Also, there seems to be some confusion around \"closed-loop\" components. On line 127 they make clear that by \"closed-loop\" they refer to autoregressive terms (e.g. functions of previous outputs), whereas on line 108 they refer to Q as a closed-loop component, even though Algorithm 3: Line 5 shows that Q combines only with past inputs, not past outputs. \n\nOverall, the supplementary material seems relatively sound, but the sloppiness and lack of clarity in the main manuscript casts doubt on the quality of the work.\n\nQUESTIONS:\nCan you please provide a clear and precise description of the problem that this paper addresses? In particular, is T assumed to be known in advance? (It seems so). Can you please comment on this assumption (in many applications, predictors are required to run for arbitrarily long T). \n\nCan you please explain the nonlinear dynamical system in line 220 more clearly?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a method to precondition time series data such as to improve sequential prediction. The main idea relies on taking the convolution of the input sequence with Chebyshev polynomial coefficients. The authors then show that this methodology improves the regret of different sequence prediction models, thereby providing strong theoretical motivations for their approach.

The authors further show that their methodology can be applied to multiple algorithms such as recurrent neural networks. And they demonstrate the improved performance of their preconditioning method on synthetic data.

STRENGTHS AND WEAKNESSES:
- This paper proposes a very creative approach to improving learning of sequences.
- The authors provide strong theoretical motivation and guarantees for their preconditioning approach
- The experimental results show significant improvement of their method over no preconditioning.
- However, the experimental section is limited to synthetic data alone, which is a pity given the trove of sequence data available. I would encourage the authors to include real-world data experiments to further strengthen the paper.

QUESTIONS:
- Could the author include a non-synthetic dataset to their experiments to validate their approach in a less controlled setup ?
- The paper takes inspiration from differencing (subtracting each observation with the last one). However, the authors do not compare against this method in the experiments. Could the authors include that comparison ?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of preconditioning for sequential prediction, introducing a universal sequence preconditioning (USP) method grounded in the application of orthogonal polynomial filters (such as Chebyshev and Legendre polynomials) to input sequences. The authors provide a theoretical analysis demonstrating improvement in regret bounds for learning in linear dynamical systems, even in cases with asymmetric and marginally stable transition matrices. The paper applies USP to both convex regression and spectral filtering techniques, with rigorous regret analyses and empirical evaluation on synthetic data generated from linear, nonlinear, and deep RNN-based dynamical systems. Experimental results suggest that polynomial-based preconditioning yields consistent performance improvements across a range of models and data types.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper provides regret bounds for sequential prediction methods when combined with the proposed preconditioning, advancing prior work especially for asymmetric, marginally stable linear dynamical systems. Theoretical claims are generally supported by concrete statements. The preconditioning method is mathematically principled, leveraging properties of Chebyshev and Legendre polynomials, and can be applied to a broad class of sequence models, including regression and spectral filtering.\n\n2. The approach provides regret bounds that do not scale with the latent dimension of the dynamical system.\n\n3. The experiments are well designed, evaluating on synthetic linear, nonlinear, and RNN-generated data. Different preconditioning schemes are compared, including learned preconditioning, and a thorough sweep across polynomial degrees and hyperparameters is conducted.\n\nWeaknesses:\n\n1. The assumption on the eigenvalues' arguments in Theorem 2.1 maybe too strong for complex neural networks, as it demands a system behavior with virtually no oscillations. If this assumption holds true, please cite some papers to validate this.\n\n2. Tables 2–4 list results for several polynomial degrees, but the text could provide deeper interpretation of how performance depends on the degree parameter, especially for higher degrees where some results appear inconsistent.\n\n3. While the proof for Algorithm 3 relies on the common theoretical assumption that the system matrix can be diagonalized (justified by density arguments), the paper could more explicitly discuss how “arbitrarily small perturbations” to achieve diagonalizability might accumulate over T steps in an online learning context.\n\n4. The proof of Theorem C.7 misapplies Lemma C.8, which explicitly requires Krylov matrices to have Hermitian arguments. However, the matrix constructed in the proof of Theorem C.7 (diag(z)) is generally not Hermitian, as its diagonal elements are typically complex numbers. Please do not hesitate to correct me if my understanding is inaccurate.\n\nMinor issues:\n\n1. The table number and title always appear before the table. Table 1-4 should be revised.\n\n2. When writing the thesis, there are few citations of related work, which affects the readability of the article. For example, Line 21-22: “It is widely acknowledged that learning this sequence is easier than learning the original sequence for a large number of modalities.”\n\n3. Line 21: “easier\", Line 127: “closed loop\"\n\nQUESTIONS:\n1. How the eigenvalue imaginary-part assumption for the transition matrix manifests in commonly encountered real-world time series?\n\n2. What are the main computational bottlenecks?\n\n3. Why analytic polynomial coefficients (Chebyshev/Legendre) are more robust than learned ones? Is it due to optimization landscape, overfitting, or initialization?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}