Title: Learning Robust Spectral Dynamics  for Temporal Domain Generalization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Domain generalization

Temporal domain generalization

Concept drift
Abstract: Modern machine learning models struggle to maintain performance in dynamic environments where temporal distribution shifts, \textit{i.e., concept drift}, are prevalent. Temporal Domain Generalization (TDG) seeks to enable model generalization across evolving domains, yet existing approaches typically assume smooth incremental changes, struggling with complex real-world drifts involving both long-term structure (incremental evolution/periodicity) and local uncertainties. To overcome these limitations, we introduce FreKoo, which tackles these challenges through a novel frequency-domain analysis of parameter trajectories. It leverages the Fourier transform to disentangle parameter evolution into distinct spectral bands. Specifically, the low-frequency components with dominant dynamics are learned and extrapolated using the Koopman operator, robustly capturing diverse drift patterns including both incremental and periodic drifts. Simultaneously, potentially disruptive high-frequency variations are smoothed via targeted temporal regularization, preventing overfitting to transient noise and domain uncertainties. In addition, this dual-spectral strategy is rigorously grounded through theoretical analysis, providing stability guarantees for the Koopman prediction, a principled Bayesian justification for the high-frequency regularization, and culminating in a multiscale generalization bound connecting spectral dynamics to improved generalization. Extensive experiments demonstrate FreKoo's significant superiority over state-of-the-art TDG methods, particularly excelling in real-world streaming scenarios with complex drifts and uncertainties.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FreKoo, a novel framework for Temporal Domain Generalization (TDG) designed to address concept drift, particularly complex drifts involving long-term periodicity and domain-specific uncertainties. The core idea is to analyze the trajectories of model parameters in the frequency domain. The authors provide theoretical grounding for their approach, including stability guarantees for Koopman prediction, a Bayesian justification for high-frequency regularization, and a multiscale generalization bound. Extensive experiments on synthetic and real-world datasets demonstrate FreKoo's superiority over existing SOTA TDG methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Addresses Critical TDG Limitations: This paper tackles two critical limitations of current TDG methods: (i) the difficulty of modeling long-term periodic dynamics, and (ii) vulnerability to overfitting domain-specific uncertainties. This is particularly relevant for the complex drift scenarios encountered in real-world applications.\n\n2. Novel Methodology: The proposed method, FreKoo, leverages a novel and well-motivated approach by applying spectral analysis to model parameter trajectories for TDG.\n\n3. Rigorous Theoretical Foundation: The paper provides a rigorous theoretical foundation for the proposed method.\n\n4. Comprehensive Experiments: The thorough experimental setup, encompassing a diverse set of synthetic and real-world benchmarks with various drift types, convincingly validates the paper's claims.\n\n5. Clear and Compelling Presentation: The paper is generally well-written and clearly structured, featuring a compelling problem statement and motivation.\n\nWeaknesses and Suggestions for Improvement:\n\n1. The paper mentions using encoders/decoders $\\left(\\varphi, \\varphi^{-1}\\right)$ for the Koopman operator. While Appendix C. 3 specifies MLP architectures, a little more detail in the main text about how K is learned (e.g., is it simply a linear layer optimized via L_koop, or are there specific constraints to encourage stability beyond the loss term?) would be helpful.\n\n2. Lemma 1 provides a tighter bound if K's spectral radius ρ(K) < 1. Is this condition actively enforced during training, or is it an assumption for the tighter bound?\n\n3. FreKoo operates on model parameter trajectories. While this is an effective proxy for concept drift, a brief discussion on the assumed relationship between parameter evolution in frequency bands and the actual shifts in P(X,Y) could add further depth. For instance, does a strong periodic component in θ_low necessarily imply a periodic shift in the data distribution itself?\n\n4. Throughout the paper, there appears to be an interchangeable use of \"periodical\" and \"periodic\" when referring to time- based cyclical patterns (e.g., \"long-term periodical dynamics\" vs. \"periodic drifts\"). For clarity and consistency, it is recommended to uniformly use \"periodic\" as it is the more standard adjective in this context.\n\n5. Please check and complete the missing author information for reference [11] (currently listed as \"--\").\n\nQUESTIONS:\nSee the Strengths and Weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method for Temporal Domain Generalization (TDG) named FreKoo that addresses performance degradation in machine learning models caused by complex concept drifts over time. By transforming model parameter trajectories into the frequency domain, FreKoo decomposes temporal dynamics into low-frequency components, which are extrapolated using the Koopman operator to capture long-term trends and periodicities, and high-frequency components, which are regularized to suppress transient noise and domain-specific uncertainties. The authors provide theoretical guarantees through a multiscale generalization bound and demonstrate FreKoo’s effectiveness in modeling stable parameter evolution. Experiments across classification and regression benchmarks show FreKoo outperforms existing TDG methods, particularly in scenarios with periodic or noisy distribution shifts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The proposed idea makes sense. The key innovation lies in modeling model parameter dynamics in the frequency domain, which allows principled separation of stable (low-frequency) and noisy (high-frequency) components.\n- Code is provided for reproducibility purposes.\n\nWeaknesses:\n- The proposed theoretical results rest on strong assumptions, including the Lipschitz continuity of the neural networks and the dependence of the future domain $T+1$ solely on the immediately preceding domain $T$ via the Koopman operator. These assumptions are overly restrictive. For instance, enforcing Lipschitz continuity may not be practical for deep neural networks, which often exhibit highly non-smooth behavior. Additionally, assuming that domain $T+1$ depends only on domain $T$, and is conditionally independent of all earlier domains ($1, 2, \\dots, T-1$), oversimplifies the complexity of real-world temporal dynamics, which often exhibit long-range dependencies.\n- There is a notable gap between the theoretical analysis and the proposed algorithm. Specifically, Theorem 1 characterizes the relationship between domains $T$ and $T+1$ only, whereas the algorithm leverages information from all preceding domains to estimate domain $T+1$. Developing a theoretical bound that captures the influence of all prior domains [1] would provide a stronger alignment between theory and algorithm, leading to a more coherent and rigorous framework.\n- The proposed algorithm assumes the existence of ground-truth or optimal model parameters that evolve smoothly over time. However, this assumption is difficult to satisfy in practice, particularly in high-dimensional settings involving complex models and data. For instance, deep neural networks may admit multiple optimal solutions with significantly different parameter sets. This ambiguity challenges the practicality and applicability of the proposed approach in real-world scenarios.\n- The experimental design lacks comprehensiveness. The authors evaluate their method only in a static setting, where the model is trained on all but the last domain and tested solely on the final domain. To more thoroughly assess the effectiveness of the proposed algorithm, additional experimental settings should be considered. For example, a dynamic setting with varying split points between training and testing domains [2] would better reflect real-world scenarios. Evaluating model performance on further future timestamps (e.g., $T+2, T+3$) is also recommended to assess long-term generalization. Furthermore, experiments on real-world, high-dimensional datasets [3,4] are crucial to demonstrate the practical utility and scalability of the method.\n- The method is designed for batch TDG and doesn't support continual or online adaptation, limiting its applicability in real-time streaming environments.\n- The architecture involves multiple encoders and a Koopman operator, and work on parameter space instead on data space, which may introduce nontrivial computational and memory overhead compared to simpler TDG baselines.\n\nReferences:\n\n[1] Pham, Thai-Hoang, Xueru Zhang, and Ping Zhang. \"Non-stationary domain generalization: theory and algorithm.\" Uncertainty in artificial intelligence: proceedings of the... conference. Conference on Uncertainty in Artificial Intelligence. Vol. 2024. 2025.\n\n[2] Yao, Huaxiu, et al. \"Wild-time: A benchmark of in-the-wild distribution shift over time.\" Advances in Neural Information Processing Systems 35 (2022): 10309-10324.\n\n[3] Ginosar, Shiry, et al. \"A century of portraits: A visual historical record of american high school yearbooks.\" Proceedings of the IEEE International Conference on Computer Vision Workshops. 2015.\n\n[4] Lin, Zhiqiu, et al. \"The clear benchmark: Continual learning on real-world imagery.\" Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 2). 2021.\n\nQUESTIONS:\n- Theorem 1 focus only on the relationship between domain $T$ and domain $T+1$, while the algorithm aggregates information from all previous domains to predict $T+1$. Can you extend the theoretical analysis to capture the influence of the full trajectory $\\{1, 2, ..., T\\}$ on the prediction of domain $T+1$? \n- The algorithm assumes that optimal model parameters evolve smoothly over time, which may not hold in practice, particularly in deep networks with high-dimensional non-convex loss landscapes. Can you provide empirical or theoretical justification for this assumption? For example, visualizing learned parameter trajectories on real-world datasets or quantifying smoothness across multiple runs could help.\n- The evaluation is limited to a static setting with fixed training/testing split (e.g., domains $1$ to $T$ for training, $T+1$ for testing). The authors should including more dynamic and realistic settings, such as: Varying the split index between training and test domains, evaluating on extended horizons (e.g., $T+2$, $T+3$) to test robustness under longer-range drift, using real-world high-dimensional datasets beyond the current benchmarks.\n- The method critically relies on the choice of spectral energy preservation ratio $\\tau$ and the decomposition into low- and high-frequency components, but these design choices are not deeply explored. Could you provide more systematic analysis on how sensitive the performance is to $\\tau$, and whether this threshold could be learned or adapted from data? Additionally, how does the model perform when trained with noisy or misaligned frequency partitions?\n- The use of Koopman operator assumes that low-frequency dynamics evolve approximately linearly in a latent space, but this may be violated in highly nonlinear or chaotic temporal processes. Can you discuss the expressivity limits of the Koopman formulation? Including empirical failure cases or comparisons to nonlinear forecasting alternatives (e.g., RNN-based parameter predictors) would be helpful.\n\nLIMITATIONS:\n- The theoretical analysis relies on Lipschitz continuity of neural networks and the linear Koopman operator to model low-frequency dynamics. These assumptions are difficult to guarantee in practice, particularly in high-dimensional and highly nonlinear deep learning settings.\n- The theory assumes that the future domain $T+1$ depends only on the immediately preceding domain $T$, neglecting long-range temporal dependencies that are common in real-world sequences. This simplification limits the theoretical alignment with the actual algorithm, which uses all past domains.\n- The method assumes that ground-truth model parameters evolve smoothly over time, which may not hold in complex real-world scenarios. In practice, multiple optimal models can exist with significantly different parameters, undermining the assumption of smooth parameter trajectories.\n- While the work focuses on academic benchmarks, the use of TDG methods in high-stakes domains (e.g., finance, healthcare, surveillance) could introduce risks if models generalize poorly across time or are misled by transient patterns. The authors should reflect on potential risks of over-reliance on extrapolation-based generalization and the consequences of failure under severe or abrupt distribution shifts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn time-series analysis, out-of-distribution generalization remains a significant challenge. The paper tackles the setting where the label-generating mechanism changes over-time. The goal is to be able to predict these changes ahead of time and adapt the model. \n\nThe proposed method treats the model parameters themselves as the signal. At every time step, an FFT vector is computed from the parameter trajectory estimated thus far. The learner separates the low-frequency components and the high frequency of the FFT trajectory and learns to predict the trajectory governed by the low frequency components and high frequency components. For the low-frequency part, the model uses the low frequency component $\\theta_t^{\\text{low}}$ and passes it via an encoder to output a latent. The latent evolution is modeled via a linear transformation (this part of the work is inspired from Koopman operators). For the temporal trajectory of the high-frequency components, a temporal difference regularization is imposed to smoothen out the changes. \n\nThe authors develop a multi-scale generalization bound to explain why this philosophy of frequency aware separation helps with improved generalization. Finally, the authors conduct experiments on seven benchmarks out of which five are classification based and two are regression based. In these experiments, the authors both establish that their approach turns out to be state-of-the-art but also provide an ablation analysis to show critical the role of different modeling components (Koopman operator, frequency decomposition etc.).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Technical novelty** Treating the parameters as signal and separation of the signal into low and high-frequency component is a new method that to the best of my knowledge was not tried before. \n\n2. **Theoretical arguments** Low-frequency component's evolution depends on the Koopman operator. The authors show the sharpening of the generalization bound provided the spectral radius of the operator is less than one. Further, the authors propose a Bayesian justification for the smoothening regularizer used for the high-frequency components. \n\n3. **Solid empirical section** The results in the experiments section show that the approach is especially beneficial in modeling periodic drifts as shown by Elec 2 and Appliance dataset. The ablations and sensitivity plots are quite informative. \n\n**Weaknesses**\n\n1. **Writing** I found the paper poorly written. The method is hard to follow as it is broken down into many small subparts. The authors made the unfortunate choice of having the algorithm in the Appendix. The authors should take the pain of writing a simple step-by-step description of the key elements of the method. Especially since the method would first involve learning theta's for each domain, it makes it all the more confusing as that is not done in standard methods. \n\n2. **Vague claims:**  First let me quote, \"Prevailing TDG methods often fall short under complex concept drifts. Typically constrained by assumptions of incremental change or local smoothness, they struggle to simultaneously capture long-range dynamics like periodicity (Challenge 1) and filter transient noise or domain-specific uncertainties (Challenge 2), thereby failing to effectively balance stability with adaptability crucial for real-world scenarios\" \nFirstly, both these challenges are vaguely stated. Can the authors be more precise on what do each of these challenges mean in a mathematical sense? \nSecondly, when the authors state that the existing TDG methods fail to capture xyz, it would be best if authors can create a simple toy example. Through this toy task, the authors can highlight why the existing methods fail and why their method succeeds. Bear in mind, the toy task is to build intuition.  \nLet me quote \"This averaging operation inherently suppresses parameter-wise variations in oscillation patterns, ensuring robustness to localized noise in individual dimensions\". Now this is again a vague claim, which is not precise and has no evidence to back it up. Can the authors explain the choice of averaging better? or was it just the most natural heuristic? \n\n\n3. **On the role of Koopman operator** In the current form, the authors use the Koopman operators to motivate their loss function. I found the whole description via Koopman operators overloaded and unnecessary. The authors might have as well stated that they are modeling latent dynamics via a linear transformation. Here is a parallel I want to draw. Consider kernel SVMs, where one motivates featurization in infinite dimensional spaces and then uses representer theorems to show how Kernel matrix is all we need to characterize the solutions. In those descriptions, the infinite dimensional formulation is actually used. In this case, the authors are referring to an infinite dimensional object but using the straightforward linear usage of it. It is like referring to kernel SVMs, but invoking the linear kernel. Hence, I recommend that the authors tone down the jargon as it feels unncessary.\n\n\n4. **On the method** Overall I find that the method currently involves several heuristic choices and is not elegant. And one important lesson that stands test of time, is that simpler methods seem to outshine more complex counterparts in the long run. If in spirit of this, the authors could think of simplifications that they can do and suggest some simplifications for future work, that would be nice. \nDid the authors consider a simple transformer-based baseline? The transformer can take in as input the parameters theta and predict their changes? It will be nice to illustrate the failures of a transformer based baseline to model these long-term periodic changes in the parameters.\n\nQUESTIONS:\nPlease refer to the weaknesses section to read the questions.\n\nLIMITATIONS:\nWhile the authors have a limitations section, the discussion of the limitations is rather sparse and can be expanded.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FreKoo for TDG (Temporal Domain Generalization). FreKoo first decomposes the model parameter into low and high frequency parts by DFT. FreKoo models the low frequency part by learnable Koopman operator, and suppresses high frequency part by adding regularizations. This paper gives a generalization bound for FreKoo, and demonstrates its practical advantage through experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\no The spectral decomposition of the parameter space of the model is interesting and well-grounded.\no Using Koopman operator to model low-frequency components and regularizations for high-frequency components are technically solid.\no The paper provides theoretical results to validate the performance of FreKoo.\no The experiments are extensive, by evaluating across various well-known datasets and benchmarks.\no The paper is well-written and well-organized.\n\nWeaknesses:\no For experiment completeness, please also include the inference and training time. This information is critical for assessing the practical\nfeasibility of the method, especially in time-sensitive applications like online learning.\no The experimental comparison would be more robust with the inclusion of TKNets, another Koopman-based approach. Incorporating it would provide a clearer understanding of how the proposed method compares against existing alternatives in the same domain.\n\nQUESTIONS:\n1. Will the number of parameters in the model also affect the performance?\n2. Will overparameterization/overfitting of the model affect the selection of high and low frequency components?\n3. In appendix C3, the implementation details of the hyperparameters, different datasets and tasks have different learning rate for prediction module, encoder-decoder module, Koopman module. Also, the spectral pereservation ratio and loss weights are different as well. How these hyperparameters were selected? Is there any underlying intuition or guiding principle behind these choices? Furthermore, why do these parameter settings appear to generalize well to all datasets except the appliance dataset?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}