Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalization

Authors: chengtao  jian
kai  yang
yang  jiao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 13:55:48

Abstract: Out-of-Distribution (OOD) generalization in machine learning is a burgeoning area of study. Its primary goal is to enhance the adaptability and resilience of machine learning models when faced with new, unseen, and potentially adversarial data that significantly diverges from their original training datasets. In this paper, we investigate time series OOD generalization via pre-trained Large Language Models (LLMs). We first propose a novel \textbf{T}ri-level learning framework for \textbf{T}ime \textbf{S}eries \textbf{O}OD generalization, termed TTSO, which considers both sample-level and group-level uncertainties. This formula offers a fresh theoretic perspective for formulating and analyzing OOD generalization problem. In addition, we provide a theoretical analysis to justify this method is well motivated. We then develop a stratified localization algorithm tailored for this tri-level optimization problem, theoretically demonstrating the guaranteed convergence of the proposed algorithm. Our analysis also reveals that the iteration complexity to obtain an $\epsilon$-stationary point is bounded by O($\frac{1}{\epsilon^{2}}$). Extensive experiments on real-world datasets have been conducted to elucidate the effectiveness of the proposed method.

Main paper: 
 introduction: 
 In machine learning, a common challenge arises when the distributions of training and test sets differ significantly [Quiñonero-Candela et al., 2009] . This mismatch demands that models, trained on specific distribution data, should generalize well on unseen distribution data, known as OOD generalization [Shen et al., 2021 , Zhou et al., 2022] . Despite a vast amount of research on the OOD generalization [Zhang et al., 2017 , Sagawa et al., 2019 , Huang et al., 2020 , Arjovsky et al., 2019] , the field of OOD generalization in time series is relatively limited and presents more significant challenges. This is primarily due to the inherent temporal dependencies and dynamic changes characteristic of time series data [Hamilton, 2020] . Therefore, a critical aspect of improving time series OOD generalization is to learn robust representations that remain stable despite shifts in distributions.Recently, the field of machine learning has witnessed remarkable advancements in pre-trained foundation models, with notable examples including Large Language Models (LLMs) such as GPT [Radford et al., 2018] , LLaMA [Touvron et al., 2023] and CLIP [Radford et al., 2021] . These models have been instrumental in capturing and leveraging complex patterns across various domains. In addition, using foundation models, especially LLMs, in processing non-linguistic data, e.g., time series is increasingly drawing attention. By fine-tuning only a few handful of parameters, these models show remarkable versatility in diverse data formats ranging from audio [Ghosal et al., 2023] , image [Lu et al., 2021] and time series [Chang et al., 2023 , Jin et al., 2023] . Studies indicate that LLMs, as part of the broader foundation model spectrum, demonstrate sophisticated reasoning and strong pattern recognition capabilities [Wang et al., 2023 , Chu et al., 2023] , fundamentally acting as pattern machines [Mirchandani et al., 2023] . Moreover, LLMs have been shown to be effective in transfer learning across various modalities, due to their data-independent self-attention mechanism [Zhou et al., 2023] .Additionally, recent advancements in vision-language foundation models have shown promising developments in OOD generalization [Zheng et al., 2022 ], yet the exploration in time series remains underdeveloped. The potential of using foundational models is highlighted by the study in Liu et al. [2023a] , Hendrycks et al. [2020] , which suggests that pre-trained transformers can improve OOD robustness. Despite existing efforts, the limited exploration of foundational model applications in time series OOD generalization suggests an emerging field.In this paper, we propose a Tri-level learning framework for Time Series OOD generalization, named TTSO. Unlike conventional OOD generalization methods that focus solely on group-level [Jiao et al., 2022a , Huang et al., 2020] or sample-level uncertainties [Zhang et al., 2017 , Zhou et al., 2021 , Han et al., 2024] , our framework uniquely addresses both by combing a minimization problem for optimal model parameter learning, a maximization problem for dynamically data re-grouping, and another maximization problem for data augmentation under a tri-level framework. To tackle this tri-level problem, we propose a stratified localization algorithm via cutting planes. Leveraging the advanced representation learning capabilities of LLMs, we adapt this tri-level learning framework for fine-tuning LLMs.Our contributions can be summarized as follows:• Tri-level Learning Framework. In contrast to most existing works in OOD generalization, which primarily focus on either group-level or sample-level uncertainties, TTSO uniquely integrates both aspects under a tri-level learning framework. Specially, this comprehensive framework emphasizes the interdependent relationship between problems of each level, advancing beyond the typical single or bi-level methodologies in OOD generalization. Moreover, a theoretical framework based on Vapnik-Chervonenkis dimension has been developed to rigorously analyze and elucidate the generalization properties of TTSO. We then leverage this tri-level framework to fine-tune LLMs, achieving an maximum 4.9% improvement in performance on time series classification in OOD scenarios.• Stratified Localization Algorithm. To tackle the aforementioned tri-level optimization problem, we develop a stratified localization method using cutting planes. Unlike traditional gradient-based methods, TTSO removes the necessity of computing the hypergradient for the outer optimization problem. This computation is typically very challenging and computationally intensive due to the nested structure of the tri-level optimization problem. Furthermore, the decomposable nature of cutting planes offers a promising avenue for enabling distributed implementations of TTSO, thereby potentially enhancing scalability and computational efficiency.• Iteration Complexity Analysis. To validate the effectiveness of our method, we conducted a thorough theoretical analysis of the algorithm. We theoretically derive that the iteration complexity of the proposed algorithm for achieving an ϵ-stationary point is bounded by O 1 ϵ 2 . 
related work: 
 In this section, we provide an overview of the foundational concepts and methodologies related to our research, including OOD Generalization and the LLM in time series.OOD Generalization. OOD Generalization research focuses on improving the model's ability to generalize when there is a difference in distribution between the training and test data, and has been widely studied in the fields of Computer Vision (CV) [Recht et al., 2019 , Salman et al., 2021] and Natural Language Processing (NLP) [Tu et al., 2020 , Schneider et al., 2020] . Existing works for out-of-distribution (OOD) generalization are diverse and can generally be categorized into approaches that consider sample-level [Zhang et al., 2017 , Zhou et al., 2021] or group-level [Sagawa et al., 2019 , Huang et al., 2020] uncertainty. However, the exploration of OOD generalization specially for time series remains relatively underdeveloped. A recent study [Lu et al., 2023] introduced 'Diversify', an innovative approach that models time series data from the perspective of distribution and obtain superior performance. In our work, we consider both sample-level and group-level uncertainties and formulate them as a tri-level optimization problem.LLM in Time Series. The integration of LLMs in time series analysis is a rapidly evolving field, drawing significant interest due to their superior pattern recognition and reasoning abilities [Wang et al., 2023 , Chu et al., 2023] . A recent example is Time-LLM [Jin et al., 2023] , which introduces an innovative method by reprogramming time series and incorporating linguistic prompts, effectively activating the extensive capabilities of LLM. In addition, the OFA framework [Zhou et al., 2023] , utilizing the frozen pretrained transformer framework, validates the ersatility and effectiveness of pre-trained models in time series analysis. Another innovative approach is PromptCast [Xue and Salim, 2023] , which employs a prompt-based learning method, transforming numerical input and output data into prompts for effective forecasting in zero-shot settings. The TEMPO [Cao et al., 2023] adapts to changes in time series distribution by decomposing time series and adding different prompts for each component and obtain competitive performance in time series forecasting. In specialized domains like traffic [Xue et al., 2022] , finance [Zhang et al., 2023] and healthcare [Liu et al., 2023b] ,LLMs have also shown unique advantages. In this work, we aim to enhance OOD robusness for time series by fine-tuning LLMs with TTSO.3 Problem Formulation and AlgorithmNotations. X and Y represent the input and target spaces of samples, respectively. The predictor f φ = h ω • r θ consists of the representation function r θ (•) with parameter θ and the classifier h ω with parameter ω. The function f φ : X → Y maps time series X ∈ X to Y ∈ Y, where X ∈ R T ×F and Y ∈ R + , with 
preliminary: 
 Given a training dataset D train = {(X i , Y i )} N i=1 sampled from the distribution P train (X, Y ). In supervised learning, the goal is to learn an optimal predictor f φ * on D train such that f φ * generalizes well on a test dataset D test sampled from the distribution P test (X, Y ). In self-supervised contrastive learning, for a given time series X, we generate two augmented views, X a1 and X a2 , using augmentation methods a 1 , a 2 ∈ A. These augmentations produce the time series representationsR 1 = r θ (a 1 (X)) ∈ R T ×M and R 2 = r θ (a 2 (X)) ∈ R T ×M ,through the representation function r θ . The objective of contrastive learning is to minimize the distance between positive pairs (R 1 , R 2 ) while maximizing the distance between positive and negative pairs. The general formula for contrastive loss, as detailed in Zhao et al. [2022] , is formulated as follows ℓ con = ℓ align (r θ ; P, π) + λℓ reg (r θ ; P, π).(1) The first term aims to minimize the distance between positive pairs in the latent space, while the second term served as a regularizer prevents representation collapse. To evaluate the performance of the model, a classifier h ω is trained using the representation functionr θEQUATIONwhere the representation function r θ * (•) is optimized via the supervised loss in Eq. ( 2). The classification is performed using f φ * = h ω * • r θ * . However, the discrepancy between the training distribution P train (X, Y ) and the test distribution P test (X, Y ) poses a challenge for generalizing f φ * to test data. Directly optimizing ℓ sup (f φ (X), Y ) may lead to overfitting, compromising performance on unseen data. To mitigate this issue, invariant representation learning [Arjovsky et al., 2019] is employed to handle distribution shifts by learning robust invariant representations across diverse distributions. To achieve this, we begin with the following assumption.Assumption 1 (Invariant Assumption [Zhao et al., 2022] ). Considering K different environments (domains) E, there exists a random variable ψ(X) such that for any e, e ′ ∈ supp(E), it holds thatP (Y | ψ (X e )) = P (Y | ψ (X e ′ )).This assumption implies that for time series X observed in different environments, invariant rationales exist and their relationship with the corresponding labels remains stable. This stability ensures that predictions remain consistent across various environments, relying on these rationales. Assuming ψ(•) represents the representation function r θ (•) parameterized by θ, then it follows that r θ * (X e ) = r θ * (X e ′ ) = ψ(X).(3) In contrastive representation learning, where labels are not available, the theoretical analysis of the downstream performance is challenging. To address this, research [Zhao et al., 2022] bridges this gap by connects contrastive loss to downstream risks,R(h ω •r θ ;P π ) ≤ c∥h ω ∥ √ Kσ(ℓ align (r θ ;P,π)) 1 4 +∥h ω ∥τ (σ,δ)+ k P π (C k )∥e k -h ω •µ k (r θ ;P π )∥ (4)where c is a positive constant, τ (σ, δ) refers to a set of constants determined by the (σ, δ)augmentation, and C k corresponds to the sample set for class k. The first term is optimized during contrastive pre-training. The second term depends on data augmentations (σ, δ). The third term, related to the linear layer h ω , is optimized in downstream tasks. As shown by Eq. ( 4), contrastive learning on distribution P with augmentation function π essentially optimizes the upper bound of the supervised risk.Each environment E corresponds to a domain distribution P Si . To learn an invariant representation over the domain set P, we first provide a mathematical definition of invariant risk minimization.Definition 1 (Invariant Risk Minimization [Arjovsky et al., 2019] ). If there exists a classifier h 0 that is optimal for all domains in P, i.e., h 0 ∈ argmin h R (h • r θ ; P Si ) , ∀P Si ∈ P, then the representation function r θ elicits an invariant predictor h 0 • r θ across the domain set P.This definition is equivalent to learning features that have a stable association with the target variable, which has been theoretically and empirically proven to improve the transferability of supervised learning across different distributions [Arjovsky et al., 2019 , Zhao et al., 2022] .To address OOD challenges, GroupDRO [Sagawa et al., 2019] propose a mimax formulation to minimizes the maximum domain supervised loss to enhance robustness against unseen data. According to Eq. ( 4), contrastive learning optimizes the upper bound of supervised risk. Thus, we extend GroupDRO by replacing the supervised loss with a self-supervised contrastive loss, aiming to learn invariant representations. We further impose constraints on the group distribution q to mitigate the risk of overfitting to specific domains. This results in a bi-level optimization problem min θ,q K i=1 q i ℓ con (r θ ; D Si , π) s.t. q = arg maxEQUATIONwhere d (•, •) denotes a distribution distance metric, such as KL divergence, Wasserstein distance, or Euclidean distance, ∆ K is a probability simplex, and τ is a constant. Following previous work [Qian et al., 2019] , we adopt the Euclidean distance due to its strong convexity, which reults in faster convergence [Rakhlin et al., 2012] . The outer optimization seeks the best parameters across all domains to optimize overall performance, while the inner optimization, representing the group-level uncertainty, optimizes the worst-case distribution to enhance representation robustness.Definition 2 (Augmentation Robust Alignment Loss [Zhao et al., 2022] ). For any two augmentation methods a, a ′ ∈ A, the robust alignment loss is defined as followsEQUATION)Theorem 1 (Upper Bound of Risk Gap Between Augmented Domains [Shen et al., 2021] ). For any two augmentation methods a, a ′ ∈ A, representation function r θ and classifier h ω , we have sup a,a ′ ∈A ∥R (h ω • r θ ; P a ) -R (h ω • r θ ; P a ′ )∥ ≤ c∥h ω ∥ℓ ar (r θ ; P train ) .(1) sample-level(2) group-level(3) both train data (class 0) classifier sample distribution train data (class 1)Figure 1 : The depiction of sample-level, group-level, and combined uncertainties.Fix r θ , let h a = arg min hω R (h ω • r; P train ), we haveEQUATION)Theorem 1 states that minimizing ℓ ar (r θ ; P train ) makes the optimal predictor more consistent across different augmentation domains, i.e., minimize ℓ ar (r θ ; P train ) can enhance the invariance of the learned representation. Nonetheless, evaluating ℓ ar (r; P train ) involves a supremum operator, and the large set A makes accurate computation infeasible. Therefore, we propose an approximation for ℓ ar (r θ ; P train ). We start with the following reasonable assumption.Assumption 2. For any pair of augmentation methods a, a ′ ∈ A, they can be viewed as introducing specific perturbations δ to the sample X, i.e., a(X)= X + δ a , a ′ (X) = X + δ a ′ .Suppose δ a and δ a ′ are sampled from P perb , representing the distribution of perturbations induced by augmentation techniques. We adopt a Gaussian Mixture Model (GMM) [Jiao et al., 2022b] to accurately characterize the uncertain perturbation distribution. Thus, the distribution of δ is given byEQUATIONwhere π m represents the weight of the m th component in the mixture, and M m=1 π m = 1. The expression for ℓ ar (r θ ; P train ) can be written as sup δ∼p(δ;π,µ,σ) K i=1 q i ℓ align (θ, δ; D Si ). Consequently, we can further extend problem (5) to the following tri-level optimization problem. min θ,q,δ K i=1 q i ℓ con (θ, δ; D Si ) s.t. q = arg maxEQUATIONwhere C 1 and C 2 are constants. The third-level optimization addresses sample-level uncertainties by maximizing the alignment loss under the worst-case perturbation distribution. Figure 1 illustrates these concepts, showing how group-level and sample-level optimizations interact within the tri-level framework. To theoretically justify our approach in Eq. ( 10), we present the following theorem.Theorem 2 (Upper Bound on Target Error). Given the previous setup, let H be a hypothesis space of Vapnik-Chervonenkis (VC) Dimension d and h * T = min h∈H ϵ T (h).Let P α = {P α | P α = i α i P Si , i α i = 1, α i ≥ 0 }.If ĥ ∈ H is the empirical minimizer on P α , then for any δ and P C ∈ P α , with probability at least 1 -δ, ϵ T ( ĥ) ≤ 3ϵ T (h * T ) + λ + d H∆H (P C , P T ) + max i,j d H∆H (P Si ,EQUATION)where λ = 2 K i=1 α i ϵ Si (h * ) and C(δ, m, d) is a statistical term. d H∆H (•,•) is a metric function which measures differences in distribution [Ben-David et al., 2010] . ϵ Si (h) and ϵ T (h) is the source error and the target error.Discussion: Theorem 2 provides a theoretical framework for estimating performance on a new target distribution. The TTSO framework in Eq. ( 10) focuses on minimizing the terms d H∆H (P C , P T ) and max i,j d H∆H (P Si , P Sj ), thereby giving a tighter bound of target error to improve generalization ability. Proof of Theorem 2 and further discussion of our motivation are in Appendix A.2.Algorithm 1 SLA: Stratified Localization Algorithm Input: Training datasets {D Si }, learning rates η θ , η q , η δ , number of iterations T . Output: Optimized parameters θ * 1: Initialize parameters θ, q, δ and initial set of cutting planes S 0 . 2: for t = 0 to T -1 do 3:Update variable θ (t+1) , q (t+1) and δ (t+1) according to Eq. ( 19), ( 20) and ( 21) t+1) , q (t+1) , δ (t+1) ) > ε then 6:4: if t mod k == 0 then 5: if h(θ (Add new cutting planes to set S t+1 according to Eq. (1) 7:end if 8:end if 9: end for 10: return θ (T ) However, solving the constrained tri-level optimization is extremely challenging. In the next subsection, we introduce a stratified localization algorithm to address this problem effectively.Due to the hierarchical structure of the tri-level problem, we develop a stratified version of the localization method [Boyd and Vandenberghe, 2007, Jiao et al., 2023] to tackle the problem presented in Eq. ( 10). First, we use exterior penalty method to reformulate the third level problem, the resulting problem is minEQUATIONwhere P 3 is a penalty term defined asP 3 = ρ 1 (max(0, ∥µ∥ -C 1 )) 2 + ρ 2 (max(0, ∥σ∥ -C 2 )) 2 + ρ 3 ( M m=1 π m -1) 2 + ρ 4 (max(0, -π m )) 2, and ρ i are penalty coefficients. Given that the third-level optimization is a constraint for the second-level optimization, we employ T 3 steps of gradient ascent to approximate the third-level problem. This technique is commonly used in previous bi-level optimization studies [Ji et al., 2021] . By defining f 3 (θ, q ′ , δ ′ ) = K i=1 q ′ i (ℓ align (θ, δ ′ ; D Si ) -P 3 and using the exterior penalty method, the resulting optimization problem can be expressed asEQUATIONwhereP 2 = λ 1 ( K i=1 q i -1) 2 + K i=1 λ 2 max(0, -q i )+λ 3 ∥δ-δ (0) - T3-1 i=0 η δ ∇ δ ′ f 3 (θ, q ′ , δ ′ )∥ 2 . Likewise, we perform T 2 steps of gradient ascent to replace the second level optimization problem. With the definition of f 2 (θ, q ′ , δ) = K i=1 q ′ i ℓ con θ, δ; D tr Si -P 2 , φ(θ, δ) = arg max q ′ f 2 (θ, q ′ ,δ) and h(θ, q, δ) = ∥q -φ(θ, δ)∥, Eq. ( 13) can be reformulated as followsmin θ,q,δ K i=1 q i ℓ con (θ, δ; D Si ) s.t. h(θ, q, δ) = 0. (14) Let f 1 (θ, q, δ) = K i=1 q i ℓ align (θ, δ; D Si ).Considering the approximations of q and δ, the above problem can be relaxed as min θ,q,δ f 1 (θ, q, δ)EQUATIONwhere ε > 0 is a constant. Inspired by the polyhedral approximation method [Bürger et al., 2013] , we utilize cutting planes to approximate the feasible region with respect to h(θ, q, δ) ≤ ε. In the (t + 1) th iteration, the set of cutting planes, denoted as S t , is defined as followsEQUATIONwherea i ∈ R N , b i ∈ R M , c i ∈ R H , d i ∈ R 1, and |S t | represents the number of cutting planes in S t . Then Eq. ( 15) can be expressed as the following approximation problem min θ,q,δ f 1 (θ, q, δ)EQUATIONThe penalty function with respect to Eq. ( 17) can be described asEQUATION)In (t + 1) th iteration, the variables are updated as followsEQUATIONEQUATIONEQUATION)Throughout the iteration process, the set of cutting planes S t is updated every k iterations for a tighter and more accurate polyhedral approximation. Before adding new cutting planes, we first check whether (θ t+1 , q t+1 , δ t+1 ) is a solution for Eq. ( 15). If it is not a feasible solution to Eq. ( 15), i.e., h(θ, q, δ) > ε, new cutting planes are added to S t based on Theorem 3 and Proposition 1.Algorithm 1 provides details of the proposed method. Theorem 3. Let T 2 = 1. If a first-order Taylor expansion is applied to the function f 2 (θ, q, δ) at the point (θ, δ), it follows that the function h (θ, q, δ) is convex with respect to (θ, q, δ). The detailed proof can be found in Appendix A.3. Proposition 1. Given the convexity of the function h(θ, q, δ), a new cutting plane is generated when the condition h(θ, q, δ) > ε is not met. This cutting plane is formally expressed asEQUATION)For the detailed derivation and proof of Proposition 1, please see Appendix A.1.LLMs have garnered considerable attention in time series applications [Jin et al., 2023 , Zhou et al., 2023] . The emergent abilities of LLMs, especially in OOD scenarios, largely depends on the robustness of their representations [Wang et al., 2023 , Chu et al., 2023] . This section connects the established theoretical foundation with the practical application of fine-tuning LLMs for time series OOD generalization. We adapt TTSO framework for fine-tuning LLMs to enhance the performance in time series OOD generalization. Our proposed method involves a dual-stage fine-tuning method tailored for time series. The main process of fine-tuning are described below.Time Series Pre-processing. Preprocessing starts with an input projection layer to bridge the gap in dimensions between raw time series data and the LLM's native embedding dimension. This step is crucial for the LLM's effective integration of time series. Following this, positional encoding is applied to preserve the sequential integrity of the time series.Dual-stage Fine-tuning Method. In the first stage, we employ TTSO framework to fine-tune LLMs, in line with the previously mentioned tri-level optimization framework as illustrate in Eq. ( 10). We adopt the contrastive loss function designed for time series from Yue et al. [2022] . In the second stage, the learned weights of the LLM, including the projection layer, are transferred to the downstream fine-tuning stage for time series classification. To retain the knowledge learned by the LLM from the corpus, we follow Chang et al. [2023] , Zhou et al. [2023] by fixing the weights of the fully connected and attention layers, using Layer Normalization Tuning [Lu et al., 2022a] to adjust only the layer normalization parameters, making the affine transformation trainable.Constrained Optimization for Fine-Tuning. Research [Wortsman et al., 2022] indicates that adopting radical strategies for fine-tuning models, such as larger learning rates, can reduce out-of-distribution robustness. Unconstrained optimization of model parameters during fine-tuning can lead to knowledge forgetting issues and decrease the model's generalization ability, as mentioned in Xuhong et al. [2018] . Therefore, during fine-tuning for downstream tasks, we impose constraints on the parameters, following Xuhong et al. [2018] , resulting in the following optimization problem minEQUATIONwhere θ 0 and θ respectively denote the weights from the first and second fine-tuning phases of the LLMs. More details of fine-tuning LLMs can be found in appendix D.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: In this paper, all theoretical results are accompanied by a full set of assumptions and complete proofs. These are clearly stated and numbered within the main text and are cross-referenced appropriately. Detailed proofs for major theorems are provided in the Appendix.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
convergence analysis: 
 Assumption 3 (Lipschitz Continuity of Gradient). Assume that the gradient of the function F is Lipschitz continuous, i.e., for any x, y, there exists L > 0 such that:∥∇F (x) -∇F (y)∥ ≤ L ∥x -y∥ .(24) Assumption 4 (Unbiasedness and Variance Bound of Stochastic Gradients). Assume for the stochastic gradients g θ , g q , g δ , the following conditions are satisfiedEQUATIONwhereE ζ t j [•]denotes the expectation over the ζ t j . Assumption 5 (Bounded Gradient). Assume that the gradient of the function F is bounded, i.e., ∀t, ∥∇ θ F (θ t , q t , δ t )∥ 2 ≤ α 2 1 , ∥∇ q F (θ t , q t , δ t )∥ 2 ≤ α 2 2 , ∥∇ δ F (θ t , q t , δ t )∥ 2 ≤ α 2 3 . Definition 3 (ϵ-Stationary Point). Following Xu et al. [2023] , Jiao et al. [2024] , a point (θ, q, δ) is considered an ϵ-stationary point (where ϵ > 0) of a differentiable function F if the sum of squares of its gradients on these variables satisfies ∥∇G t ∥ ≤ ϵ. Let T (ϵ) be the index of the first iteration that satisfies ∥∇G t ∥ ≤ ϵ, i.e., T (ϵ) = min{t | ∥∇G t ∥ ≤ ϵ, t > t 1 }. Theorem 4 (Convergence Guarantee). With the continuous addition of cutting planes, the optimal objective value of the approximated problem, delineated in Eq. ( 17), is guaranteed to converge monotonically. For further details, see the proof of Theorem 4 in appendix A.4. Theorem 5 (Convergence Rate). Under the assumptions 3, 4, and 5, by setting the step-sizes as η θ = η q = η δ = 1 √ T1-t1 and the batch size as B, for a given ϵ, it follows thatEQUATION)where F * represents the lower bound of F . The proof of Theorem 5 is detailed in appendix A.5.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Our paper provides detailed descriptions of the experimental setup. Detailed information regarding datasets, domain setting, data pre-processing, network architecture and hyperparameters can be found in Appendix B.1, B.2, B.3 and C. 
experiment: 
 To evaluate the proposed TTSO framework, we conduct experiments on 6 real-world time series datasets using the leave-one-domain-out setting, including HHAR [Blunck et al., 2015] , PAMAP [Reiss, 2012] , WESAD [Philip Schmidt et al., 2018] , SWELL [Koldijk et al., 2014] , USC-HAD [Zhang and Sawchuk, 2012] and DSADS [Barshan and Altun, 2013] . We compare with baseline method ERM [Vapnik, 1991] and 8 general OOD generalization methods: IRM [Arjovsky et al., 2019] , GroupDRO [Sagawa et al., 2019] , ANDMask [Parascandolo et al., 2020] , RSC [Huang et al., 2020] , Mixup [Zhang et al., 2017] , VERx [Krueger et al., 2021] , DIFEX [Lu et al., 2022b] .And we further compare with 2 recent strong approach in time series: AdaRNN [Du et al., 2021] and GILE [Qian et al., 2021] . We also include DIVERSIFY [Lu et al., 2023] , DFDG [Zhang et al., 2021] , and CCDG [Ragab et al., 2022] , three methods specifically designed for time series OOD generalization. To guarantee a fair comparison, we implement all methods using the same backbone architecture (except AdaRNN and GILE), TCN [Bai et al., 2018] , a model widely used in time series analysis. In addition, we fine-tune the pre-trained Large Language Model, GPT2 [Radford et al., 2018] , within our TTSO framework to harness its sophisticated representation learning capabilities. Detailed information regarding datasets, domain setting, data pre-processing, network architecture and hyperparameters are provided in Appendix B.1, B.2, B.3 and C.We report the average results over 3 runs for each dataset, along with the standard deviation. The results for the HHAR, PAMAP, and WESAD datasets are shown in Tables 1, where our method outperforms the second-best baseline by 2.8%, 4.8%, and 4.9% respectively. Additional results are provided in Appendix E.1 (Table 4 ). These results demonstrate the superiority and effectiveness of the TTSO framework, as it accounts for both sample-level and group-level uncertainties, which optimizes the upper bound in Theorem 2.Compared to traditional methods like ERM, IRM, and GroupDRO, both TTSO and TTSO * show more consistent and generally superior performance in OOD generalization, highlighting the advantages of a tri-level learning framework. The TTSO * method, which incorporates LLM fine-tuning, consistently outperforms other approaches, demonstrating the effectiveness of LLM with TTSO fine-tuning in enhancing OOD generalization for time series. Concurrently, the TTSO method, even without LLM fine-tuning, shows strong generalization performance, especially on the HHAR dataset where it closely matches TTSO * results. This indicates that the TTSO framework is highly effective in generalizing across different scenarios, even in the absence of LLM. This ablation study is conducted to further understand the impact of our TTSO framework's fine-tuning on model performance. We compare four distinct variants: a pretrained GPT2 fine-tuned with TTSO (TTSO ++ ), a pretrained GPT2 without TTSO fine-tuning (TTSO +-), a randomly initialized GPT2 fine-tuned with TTSO (TTSO -+ ), and a randomly initialized GPT2 without TTSO finetuning (TTSO --). This comparison helps in quantifying the effectiveness of the TTSO fine-tuning strategy in enhancing the model's OOD generalization capabilities.The ablation results are presented in Figure 2 . From this results, we can see that: (a) TTSO ++ demonstrates the best performance in all scenarios, further validating that the combination of a pre-trained GPT2 model with TTSO fine-tuning can significantly improve the model's OOD generalization capabilities. (b) Although TTSO +-does not employ TTSO fine-tuning, it still exhibits relatively good performance. This suggests that the pre-trained GPT2 model has an intrinsic capacity for OOD generalization, consistent with previous empirical studies [Zheng et al., 2022 , Hendrycks et al., 2020] . (c) Compared to TTSO --, TTSO -+ applies TTSO to fine-tune on a randomly initialized GPT2, TTSO ++ achieves improved performance.This demonstrates that even in the absence of a pre-trained model, TTSO fine-tuning can effectively enhance the model's OOD generalization capabilities, though not as significantly as that with a pre-trained GPT2.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: While the data used in our study is publicly available, we are currently unable to provide open access to the code. However, we have included detailed descriptions of the data access, preprocessing steps, model architecture, and experimental setup in the paper and Appendix. These details should be sufficient for others to reproduce the experimental results.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 Existing OOD generalization methods mainly focus on sample-level uncertainties or group-level uncertainties, often overlooking the interplay between these two aspects. In light of this, we propose the TTSO framework to integrate both sample-level and group-level uncertainties within a unified tri-level learning approach, thereby enhancing the model's robustness and adaptability in facing diverse and unforeseen distribution shifts. In addition, this innovative framework introduces a fresh perspective for the development and analysis of the Out-of-Distribution (OOD) generalization problem. Based on this formulation, we develop a stratified localization algorithm for the tri-level optimization problem and provide theoretical analysis regarding the iteration complexity of the proposed algorithm. Comprehensive studies have been carried out to assess the performance of the proposed algorithm and substantiate the theoretic claims. It is seen that TTSO with LLM can considerably improves the performance of time series OOD generalization. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report the mean accuracy and standard deviation across three runs with different random seeds.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide the type of compute workers (GPU) in the Appendix C Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: This paper does not use existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: This paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Proof of Proposition 1Since h(θ, q, δ) is a convex function, we have thatEQUATION)According to theorem 3, combine with Eq. ( 15) and Eq. ( 27), the new cutting plane will be generated asEQUATION)From the inequalities, we can derive the coefficients a i , b i , c i and d i as followsEQUATIONwhich concludes the proof.A.2 Proof of Theorem 2For a sample distribution P S on inputs space X with a binary labeling function f and a hypothesis h, the error (or risk) is defined as followsϵ S (h, f ) = E x∼P S [|h(x) -f (x)|].(30) For simplicity, we use the shorthand ϵ S (h) = ϵ S (h, f S ) for source risk and ϵ T (h) = ϵ T (h, f D ) for target risk, where f S and f D represent the labeling function of source and target domain, respectively.To bound the target error, following Ben-David et al. [2010] , we define the H-divergence. Given source distribution P S and target distribution P T over input space X , let H be a hypothesis class on X . The H-divergence between P S and P T is d H (P S , P T ) = 2 supEQUATIONwhereI h = {x ∈ X : h(x) = 1, h ∈ H}.In addition, for a hypothesis space H, the symmetric difference hypothesis space d H∆H is the set of hypotheses. And the d H∆H in [Ben-David et al., 2010] is defined asEQUATION)Theorem 6. (Modified from Theorem 4 in [Ben-David et al., 2010] ) Let H be a hypothesis space of VC dimension d. For each i ∈ {1, . . . , K}, let D Si be a labeled sample of size β i m drawn from P Si and labeled according to function f Si . If ĥ ∈ H is the empirical minimizer of εα (h) for a fixed weight vector α on these samples and h * T = min h∈H ϵ T (h) is the target error minimizer, then for any δ ∈ (0, 1), with probability at least 1 -δ, ϵ T ( ĥ)EQUATIONwhere λ i = min h∈H {ϵ T (h) + ϵ Si (h)}.According to the definition of d H∆H in Eq. ( 32), we have d H∆H (P Si , P T ) = 2 sup h,h ′ ∈H Pr x∼P S i [h(x) ̸ = h ′ (x)] -Pr x∼P T [h(x) ̸ = h ′ (x)] = 2 sup h,h ′ ∈H Pr x∼P S i [h(x) ̸ = h ′ (x)] -Pr x∼P C [h(x) ̸ = h ′ (x)] + (Pr x∼P C [h(x) ̸ = h ′ (x)] -Pr x∼P T [h(x) ̸ = h ′ (x)])| ≤ 2 sup h,h ′ ∈H Pr x∼P S i [h(x) ̸ = h ′ (x)] -Pr x∼P C [h(x) ̸ = h ′ (x)] + 2 sup h,h ′ ∈H |Pr x∼P C [h(x) ̸ = h ′ (x)] -Pr x∼P T [h(x) ̸ = h ′ (x)]| = d H∆H (P Si , P C ) + d H∆H (P C , P T ) (34) Since P α = {P α | P α = i α i P Si , i α i = 1, α i ≥ 0Substitute Eq. ( 36) into Eq. ( 33), we obtain thatEQUATIONSinceλ i = min h∈H {ϵ T (h) + ϵ Si (h)} = ϵ T (h * ) + ϵ Si (h * ), by setting λ = 2 N i=1 α i ϵ Si (h * ) and C(δ, m, d) = 2 N i=1 α 2 i βi d log(2m)-log(δ) 2myields the proof.Theorem 2 provides a theoretical framework for estimating performance on a new target distribution. The first term, ϵ T (h * T ), represents the target error under the ideal hypothesis h * T . The second term, λ = 2 N j=1 α i ϵ Si (h), aggregates the combined error over all source distributions weighted by α i and can be minimized via supervised loss with labels. The third term, d H∆H (P C , P T ), measures the distributional discrepancy between a composite source distribution and the target distribution. The fourth term, max i,j d H∆H (P Si , P Sj ), quantifies the maximum discrepancy between any two source distributions. The last term, C(δ, m, d), is a statistical term which depends on the confidence level δ, sample size m, and VC dimension d.The tri-level learning framework proposed in Eq. ( 10) aims to minimize the third term, d H∆H (P C , P T ), and the fourth term, max i,j d H∆H (P Si , P Sj ). These two terms correspond to the group-level and sample-level uncertainties, respectively. Below, we discuss how these two terms align with the motivation for our tri-level optimization.For the term d H∆H (P C , P T ), the goal of time series OOD generalization is to learn a model that generalizes well to unseen domain distributions, which makes direct optimization infeasible due to the unavailability of target dataset. To minimize this discrepancy, we can only enlarge the set P C . Specifically, we manipulate δ-perturbations applied to individual samples in the third level of our tri-level learning framework. By optimizing these perturbations, we explore a broader range of variations within each source domain, which potentially minimizes the term d H∆H (P C , P T ), enhancing the robustness of learned representations.For the term max i,j d H∆H (P Si , P Sj ), the second-level optimization in our tri-level framework adjusts the weights α i that define the mixture of source distributions P C . By dynamically modifying these weights based on the 'worst-case' distribution, we minimize the term max i,j d H∆H (P Si , P Sj ).Our approach not only enhances representation invariance across diverse domains but also improves the model's resilience against variations within individual samples. The effectiveness of this tri-level framework is rooted in the interdependence between the problems at each level; adjustments in one level influence the conditions and outcomes of the others. This demonstrates the necessity of our tri-level learning optimization, as it requires a coordinated strategy that simultaneously considers sample-level, group-level, and parameter-level dynamics.First, the first-order Taylor expansion of f 2 (θ, q, δ) at the point (θ, δ) is obtained as followsf2 (θ, q ′ , δ) = f 2 (θ, q ′ , δ) + ∇ θ f 2 (θ, q ′ , δ) T (θ -θ) + ∇ δ f 2 (δ, q ′ , δ) T (δ -δ).(38) Since T 2 = 1, therefore, we have: φ(θ, δ) = q ′ 0 -η q ∇ q ′ f2 (θ, q ′ , δ) .(39) Combine with Eq. ( 38) and ( 39), we can obtain thatEQUATIONwhereC = q ′ 0 -η q ∇ q ′ f 2 (θ, q ′ , δ)+∇ q ′ ∇ θ f 2 (θ, q ′ , δ)θ +∇ q ′ ∇ δ f 2 (θ, q ′ ,δ)δ is an affine function. Therefore, φ(θ, δ) is a convex function. According to preserve convexity [Boyd and Vandenberghe, 2004] , h(θ, q, δ) = ∥q -ϕ(θ, δ)∥ is convexity, which concludes the proof of theorem 3.Assume that in the t th iteration, a new cutting plane is added, and the selected point (θ t+1 , q t+1 , δ t+1 ) always lies within the region R c t formed by the cutting plane set C t , we haveEQUATIONLet H t denote the feasible region of problem in Eq. ( 17) at the t th iteration, and R represent the feasible region of problem in Eq. ( 15), then it follows thatEQUATIONLet F (θ t , q t , δ t ) denote the optimal value of problem in Eq. ( 17) at the t th iteration, and f * 1 represent the optimal value of the problem Eq. ( 15). Based on equation ( 42), we can obtainEQUATIONIt's seen that the sequence F (θ t , q t , δ t ) is monotonically increasing. As T 1 → ∞, f 1 monotonically converges to a certain fixed value. It is worth mentioning that h(θ, q, δ) = ∥q -ϕ(θ, δ)∥ is a convex function. Since the sublevel set of a convex function is convex, the feasible region of problem in Eq. ( 17) is convexity. This implies that the iterative procedure, by continuously adding a cutting plane, is progressively converging to the optimal value f * 1 of the problem as referenced in Eq. (15).To begin with, we introduce a fundamental lemma that is pivotal for the subsequent analysis.Lemma 1. For 1 m m j=1 g θ θ t , q t , δ t ; ζ t j , 1 m m j=1 g q (θ t , q t , δ t ; ζ t j ), 1 m m j=1 g δ θ t , q t , δ t ; ζ t j , they are unbiased and bounded, that is,EQUATIONEQUATIONEQUATION)Here, E {ζ t j } [•] denotes the expectation with respect to a set of variables{ζ t 1 , • • • , ζ t m }.Taking the variable θ as an example, according to Assumption 4, for all i = 1, • • • , n, we haveEQUATION)Based on Assumption 5, the variance of ∇ θ F (θ t , q t , δ t ) is bounded, from which we can deduceEQUATION)The proofs of Eq. ( 45) and Eq. ( 46) follow a similar logic. Thus, we complete the proof of Lemma 1.Combine with lemma 1, we now proceed to derive Theorem 5. Under Assumption 3 that the gradient of F is Lipschitz continuous, for t > t 1 , it follows thatEQUATION)Taking the expectation of both sides of Equation ( 49) with respect toEQUATION)The inequality (i) is based on lemma 1. Taking the total expectation of both sides of Eq. ( 50), we haveEQUATIONwhere E[•] denotes the expectation over all terms. Summing Eq. ( 51) from t = t 1 to t = T 1 -1, we obtainE F θ T1 , q T1 , δ T1 ≤ E F θ t1 , q t1 , δ t -η θ T1-1 t=t1 E ∇ θ F θ t , q t , δ t 2 -η q T1-1 t=T1 E ∇ q F θ t , q t , δ t 2 -η δ T1-1 t=t1 E ∇ δ F θ t , q t , δ t 2 + T1-1 t=t1 L (η θ ) 2 2 α 2 1 + σ 2 1 m + T1-1 t=t1 L (η q ) 2 2 α 2 2 + σ 2 2 m + T1-1 t=t1 L (η δ ) 2 2 α 2 3 + σ 2 3 m =E F θ t1 , q t1 , δ t -η θ T1-1 t=t1 E ∇ θ F θ t , q t , δ t 2 -η q T1-1 t=T1 E ∇ q F θ t , q t , δ t 2 -η δ T1-1 t=t1 E ∇ δ F θ t , q t , δ t 2 + L 2 T1-1 t=t1 (η 2 θ α 2 1 + η 2 q α 2 2 + η 2 δ α 2 3 ) + L 2m T1-1 t=t1 (η 2 θ σ 2 1 + η 2 q σ 2 2 + η 2 δ σ 2 3 ) (52) Let η θ = η q = η δ = 1 √T1-t1 , Combining with Eq. ( 52), we have thatEQUATION)Combining the definition of ϵ-stationary point described in Definition (3) with Eq. ( 53), we haveEQUATIONthat is,EQUATION)According to Eq. ( 55), we can obtainT 1 (ϵ) ∼ 1 ϵ 2 (56)Hence, it can be concluded that there exists a T 1 (ϵ) such that ∥∇G t ∥ = E ∇ θ F θ t , q t , δ t 2 + ∇ q F θ t , q t , δ t 2 + ∇ δ F θ t , q t , δ t 2 ≤ ϵ, which concludes the proof of Theorem 5.B.1 Datasets Information HHAR [Blunck et al., 2015] collected activity data from 9 subjects engaging in 6 activities, using smartphones that capture 3D accelerometer data from various positions. PAMAP [Reiss, 2012] encompasses 18 physical activities data from 9 subjects, recorded using wearable sensors monitoring physiological and movement metrics. WESAD [Philip Schmidt et al., 2018] focuses on stress and affective state detection from 15 subjects, employing wearable sensors for ECG, EMG, respiration, and temperature under varied conditions. SWELL [Koldijk et al., 2014] recorded stress responses in a work environment from 25 participants using ECG, EDA, and heart rate sensors during typical office tasks under stressors. USC-HAD [Zhang and Sawchuk, 2012] comprises detailed motion and orientation data from 14 subjects performing various activities, captured via a MotionNode device with high sampling rate. DSADS [Barshan and Altun, 2013] consists of 19 activities recorded from 8 subjects at Bilkent University, using body-worn sensors on torso and limbs, with data segmented for detailed analysis. Table 2 shows the information of the 6 datasets we used in our experiments. The domain setting is summarized in Table 3 . This setting is done to balance the number of samples and classed across different domains.For all datasets, we configure the window size as 128 and the step size as 64, resulting in a 50% overlap between two adjacent time series samples. Each sample is standardized using the formula x = x-µ σ , where µ and σ represent the mean and standard deviation of the dataset, respectively. It's important to note that the µ and σ used here is specific to each domain, rather than the entire dataset. This approach is intended to maximize the distinction between data from different domains.Our baseline experiments were conducted using a network architecture consisting of 10-layers dilated convolutions network. The dilation rate for each layer is set to 2 k , where k is the layer number. We used the same kernel size of 3 across all layers. Optimization was performed using the Adam optimizer with a weight decay of 3 × 10 -4 . For all baseline experiments, we set the batch size to 256 and the learning rate to 0.002. The training was set to run for a maximum of 50 epochs. All the methods are implemented with PyTorch [Paszke et al., 2019] version 1.7.1 on an NVIDIA GeForce RTX 4090 graphics card.In the TTSO fine-tuning experiments, we employed GPT-2 as the language model. Fine-tuning was performed in two stage: In the first stage, the learning rate for the large model was 1 × 10 -4 , and for the input embedding, it was set at 0.001. In the second phase, we adjusted the learning rate for the large model to 5 × 10 -5 , aiming to further refine the model's performance. During the evaluation phase, we froze the parameters of the language model, only fine-tuning the classifier with a learning rate of 0.003 to adapt to the specific classification tasks. The batch size was consistently set at 16 for all experimental stages.The structure of LLM Fine-tuning with TTSO is illustrated in figure 3a Figure 3 : Structure of LLM Fine-tuning with TTSO, illustrating the two-phase approach starting with alignment fine-tuning followed by downstream fine-tuning, adapted specifically for time series out-of-distribution generalization tasks.Input Embedding: The first is the input embedding, where raw time series is transformed into embedding space that is amenable for processing by the language model. In our experiments, the input embedding is a linear layer. As shown in Figure 3 , the time series (indicated by the dashed box below the waveform) is combined with positional encoding to form the input representation.Language Model: This is the core part of the model, typically comprising multiple pretrained transformer encoder. This model is used for processing the input embeddings and producing advanced feature representations for subsequent classification tasks. Note that, to retain the intrinsic information of the language model, only the parameters of layer normalization can be tuned.Classifier: In the second stage of fine-tuning, the classifier tailors the language model to the specific time series classification task. It takes the advanced feature representations from the language model and fine-tunes the model for downstream tasks. We use a linear layer for the classifier, with cross entropy as the supervised loss.Contrastive Loss: A contrastive loss function is employed to enhance the discriminative of the representations in the first stage of the fine-tuning process. This loss function aims to ensure that the representations of similar time series samples are brought closer together in the representations space, while representations of dissimilar samples are pushed apart. Specifically, during this stage, the contrastive loss acts as a guiding signal for the language model, encouraging it to learn representations that effectively capture the underlying patterns and distinctions within the time series data, thereby adapting the language model to time series data more effectively.We conducte on more datasets to demonstrate the superior performance of our framework. The results is summarized in Table 4 . To illustrate the concepts of sample-level and group-level uncertainties, we use the x-axis values from accelerometer data collected by the 'samsungold_1' device from four users in the HHAR dataset.In Figure 4 , sample-level uncertainty is shown by plotting time series data from a specific label (e.g., 'walking'), where each line represents a different time window. The variations among these lines illustrate the inherent noise, which represents sample-level uncertainty. Figure 5 demonstrates the group-level uncertainty by displaying the distribution of x-axis values from the accelerometer across different groups (users). Each color represents a distinct group, and each group's unique characteristics contribute to the overall group-level uncertainty.To further investigate the impact of various architectures and parameter settings of LLMs, we conducted additional ablation experiments that focused on different LLM architectures and parameter sizes (e.g., base model and large model). These experiments included encoder-only models (e.g., BERT), decoder-only models (e.g., GPT-2), and encoder-decoder models (e.g., BART) to determine which configurations yield the greatest benefits during fine-tuning. The results are summarized in the table 5 . The results indicate that decoder-only architectures, specifically the GPT-2 base model in this experiments, achieve the best performance. However, increasing the number of parameters in all three architectures leads to a significant drop in performance across 3 architectures for time series OOD generalization. To further explore how the number of parameters affects performance, we conducted experiments using GPT-2 models with varying numbers of Transformer layers on 3 datasets to evaluate their OOD• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This paper focuses on time-series out-of-distribution generalization with LLMs, provides new methods that use uncertainties and localization, and gives theoretical analysis on the generalization properties and convergence. Reviewers appreciated the application of LLMs to the problem domain and found good improvements in the experimental section. Although the theoretical work may not be easily accessible to a wide audience, most of the reviewers appreciated the work to show convergence properties.

While the paper restricted its focus to time series and did not test on other modalities, reviewers noted that the methods did not seem to rely on any specific aspects of time-series problems, and in that sense could have been tested more broadly.

In the discussion phase, the authors showed new results using a variety of architectures and LLM sizes. Somewhat curiously, larger LLMs led to worse performance on the paper’s tasks. The authors are encouraged to think more about the possible causes of this observation and provide substantive discussion, as a practitioner could pick up this technique, think to use the most advanced LLM available, and find poor OOD generalization results.

Reviews: review: paper_summary: paper_summary: Out-of-Distribution（OOD）generalization in ML emphasizes on improving model adaptability and robustness aginst unseen and potentially adversarial data. This paper explores OOD generalization for time series data with pre-trained Large Language Models and proposes a novel tri-level learning framework to handle the data distribution uncertainties. It goal is to address not only sample-level but also group-level uncertainties in the new dataset. The paper offers a theoretical analysis to justify the method and develops a cutting plane strategy for the tri-level optimization problem. It demonstrates guaranteed convergence. Extensive experiments on real-world datasets confirm the effectiveness and efficiency of the proposed method.
strength_weakness: strength_weakness: While the paper is well organized overall, the clarity of this paper can be further enhanced via providing examples to illustrate the concepts of sample-level and group-level uncertainties in time series.
questions: questions: Can the TSSO framework be employed to manage data uncertainties in other type of data, for example natural language data?
limitations: limitations: N/A
paper_summary: paper_summary: The paper studies the problem of OOD generalization in time series tasks, building on recent observations that use data level uncertainties and group level uncertainties. This has been a successful way to build robust, transferrable representation. The paper additionally also includes an additional maximization for data augmentation that makes it the "tri-level" learning framework. The paper also theoretically analyzes generalization properties of the proposed algorithm. 

The TTSO method is also used to fine-tune LLMs which is used for time series classification in OOD scenarios. In this part, it builds on some recent works that utilize pre-trained or fine-tuned LLMs in other domains, leveraging the superior feature learning capacities of these large frontier/foundation models.
strength_weakness: strength_weakness: On the surface of it, the paper shows a good improvement over several commonly used methods in OOD generalization, improving on different benchmarks. However, many aspects of the paper are unclear:  
* Two of the main aspects of the paper are not sufficiently well motivated in my opinion -- 
	* (a) _OOD generalization in time series_: Why is this particular method effective for time series? as far as the assumptions made in the paper go, this is a general technique that can be applied to any arbitrary dataset/modality. While it is true that time series problems do not receive as much attention, that alone is not motivation enough. What assumptions in this paper restrict the use of TTSO for a broader set of modalities? If none, then how does this compare on other benchmarks?
	* (b) _LLMs for augmentation_: Why is the LLM necessary? The idea that LLMs can be used to produce different views of the sample is interesting, but the empirical performance appears to be too be marginal at best (+1.4% gain vs not using it from table 1). It doesn't seem to justify the additional computational burden for the small gain in performance. It would be helpful if the paper clearly articulates what the hypothesis is exactly here with regard to LLMs -- the ablation study in Fig 2 is a good start -- what kinds of LLMs benefit when fine-tuning? Is there a bias within the GPT-2 model used that is beneficial for these datasets? What about more advanced architectures/or LLMs trained on larger datasets, are they expected to give a bigger boost in performance or does it plateau?
* I think the paper is interesting and has a lot to say, but i would recommend making the core hypothesis and idea cleaner as I have outlined above.
questions: questions: Please see above
limitations: limitations: Yes, it appears so.
paper_summary: paper_summary: The paper explores the challenge of OOD generalization in time series data using pre-trained LLMs. The authors propose a novel framework TTSO that integrates both sample-level and group-level uncertainties. They also introduce a stratified localization algorithm tailored to this tri-level optimization problem, theoretically demonstrating its convergence. The paper presents extensive experiments to validate the method’s effectiveness.
strength_weakness: strength_weakness: 1. The tri-level learning framework might be overly complex for practical applications, potentially limiting its adoption.
2. The proposed method, especially the stratified localization algorithm, may incur high computational costs, which could be a barrier for large-scale applications.
3. The paper could benefit from a more comprehensive comparison with other state-of-the-art methods in time series OOD generalization.
4. There is a need for more discussion on the real-world applicability and potential limitations of the proposed method in various domains.
5. Some sections of the paper are dense and challenging to follow, particularly the theoretical analyses, which might be difficult for a broader audience to understand.
questions: questions: 1. How does the tri-level learning framework handle scenarios with highly imbalanced time series data?
2. Can the proposed stratified localization algorithm be adapted for other types of data beyond time series?
3. What are the specific computational requirements for implementing the proposed method on large-scale datasets?
limitations: limitations: NA
paper_summary: paper_summary: The paper presents a novel approach for time series out-of-distribution generalization via pre-trained large language models.
The authors introduce a Tri-level learning framework that combines sample-level and group-level uncertainties accompanied by a theoretic perspective. Furthermore, a stratified localization algorithm is proposed for the tri-level optimization problem followed by a theoretical convergence guarantee. The paper demonstrates performance gain in six real-world time series datasets, demonstrating the effectiveness of the method.
strength_weakness: strength_weakness: I do not see any major weaknesses.
questions: questions: Can this method be adapted for OOD time series regression and forecasting?
limitations: limitations: The authors adequately addressed the limitations.

Comments: title: Summary of Contributions and Modifications
Acknowledgement of Rebuttal
comment: Thank you for the reply. I have read through your rebuttal. I will keep my score unchanged.
Dear Reviewer tk3X,

With the discussion period ending soon, I wanted to thank you for your valuable feedback on our paper. We have made revisions based on your suggestions, which have significantly improved our work.

If you find that our revisions have addressed your concerns, we would greatly appreciate any additional feedback you may have.

Thank you for your time and consideration.

Best regards,

Authors #16764
We sincerely thank the Area Chairs for managing the review process and the reviewers for their thoughtful consideration and evaluation of our work. To facilitate further discussions, we have summarized our contributions and modifications as follows:

**Contributions**

* We propose a novel **tri-level learning** framework, TTSO, to tackle time series OOD generalization with pre-trained LLMs. By integrating both sample-level and group-level uncertainties, TTSO advances beyond traditional single or bi-level methods in OOD generalization. We provide a theoretical analysis based on the Vapnik-Chervonenkis dimension to elucidate **generalization properties** of TTSO, and demonstrate its effectiveness through extensive real-world datasets.
* We develop a stratified localization method using cutting planes, which is **more computationally** efficient compared to hypergradient-based tri-level optimization methods. And the decomposable nature of cutting planes provides a promising pathway for distributed implementations of TTSO, allowing the algorithm to be effectively applied to large-scale applications.
* We conducted a **thorough theoretical analysis** of the proposed algorithm. The proposed stratified localization algorithm for tri-level optimization shows guaranteed convergence and we theoretically derive that the iteration complexity of the proposed algorithm for achieving an $\epsilon$-stationary point is bounded by $\mathcal{O}(\frac{1}{\epsilon^2})$.

**Modifications**

* (Clarity Enhancements) We have included examples using the HHAR dataset to better illustrate the concepts of sample-level and group-level uncertainties in time series data (Appendix H).
* (Discussion on Applicability and Limitations) We have added more discussion of TTSO framework on the real-world applicability and potential limitations (Appendix G & F).
* (Ablation Experiments) To clarify the efficacy of LLMs in our framework, we conducted additional ablation experiments, the results of which are now included in the revised manuscript (Appendix I).
* (More Baselines) We have expanded our experimental analysis to include further comparisons with more time series OOD generalization methods in the revised manuscript (Section 5). 
* (Explanations to Theorem) We have added more explanations to make theoretical sections more accessible to a broader audience (Appendix A).

We hope these revisions will address the concerns raised and better demonstrate the contributions of our work.
>  (a) Regarding time series, I agree versatility is a strength, but the reason for studying time-series OOD generalization needs to be motivated better other than it is under studied. There is no clear reasoning for why this method is suitable for time series, which have their own sets of challenges and problems from noisy data, rate variation, length variation etc. different from images/text problems.

**Reply to (a)** Thank you for your insightful comments. We agree that time series data presents unique challenges, such as noise, rate variation, and length variation. This type of data is crucial in many real-world applications, including healthcare, sports training, and abnormal behavior detection. Despite its significance, OOD generalization in time series has been relatively under-explored. To fill this gap, we have introduced the TTSO framework, which is **theoretically grounded** and specifically designed to tackle both sample-level and group-level uncertainties inherent in time series data. The novelty of our approach lies in how it models these uncertainties under a **tri-level learning** framework, providing a robust solution for time series OOD generalization. 

Thank you for agreeing that time series OOD generalization problem is under-explored and this constitutes a motivation of our study. Please note that imposing strong assumptions on time series data is impractical and will inevitably compromise the effectiveness of the proposed method since time series data comes in diverse formats and properties, such as periodic versus non-periodic patterns, regular versus irregular sampling, and stationary versus non-stationary behavior etc. Developing a general OOD generalization framework for time series data is already a significant challenge in itself. Finally, as previously stated, our primary research area is time series learning. Concentrating on the time series OOD generalization allows us to delve deeply into this specific problem and conduct thorough analysis and experiments.

While temporal order is an inherent characteristic of all time series data, the presence of other properties—such as seasonality, trends, and irregular sampling—is not universal. These characteristics vary depending on the specific context and nature of the time series. For instance, some time series may exhibit pronounced seasonal patterns or trends, while others may be stationary with no significant autocorrelation. The diverse formats and characteristics of time series data motivate us to develop a versatile and flexible framework (TTSO) that can effectively address these variations. In summary, TTSO is particularly well-suited for time series data due to its ability to accommodate the diverse and variable properties inherent in such data.

> (b) The fact that LLMs are optional cannot be an argument when it is in the title of the paper -- it is indeed a strength that LLMs can be incorporated, but this is not motivated sufficiently enough. The performance does not show a clear benefit, and the new results seem to further make this a bit muddled since we see larger models doing poorer than smaller ones, which is pretty much counter to current intuition on scaling LLMs.

**Reply to (b)** Thank you for your constructive suggestion. Larger models are generally expected to perform better due to their enhanced capacity to capture complex patterns, which aligns with the established scaling laws observed in natural language models. OpenAI's research on scaling laws[1] provides substantial evidence that, within the domain of **natural language**, larger models tend to exhibit better performance when scaled **appropriately** in terms of model size, dataset size, and computational resources. 

However, please note that these scaling laws are not **universally applicable** across all domains. In the context of time series data, the applicability of scaling laws remains an **open question**. The characteristics of time series data differ significantly from those in natural language, and there is **no theoretical guarantee** that scaling laws will always hold in this domain. This uncertainty highlights a key area for future research, as understanding whether and how scaling laws apply to time series data could yield valuable insights.

As we all agree that the TTSO framework is versatile and while it has been thoroughly studied in this paper for time series data, the proposed framework has the potential to be applied to other types of data beyond time series. In summary, our work opens up new avenues for OOD generalization through the introduction of a tri-level optimization framework. We hope that this innovative approach will inspire further research and development in the area of OOD generalization.

Moreover, if the inclusion of LLMs in the title of the paper is a concern, we are open to considering its removal, if allowed. We appreciate your feedback on this matter.
Thank you for your clarifications and additional analysis. 
(a) Regarding time series, I agree versatility is a strength, but the reason for studying time-series OOD generalization needs to be motivated better other than it is under studied. There is no clear reasoning for why this method is suitable for time series, which have their own sets of challenges and problems from noisy data, rate variation, length variation etc. different from images/text problems. 
(b) The fact that LLMs are optional cannot be an argument when it is in the title of the paper -- it is indeed a strength that LLMs can be incorporated, but this is not motivated sufficiently enough. The performance does not show a clear benefit, and the new results seem to further make this a bit muddled since we see larger models doing poorer than smaller ones, which is pretty much counter to current intuition on scaling LLMs. 

Given these issues, I will maintain my score as it is.
Thanks for your response, after reading your response and other reviews, I keep the score unchanged.