Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalization
Authors: Chengtao Jian
Kai Yang
Yang Jiao
Venue: NeurIPS 2024 poster (2024)
Keywords: tri-level optimization
OOD generalization
Primary area: safety_in_machine_learning

Abstract: Out-of-Distribution (OOD) generalization in machine learning is a burgeoning area of study. Its primary goal is to enhance the adaptability and resilience of machine learning models when faced with new, unseen, and potentially adversarial data that significantly diverges from their original training datasets. In this paper, we investigate time series OOD generalization via pre-trained Large Language Models (LLMs). We first propose a novel \textbf{T}ri-level learning framework for \textbf{T}ime \textbf{S}eries \textbf{O}OD generalization, termed TTSO, which considers both sample-level and group-level uncertainties. This formula offers a fresh theoretic perspective for formulating and analyzing OOD generalization problem. In addition, we provide a theoretical analysis to justify this method is well motivated. We then develop a stratified localization algorithm tailored for this tri-level optimization problem, theoretically demonstrating the guaranteed convergence of the proposed algorithm. Our analysis also reveals that the iteration complexity to obtain an $\epsilon$-stationary point is bounded by O($\frac{1}{\epsilon^{2}}$). Extensive experiments on real-world datasets have been conducted to elucidate the effectiveness of the proposed method.

Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalization
Chengtao Jian
Kai Yang"
Tongji University, Shanghai, China jet@tongji. edu. en
Tongji University, Shanghai, China kaiyangerongji. edu. on
Yang Jiao
Tongji University, Shanghai, China yangj vangjiao@tongji. edu. cn
Abstract
Out-of-Distribution (OOD) generalization in machine learning is a burgeoning of study. Its primary goal to enhance the adaptability and resilience of machine learning models when faced with new, unseen, and potentially adversarial data that significantly diverges from their original training datasets. In this paper, we investigate time series OOD generalization via pre-trained Large Language Models (LLMs). We first propose novel Tri-level learning framework for Time Series OOD generalization, termed TTSO, which considers both sample-level and group-level uncertainties. This formula offers fresh theoretic perspective for formulating and analyzing OOD generalization problem. In addition, we provide a theoretical analysis to justify this method is well motivated. We then develop a stratified localization algorithm tailored for this tri-level optimization problem, theoretically demonstrating the guaranteed convergence of the proposed algorithm. Our analysis also reveals that the iteration complexity to obtain an e-stationary point is bounded by O( Extensive experiments on real-world datasets have been conducted to elucidate the effectiveness of the proposed method.
1 Introduction
In machine learning, a common challenge arises when the distributions of training and test sets differ significantly (Quinonero-Canella et al.) 2009]. This mismatch demands that models, trained
specific distribution data, should generalize well unseen distribution data, known OOD
generalization [Shen et al.) 2021 Zhou et al. 2022]. Despite a vast amount of research on the OOD
generalization IZhang al. [2017 (Sagawa al. 2019 Huang et al. 2020 Arroossy al. 2019]. the field of OOD generalization in time series is relatively limited and presents more significant challenges. This is primarily due to the inherent temporal dependencies and dynamic changes characteristic of time series data [Hamilton] 2020). Therefore, a critical aspect of improving time series OOD generalization is learn robust representations that remain stable despite shifts in distributions.
Recently, the field of machine learning has witnessed remarkable advancements in pre-trained foundation models, with notable examples including Large Language Models (LLMs) such as GPT [Radford et al. 2018) LLaMA Touvron et al., [223) and CLIP [Radford et al. 2021 These models have been instrumental in capturing and leveraging complex patterns across various domains. In addition, using foundation models, especially LLMs, in processing non-linguistic data, e.g., time series is increasingly drawing attention. By fine-tuning only a few handful of parameters, these
Cooresponning author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

models show remarkable versatility in diverse data formats ranging from audio [Ghosal et al. [023]
image [Lu et al. 2021] and time series [Chang et al. 2023 Jin et al., 2023 Studies indicate that L.Ms,, as part of the broader foundation model spectrum, demonstrate sophisticated reasoning and strong pattern recognition capabilities [Wang et al. 2023 Chu et al. 2023]. fundamentally acting as pattern machines [Mirchandani et al. 2023]. Moreover, LLMs have been shown to be effective in transfer learning across various modalities, due to their data-independent self-attention mechanism
Zhou et al. 2023
Additionally. recent advancements vision-language foundation models have shown promising developments in OOD generalization [Zheng et al. [022], yet the exploration in time series remains underdeveloped. The potential of using foundational models is highlighted by the study in Liu et al.) 2223a Hendrycks et al. [2020), which suggests that pre-traine trained transformers can improve OOD robustness. Despite existing efforts, the limited exploration of foundational model applications in time series OOD generalization suggests emerging field.
In this paper, we propose Tri-level learning framework for Time Series OOD generalization, named TTSO. Unlike conventional OOD generalization methods that focus solely on group-level Jiao
al. 2022a Huang et al. 2020] or sample-level uncertainties (Zhang al. 2017 Zhou et al. 2021
Han et al. 2024). our framework uniquely addresses both by combing a minimization problem for optimal model parameter learning, a maximization problem for dynamically data re- re-grouping, and another maximization problem for data augmentation under a tri-level framework. To tackle this tri-level problem, we propose stratified localization algorithm via cutting planes. Leveraging the advanced representation learning capabilities of LLMs, we adapt this tri-level learning framework for fine-tuning L.Ms.
Our contributions can be summarized as follows:
Tri-level Learning Framework. contrast to most existing works in OOD generalization, which primarily focus on either group-level or sample-level uncertainties, TTSO uniquely integrates both aspects under a tri-level learning framework. Specially, this comprehensive framework emphasizes the interdependent relationship between problems of each level, advancing beyond the typical single or bi-level methodologies in OOD generalization. Moreover, theoretical framework based on Hlpnik-Chervonenki. dimension has been developed to rigorously analyze and elucidate the generalization properties of TTSO. We then leverage this tri-level framework fine-tune LLMs, achieving an maximum 4.9% improvement in performance on time series classification in OOD scenarios.
Stratified Localization Algorithm. To tackle the aforementioned tri-level optimization problem, we develop a stratified localization method using cutting planes. Unlike traditional gradient-based methods, TTSO removes the necessity of computing the hypergradient for the outer optimization problem. This computation is typically very challenging and computationally intensive due to the nested structure of the tri-level optimization problem. Furthermore, the decomposable nature of cutting planes offers a promising avenue for enabling distributed implementations of TTSO, thereby potentially enhancing scalability and computational efficiency.
Iteration Complexity Analysis. To validate the effectiveness of our method, we conducted a thorough theoretical analysis of the algorithm. We theoretically derive that the iteration complexity of the proposed algorithm for achieving an e-stationary point is bounded by
2
Related Work
In this section, we provide an overview of the foundational concepts and methodologies related to our research, including OOD Generalization and the LLM in time series.
OOD Generalization. OOD Generalization research focuses on improving the model's 's ability to generalize when there is a difference in distribution between the training and test data, and has been widely studied the fields of Computer Vision (CV) RRectt et al. 2019 Salman et al., 2021 and
Natural Language Processing (NLP) [Tu et al.) 2020 Schneider et al. 2020]. Existing works for -uttof-diitribution (OOD) generalization are diverse and can generally be categorized into approaches
that consider sample-level [Zhang al. 2017 Zhou et al. 202 or group-level (Sagawa et al. 2019

---

Huang et al. 2020]
uncertainty. However, the exploration of OOD generalization specially for time series remains relatively underdeveloped. A recent study (Lu et al. 2023] introduced 'Diversify' an innovative approach that models time series data from the perspective of distribution and obtain superior performance. In our work, we consider both sample-level and group-level uncertainties and formulate them as tri-level optimization problem.
The integration of LLMs in time series analysis a rapidly evolving field, drawing significant interest due to their superior pattern recognition and reasoning abilities [Wang
LLM in Time Series.
al. 2023 Chu al. 2023]. recent example Time-LLM (Jin et al. 2023]. which introduces an innovative method by reprogramming time series and incorporating linguistic prompts, effectively activating the extensive capabilities of LLM. In addition, the OFA framework (Zhou al.) [203]] utilizing the frozen pretrained transformer framework, validates the ersatility and effectiveness of pre-trained models time series analysis. Another innovative approach is PromptCast (Xue and Salim, 2023). which employs prompt-based learning method, transforming numerical input and output data into prompts for effective forecasting in zero-shot settings. The TEMPO [Cao et al. 2023)
adapts changes in time series distribution by decomposing time series and adding different prompts for each component and obtain competitive performance in time series forecasting. In specialized
domains like traffic [Xue et al. 2022], finance [Zhang et al. 2023 and healthcare [Liu et al. 2023b),
LLMs have also shown unique advantages. In this work, we aim to enhance OOD robusness for time series by fine-tuning LLMs with TTSO.
3 Problem Formulation and Algorithm
Notations. X' and Y' represent the input and target spaces of samples, respectively. The predictor f.p = h rg consists of the representation function r- with parameter O and the classifier h with parameter w. The function fo X + Y maps time series X E X' to Y E Y, where X E T and Y E R., with T as the time series length and F as the feature dimensions. The multivariate time series X, composed of F' univariate time series each with T observations, is sampled i.i.d. from distribution I' and represented as X = [21, Ø2..... ... Fy). where I; = [T Teg ..... Frl for i = 1,..., F. Assume source domain distributions are Ps,, for i E (1,2..... K} and the target domain distribution is Pr. The source domain data Ds, is sampled i.i.d. from IP Ps,, and the target domain data Df is sampled i.i.d. from Pr.
3.1 Preliminary
Given training dataset Drrain {(x,, Y.) sampled from the distribution Prain Crain((X (X, Y). In supervised learning, the goal is to learn an optimal predictor f.- on D wain such that fa- gener- alizes well test dataset Dust sampled from the distribution (xs(XX Y). In self-supervised contrastive learning, for a given time series X, we generate two augmented views, Xa, and Xaz. using augmentation methods 01 a2 A. These augmentations produce the time series represen- E TR. T ttroughh the representation function rg- The objective of contrastive learning is to minimize the distance between positive pairs (R1, R2) while maximizing the distance between positive and negative pairs. The general formula for contrastive loss, as detailed in Zhao et al. 2002 is formulated as follows
tations Rg = re (X)) E TR T'M and R2 = re
Loon Calign (ro; P. )) + Afreg (ræ; .T).
(I)
The first term aims to minimize the distance between positive pairs in the latent space, while the second term served as a regularizer prevents representation collapse. To evaluate the performance of the model, classifier hn is trained using the representation functionrg.
hw- arg mink.,. [E Eup (he o ro- (X)Y),
(2)
where the representation function rg- (.) is optimized via the supervised loss in Eq. (2). The classification performed using fo- h rgg. However, the discrepancy between the training distribution Preim [X, YY) and the test distribution P (res(XXX .XYY Y) poses a challenge for generalizing fo- to test data. Directly optimizing bxp (X)Y) may lead to overfitting, compromising performance on unseen data. To mitigate this issue, invariant representation learning (Arjovsky et al. 2019 is employed handle distribution shifts by learning robust invariant representations across diverse distributions. To achieve this, we begin with the following assumption.
3

---

Assumption 1 (Invariant Assumption (Zhao et al. 2022)). Considering K different environments (domains) E, there exists a random variable 10 (X such that for any e, e E supp(E), holds that P (Y v (X.)) = P (Y (X
This assumption implies that for time series X observed in different environments, invariant rationales exist and their relationship with the corresponding labels remains stable. This stability ensures that predictions remain consistent across various environments, relying these rationales. Assuming (-) represents the representation function re(-) parameterized by 0, then it follows that
Tg. (X re. (X²) = (x
In contrastive representation learning, where labels not available, the theoretical analysis of the downstream performance is challenging. To address this, research (Zhao et al. 2022] bridges this gap by connects contrastive loss downstream risks,
(4) where is a positive constant, T(a, 5) refers to a set of constants determined by the (o, 6)- augmentation, and C₁ corresponds to the sample set for class k. The first term is optimized during contrastive pre- traininiing..... The second term depends on data augmentations (a, 8). The third term, related the linear layer h optimized in downstream tasks. As shown by Eq. ( contrastive learning on distribution P with augmentation function essentially optimizes the upper bound of the supervised risk.
ora:Px) S e|| ("aggnaarrrr (resP,,r)) II
+I/, (CWlle o
Each environment & corresponds to a domain distribution Ps.. To learn an invariant representation over the domain set P, first provide mathematical definition of invariant risk minimization.
Definition (Invariant Risk Minimization (Arjovsky et al. 2019)). Ifthere exists a classifier hg that is optimal for all domains in P, i.e. ho E argminn, R (h re; Ps.. v.'s E P, then the representation function Tag elicits an invariant predictor hg o Ta across the domain set P.
This definition is equivalent to learning features that have stable association with the target variable, which has been theoretically and empirically proven to improve the transferability of supervised learning across different distributions (Arjovsky et al. 2019 Zhao et al. 2022
3.2 A Tri-level Learning Framework
To address OOD challenges, GroupDRO [Sagawa et al., 2019] propose mimax formulation to minimizes the maximum domain supervised loss to enhance robustness against unseen data. Accord- ing to Eq. ( contrastive learning optimizes the upper bound of supervised risk. Thus, we extend GroupDRO by replacing the supervised loss with self-supervised contrastive loss, aiming to learn invariant representations. We further impose constraints on the group distribution q to mitigate the risk of overfitting to specific domains. This results in a bi-level optimization problem
min 0.9
Σ Uihcon Ds.;
s.t.
9 = arg max K delcon(ro: (ro: d'EAN
(5)
s.t. d(p,)'))))))))) TT
where d (.:)) denotes a distribution distance metric, such as KI. divergence, Wasserstein distance, or Euclidean distance, A probability simplex, and is a constant. Following previous work [Qian et al., 2019), we adopt the Euclidean distance due to its strong convexity, which reults in faster convergence [Rakhlin et al. 2012). The outer optimization seeks the best parameters across all domains to optimize overall performance, while the inner optimization, representing the group-level uncertainty, optimizes the worst-case distribution to enhance representation robustness.
Definition (Augmentation Robust Alignment Loss [Zhao et al. methods a, a' E A. the robust alignment loss is defined follows
2022 For any two augmentation
(ro; TP) : Ex (l)(a,,a)EA [|ro(a(X)) ))- - ro (a' (2)
Theorem (Upper Bound of Risk Gap Between Augmented Domains Shen et al. 2021 ). For any two augmentation methods a, a' A. representation function ro and classifier h have
suPa,a'EA | (h. o rg: Pa) - R (hw o Po; S cellwwllerr (re: P main
(7)

---

* train data (class
train data (class
I)
sample distribution
... classifier
(1) sample-level
(2) group-level
(3) both
Figure I: The depiction of sample-level, group-level, and combined uncertainties.
Fix ra, let ha = arg min R (hw o o Pmin ). we have
[R (ha re; Pa.) - R (ha' or; s 2c (1 ll + h/ []) Par (ra; Irainin)
(8)
Theorem states that minimizing far (ro: P triin) makes the optimal predictor more consistent across different augmentation domains, i.e., minimize Ear (re; I' Puain)) can enhance the invariance of the learned representation. Nonetheless, evaluating far (r; (riii) involves a supremum operator, and the large set A makes accurate computation infeasible. Therefore, we propose an approximation for la (ra; IP m.in). We start with the following reasonable assumption.
Assumption 2. For any pair of augmentation methods a, a' E A. they can be viewed as introducing specific perturbations 5 to the sample X, i.e., a(X) = X 5.. aX) X + 6g'-
Suppose 6, and S are sampled from P perh. representing the distribution of perturbations induced by augmentation techniques. We adopt Gaussian Mixture Model (GMM) (Jiao al.] [022b) to accurately characterize the uncertain perturbation distribution. Thus, the distribution of ò is given by
p π,, μ4, σ) EM NoN (6: Mm )
where Tm represents the weight of the th component in the mixture, and Em=1 Tm 1. The ex- pression for for (ro; Firain) can be written as suP8=p(0;T,,..o E qilatign (0, 6; Ds.). Consequently, we can further extend problem (5) to the following tri-level optimization problem.
M
K E/=1 Poo (0,8;Ds))
min 0.9.5
s.t.
q arg max Σ Lfl (0, 8; Ds.)
D'EAK
s.t. d (p, 4') ST
(10)
8 arg max
Σ K
(0,0''
8' 8'~p(8':,,11000 sp,
s.l. | S C1,1lo|l S C2 MM Tm. 1, mm 2 0,
where C1 and C2 are constants. The third-level optimization addresses sample-level uncertainties by maximizing the alignment loss under the worst-case perturbation distribution. Figure [ Iillustrates these concepts, showing how group-level and sample-Ieve optimizations interact within the tri-level framework. To theoretically justify our approach in Eq. (T0), we present the following theorem.
Theorem 2 (Upper Bound on Target Error). Given the previous setup, let H be hypothe- sis space of Vapnik-Chere Chenkk--hervoneniis (VC) Dimension d and hi = minkeH eT (h). Let Pa (IP P., E; ail's,, E.a, = 1, aq 2 }. If h E H is the empirical minimizer on Pa. then for any ó and Pc E Pa. with probability at least 1 5.
(T( < 3er (hT)) + x dHAH (Pc, Pr) + max,,j dHAH (HAH(Ps,. (Ps., + C(ő, m.d). d),
(11)
where X = ZK ases, (hr) and C(8, m, d') is a statistical term. duau(;;)) -) is a metric function which measures differences in distribution [Ben- Devid et al. 2010 es, (h) and er(h) is the source error and the target error.
Discussion: Theorem provides theoretical framework for estimating performance on a new target distribution. The TTSO framework in Eq. (10) focuses on minimizing the terms dæa# (Pc, PT) and max,,, dæAH (Pss, ). thereby giving tighter bound of target error to improve generalization ability. Proof of Theorem 2 and further discussion of our motivation are in Appendix A.2
5

---

Algorithm SLA: Stratified Localization Algorithm
Input: Training datasets {Ds, learning rates T]e, Tl TJ. number of iterations T'.
Output: Optimized parameters 6
1: Initialize parameters 0, 9, ó and initial set of cutting planes S
2: for = 0 to T 1 do
3: Update variable ((+++1, g(++1) and ((++1) according Eq. 19 20 and 21
4: mod == then
5: if ((( (((++1 5 ((++1) > then
6: Add new cutting planes to set S according to Eq.
7: end if
8: end if
9: end for
10:
return ((2)
However, solving the constrained tri-level optimization is extremely challenging. In the next subsec- tion, we introduce stratified localization algorithm to address this problem effectively.
3.3 Stratified Localization Algorithm
Due the hierarchical structure of the tri-level problem, develop stratified version of the localization method [Boyd and Vandenberghe 2007]]Jia et al. 2023) to tackle the problem presented in Eq. (TO). First, we use exterior penalty method to reformulate the third level problem, the resulting problem
E gilcon (0,) min 0.q.6
s.t.
arg max Li=l Loloon (0,6; Ds,)
Σ
d'EAR d (p, 9')
(12)
ΣK ahaanna (0.8':Ds.) P3:
arg max 8' 8'mpp('' (p).o)))
where P3 a penalty term defined as P3 = pi (ma ||]|| -1)) (2 |||||| - 22))
Pa() Pa(2 Tm - 1)2 (max(00, )x,) and P; are penalty coefficients.
Given that the third-level optimization is a constraint for the second-level optimization, we em- ploy Ta steps of gradient ascent to approximate the third-level problem. This technique is com- monly used in previous bi-level optimization studies [Ji et al. 2021 By defining q',
ΣK (Caligg (0,8'; Ds.) P's and using the exterior penalty method, the resulting optimization
problem can be expressed as
min 0.q.8
Σ qoloon (0.6: Ds.)
K
s.t.
q arg max
5K K
deloam (0,; P2,
(13)
q'EAK
where P2 = X +) Di A2 max -4. +\g|0 - V f3 d', 8')12.
-TT-
Likewise, we perform T₃ steps of gradient ascent to replace the second level optimization problem. aefoom (0, 6: Dtr P2, (0, arg ff can be reformulated as follows
With the definition of 20 5)
and h(0, g, 5) ||q (0 d). Eq.
13
min
Σ gileon (0, 6; Ds,)
0.q.8
(14)
s.t.
h(0.q.8)
Let f1 (0. problem can be relaxed
= >K g: Lalignn (0,5; Ds.). Considering the approximations of and 8, the above
min
f1 (0,
0.q.5
(15)
s.t.
1(00, q, 8) Se,
where e > a constant. Inspired by the polyhedral approximation method [Bürger al. 2013
utilize cutting planes to approximate the feasible region with respect to h(0, 8) S In the

---

(t + 1)th iteration, the set of cutting planes, denoted as S", defined as follows
St {a + b, q c 8 d, S 0, 1..--.1151)..
(16)
where a; E ER.D b, E RM c, E TR. d, E TR! and |S represents the number of cutting planes in S¹. Then Eq. (1) can be expressed the following approximation problem
min
f1 (0,
0.4.8
(17)
s.t.
a 0 b, 9 8 8+ d. 50,
18%
The penalty function with respect to Eq. 17 can be described as
F (0, 8) = S1 (0, q. 6) + Σx X max(0, a. 0 + b, q + 6 d.)
(18)
In (t 1)th iteration, the variables are updated as follows
g O 71 F(O q 00)8), q ++ q' - - V F(O0", q .') 5 g++1 5' -77 V. (0 .5)
(19)
(20)
(21)
Throughout the iteration process, the set of cutting planes S is updated every k iterations for tighter and more accurate polyhedral approximation. Before adding new cutting planes, we first
check whether (0' _ 8 is a solution for Eq. 15). If not feasible solution to Eq. (15), i.., h(0, q, 8) E, cutting planes are added to S based on Theorem 3 and Proposition 22 Algorithm provides details of the proposed method.
Theorem 3. Let T2 1. If first-order Taylor expansion is applied to the function f2(0,q, q, 6) at the point (0, () it follows that the function h (0, q, 8) is convex with respect to (0, q, 8). The detailed proof can be found in Appendix|A.3
Proposition 1. Given the convexity of the function 1( 9, 8), new cutling plane is generated when the condition 1(0,, 9, 8)
> & is not met. This cutling plane is formally expressed as
Voh(O t+ 8411 v g 84+1
θ 6
h( q1+1 1(+1)
+
q q" +1
SE.
(22)
Vsh(0++1 g4+1)
8 gt+1
For the detailed derivation and proof of Proposition 22
please see Appendix A.I
3.4
TTSO for Fine-tuning LLMs
LLMs have garnered considerable attention in time series applications [Jin et al. 2023 Zhou et al. 2023].
The emergent abilities of LLMs, especially in OOD scenarios, largely depends on the robustness of their representations (Wang al. 2023 Chu et al. 2023]. This section connects the established theoretical foundation with the practical application of fine-tuning LLMs for time series OOD generalization. We adapt TTSO framework for tuning LLMs to enhance the performance in time series OOD generalization. Our proposed method involves a dual-stage fine-tuning method tailored for time series. The main process of fine-tuning are described below.
Time Series Pre-processing. Preprocessing starts with an input projection layer to bridge the gap in dimensions between time series data and the LLM's native embedding dimension. This step is crucial for the LLM's effective integration of time series. Following this, positional encoding is applied to preserve the sequential integrity of the time series.
Dual-stage Fine-tuning Method. In the first stage, we employ TTSO framework to fine-tune LLMs, in line with the previously mentioned tri-level optimization framework as illustrate in Eq. (10). We adopt the contrastive loss function designed for time series from Yue et al. [2022]. In the second stage, the learned weights of the LLM, including the projection layer, are transferred to the downstream fine-tuning stage for time series classification. To retain the knowledge learned by the LLM from the corpus, follow Chang et al. (2023]. ). Zhou al. (2023 by fixing the weights of the fully connected and attention layers, using Layer Normalization Tuning [L et al. 2022a to adjust only the layer normalization parameters, making the affine transformation trainable.
Constrained Optimization for Fine-Tuning, Research WWotsmaan et al. 2022 indicates that adopting radical strategies for fine-tuning models, such as larger learning rates, can reduce out-of-
7

---

distribution robustness. Unconstrained optimization of model parameters during fine-tuning can lead to knowledge forgetting issues and decrease the model's s generalization ability, as mentioned in Xuhong et al. [0188) Therefore, during fine-tuning for downstream tasks, we impose constraints on the parameters, following Xuhong al. (2018), resulting in the following optimization problem
min s.t. 1 Poll %
lats (ro o hw; D)
(23)
where Og and 0 respectively denote the weights from the first and second fine-tuning phases of the LLMs. More details of fine-tuning LLMs can be found in appendix D
Convergence Analysis
Assumption 3 (Lipschitz Continuity of Gradient). Assume that the gradient of the function F is L-Lipschitz continuous gradient, i.e., for any a, y, there exists L > 0 such that:
[VF((z) VF S L| yll
(24)
Assumption 4 (Unbiasedness and Variance Bound of Stochastic Gradients). Assume for the stochastic gradients go, 94 98, the following conditions are satisfied
3 (0" q², 8%; (5) VoF(O', q' 8')] 0,
E 19, ( (0' q¹, 8¹; (9) VgF(e', .' ,')8') = 0,
E 195 00 q- 8"; (C) VsF(O', ,') [ =0
(25)
1 ( 0t 8 ( VoF(O', {'
(0'
00' o' 8 {) v,F(0' {')')
E
11 0 8.:8 - V&F q¹ 8 S oj.
where [ denotes the expectation over the Cj-
Assumption 5 (Bounded Gradient). Assume that the gradient of the function F is bounded, i.e., Vt. ( q', )'12) I a², F q', 8')5)12 S a², F 0¹ .q', 5¹)) < a².
Definition (c-Stationary Point). Following Xu et al. (2023), Jiao et al. 22024). point (0, q. 5) considered an e-stationary point (where >> 0) a differentiable function F Ff the sum of squares of its gradients on these variables satisfies V S e. Let T(e) be the index of the first iteration that satisfies ||VG S i.e., T(e)= = minfl ||V S Se,l> L1).
Theorem 4 (Convergence Guarantee). With the continuous addition of cutting planes, the optimal
objective value of the approximated problem, delineated in Eq. (17). is guaranteed to converge monotonically. For further details, see the proof of Theorem 4 in appendix A.4
Theorem 5 (Convergence Rate). Under the assumptions
4 and 5 by setting the step-sizes as Vhi-t, and the batch size B, for given e, it follows that
V/ 1/1 1/6
T(c) 0 (++
[3(m(af a3 + 0z) + of + 03 00)2
(26)
4m²(e F(O": ,ith 8Th) F+
where F* represents the lower bound of F. The proof of Theorem 5 is detailed in appendix A.5
Experiment
To evaluate the proposed TTSO framework, we conduct experiments on 6 real-world time se-
ries datasets using the leave-one-domain-ouu setting. including HHAR [Blunck et al. 2015).
PAMAP [Reiss] 2012]. WESAD (Philip Schmidt al. 2018). SWELL Koldijk al. 2014]
USC-HADIZhan and Sawchuk 2012 and DSADS [Barshan and Aluun 2013 We compare with
baseline method ERM [Vannik] [991] and general OOD generalization methods: IRM (Arjovsky
et al. 2019]. GroupDRO (Sagawa et al. 2019]. ANDMask (Parascandolo et al.) 2020], RSC HHung
ct al. 2020), Mixup ZZhang et al. 2017]. VERx (Krueger et al.) 2021], DIFEX|Lu et al., 2022b). J
And we further compare with 2 recent strong approach in time series: AdaRNN et al. 2021
and GILE (Qin et al., 202T]. We also include DIVERSIFY[Lu et al.) 2023), DFDG et al.
2021], and CCDG Ragab et al. 2022 three methods specifically designed for time series OOD
8

---

generalization. To guarantee fair comparison, we implement all methods using the same backbone architecture (except AdaRNN and GILE), TCN [Bai et al. 2018 a model widely used in time series analysis. In addition, we fine-tune the pre-trained Large Language Model, GPPT[[Radford et al. 2018), within our TTSO framework to harness its sophisticated representation learning capabilities. Detailed information regarding datasets, domain setting, data pre-processing, network architecture
and hyperparameters are provided in Appendix B.1 B.2
B.3 and
5.1 Main Results
We report the average results over 3 runs for each dataset, along with the standard deviation. The results for the HHAR, PAMAP, and WESAD datasets are shown in Tables [ where our method outperforms the second- best baseline by 2.8%. 4.8%, and 4.9% respectively. Additional results are provided in Appendix [E.1 (Table 4 These results demonstrate the superiority and effectiveness of the TTSO framework, as it accounts for both sample-level and group-level uncertainties, which optimizes the upper bound in Theorem 2
Compared to traditional methods like ERM, IRM, and GroupDRO, both TTSO and TTSO" show more consistent and generally superior performance in OOD generalization, highlighting the advantages of a tri-level learning framework. The TTSO" method, which incorporates LLM fine-tuning, consistently outperforms other approaches, demonstrating the effectiveness of L.LM with TTSO fine-tuning in enhancing OOD generalization for time series. Concurrently, the TTSO method, even without LLM fine-tuning, shows strong generalization performance, especially on the HHAR dataset where closely matches TTSO" results. This indicates that the TTSO framework is highly effective in generalizing across different scenarios, even in the absence of LLM.
Table 1: Classification accuracy (%) HHAR, PAMAP, and WESAD datasets. Bold indicates the best, underline the second- best performance. Standard deviation is shown in the lower right corner.
WESAD
Method
ADARNN
73.kux
65 0k
Nue
67.4 67.0k..
71.8cxx
72.Aux
53.8cx
74 Baoa
72 2um
43.4u
47 17.5aa
GILE
65.3ux
61.5c
79.2
58.4c
66:1
70.1cs
74.5uo
45.0uc
66.0ku
64.1
42.2ax
72.7xx
70.5um
46.9us
58.1
ERM
714ux
66.4.
Suse
68.0
72.3cx
58.9
68.7asx
40.5ax
21 21.
65.5u:
48.0um
66.4
72.8
63.9k
79.2
08.1.
21.4c
83.3use
58.5
71.3as
45.3am
71.8amr
67.0um:
45.5
06.5
GRmpDRO
71.
75 Peue
67.5
71.3e.
80 Texn
57.9km
70 Seoo
50.7am:
70 10-2
62.5am
53.7am
66.4
ANDMMask
73.0go
62. m
68.5).m
74.0
11.0Am
55.3uc
72. Sex
44.3az:
70.0ue
61.7am
47.6uu
66.0
RSC
76.3cs
63.9 63.9
A8
70.7
74.6ucc
84 Seo
38.9 38
72. 2e.
72.6
Stc
70 Sue
67.2
57.4um
63.1
68.8
Mixup
65.0 65.0
lese
69. 60
71.9
c3.6u
87 3u.
mn
69 9k..
72.5
52.9um
70 Ouer
73.
64 Ast
65.2
69.9
VRRs
67.5xx 63.2cz
68 as
74.7gu
85. fu.
59.5 Sco
70/0uo
72.0gm
Juz
50 Fus
61.8
68.0
DIFEX
71.5cc
62.0k
$1.5mcc
63.5cm
70.1
13.fuu
$4.
59.2gxx
71.4ex
72.1
48.2azi
71.7as
63.2xc
51.3um
58.6
66.9
DJDG
71.2xx
65.8
74.loc
70
70.3
33.1ax
30 30.5um
59.2csc
70. 20x
70,8
49.8uz:
71.6un
71.1uo
50.7as
60.8
67.3
CCDG
73.0ua
63.2cm
77.3ua
71.5
12.
$4.8ua
30.6ua
72. lec
71.5
58.5
70.5ez
69.8
36.1um
62.2
68.4
DIVERSIFY
73.7ax
64.2c
78.9
71.2
71.8
74.0kz
84.0ue
56.5cm
22.9um
72.0
57.
73.0mm
72.6uz
57.3um
64.6
60.3
TISO
76.5.
67.5
20.2..
68.1cm
13.1
15.b
36. lus
60.5...
72.5..
73.6
59.9
21.3aa
76.2.m
63.0p
67.6
21.4
TISO'
77.6.c
67.3
30.4aa
69.9
73.9
78.5.z
89.6.o
61.4.
75.0..
76.1
5m.5
21.3pm
77.5m
65.0mm
6864
72.8
5.2 Ablation Study
This ablation study is conducted to further understand the impact of our TTSO framework's 's fine-tuning on model performance. We compare four distinct variants: pretrained GPT2 ine-tuned with
76
TTSO
TTS0-1
TTSO
TTS0--
Accuracy
TTSO (TTSO* ++ +t)++++++++++ a pretrained GPT2 without TTSO fine-tuning (TTSO*>) randomly initialized GPT2 fine-tuned with TTSO (TTSO-+) ) and a randomly initialized GPT2 without TTSO fine- tuning (TTSO- ". This comparison helps in quantifying the effec- tiveness of the TTSO fine-tuning strategy in enhancing the model's OOD generalization capabilities.
HHAR
PAMAP
WESAD
Figure 2: Ablation study of TTSO*
The ablation results are presented in Figure From this results, we can see that: (a) TTSO demonstrates the best performance in all scenarios, further validating that the combination of a pre-trained GPT2 model with TTSO fine-tuning can significantly improve the model's s OOD generalization capabilities. (b) Although TTSO +- does not employ TTSO fine-tuning, it still exhibits relatively good performance. This suggests that the pre-trained GPT2 model has an intrinsic capacity for OOD generalization, consistent with previous empirical studies (Zheng et al. 2022 Hendrycks et al. 2020]. (c) Compared to TTSO TTSO applies TTSO to fine-tune randomly initialized GPT2, TTSO+T ++ achieves improved performance.

---

This demonstrates that even in the absence of a pre- pre-trained model, TTSO tine-tuning can effectively enhance the model's OOD generalization capabilities, though not as significantly that with pre-trained GPT2.
Conclusion
Existing OOD generalization methods mainly focus sample-level uncertainties group-level uncertainties, often overlooking the interplay between these two aspects. In light this, we propose the TTSO framework integrate both sample-level and group-level uncertainties within unified tri-level learning approach, thereby enhancing the model's robustness and adaptability in facing diverse and unforeseen distribution shifts. In addition, this innovative framework introduces fresh perspective for the development and analysis of the Sit-of--istributiion (OOD) generalization problem. Based on this formulation, we develop stratified localization algorithm for the tri-level optimization problem and provide theoretical analysis regarding the iteration complexity of the proposed algorithm. Comprehensive studies have been carried out assess the performance of the proposed algorithm and substantiate the theoretic claims. It is seen that TTSO with LLM can considerably improves the performance of time series OOD generalization.
7
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 12371519 and 61771013; in part by Asiainfo Technologies; in part by the Fundamental Research Funds for the Central Universities of China; and part by the Fundamental Research Funds of Shanghai Jiading District.
References
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907. 00089,022,, 2019.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and
recurrent networks for sequence modeling. arXiv preprint arXiv:1803.0127. 2018.
Billur Barshan and Kerem Altun. Daily and Sports Activities. UCI Machine Learning Repository, 2013. DOI:
Shai Ben--aviid John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
Henrik Blunck, Sourav Bhattacharya, Thor Prentow, Mikkel Kjrgaard, and Anind Dey. Heterogeneity activity recognition.
UCI Machine Learning Repository, 2015. DOI:
Stephen Boyd and Lieven Vandenberghe. Localization and cutting-plane methods. From Stanford EE 364b lecture notes, 386, 2007.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Mathias Bürger, Giuseppe Notarstefano, and Frank Allgöwer. A polyhedral approximation framework for convex and robust distributed optimization. IEEE Transactions on Automatic Control, 59(2): 384-395 2013.
Defu Cao, Furong Jia, Sercan Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint 1000422310.00498,, 2023.
Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. IIm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained 1lms. arXiv preprint arXiv:2308.08 2023.
10

---

Zhixuan Chu, Hongyan Hao, Xin Ouyang, Simeng Wang, Yan Wang, Yue Shen, Jinjie Gu, Qing Cui, Longfei Li, Siqiao Xue, al. Leveraging large language models for pre-trained recommender systems. arXiv preprint arXiv:2308. 10837, 2023.
Lucas Deecke, Timothy Hospedales, and Hakan Bilen. Visual representation learning over latent domains. In International Conference on Learning Representations, 2021.
Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. Adarnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM international conference on information & knowledge management, pages 402-411, 2021.
Deepanway Ghosal, Navomil Majumder, Ambuj Mchrish, and Soujanya Poria. Text-to-audio gener- ation using instruction-tuner Ilm and latent diffusion model. arXiv preprint arXiv:330. 2023.
James D Hamilton. Time series analysis. Princeton university press, 2020.
Pengchao Han, Xingyan Shi, and Jianwei Huang. Fedal: Black-box federated knowledge distillation
enabled by adversarial learning. IEEE Journal on Selected Areas in Communications, 2024.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribttion robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2744-2752 2020.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt II 16, pages 124-140. Springer, 2020.
Kaiyi Ji. Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International conference on machine learning, pages
PMLR, 2021.
Yang Jiao, Kai Yang, and Dongjin Song. Distributed distributionally robust optimization with non-convex objectives. Advances in neural information processing systems, 35:7987-7999, 2022a.
Yang Jiao, Kai Yang. Dongjing Song, and Dacheng Tao. Timeautoad: Autonomous anomaly detection with supervipe contrastive loss for multivariate time series. IEEE Transactions on Nenwork Science and Engineering, (133:1604-16) 2022b.
Yang Jiao, Kai Yang, Tiancheng Wu, Dongjin Song, and Chengtao Jian. Asynchronous distributed bilevel optimization. In The Eleventh International Conference Learning Representations, 2023.
Yang Jiao, Kai Yang. Tiancheng Wu, Chengtao Jian, and Jianwei Huang. Provably convergent federated trilevel learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2998-12937, 2024.
Ming Jin, Shiyu Wang, Lintao Ma. Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yux- uan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728. 2023.
Saskia Koldijk, Maya Sappelli, Suzan Verberne, Mark A Neerinex, and Wessel Kraaij. The swell knowledge work dataset for stress and user modeling research. In Proceedings of the 16th international conference on multimodal interaction, pages 291-298, 2014.
David Krueger, Ethan Caballero, Joern-Henrik Jacohsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. -ut-of-distribution generalization via risk extrap- olation (rex). In International Conference on Machine Learning, pages 5815-5826. PMLR, 2021.
Bo Liu, Liming Zhan, Zexin Lu, Yujie Feng, Lei Xue, and Xiao-Ming Wu. How good are large language models at out-of-distribution detection? arXiv preprint arXiv:2308. 10261, 2023a.
Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer- evy, Jacob Sunshine, Jiening Zhan, Ming- Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learners. arXiv preprint 15555:2305.15525, 2023b.

---

Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch, Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.05247, 1, 2021.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 7628-7636, 2022a.
Wang Lu, Jindong Wang, Haoliang Li, Yiqiang Chen, and Xing Xie, Domain-invariant feature exploration for domain generalization. arXiv preprint arXiv:2207. 2022b.
Wang Lu, Jindong Wang, Xinwei Sun, Yiqiang Chen, and Xing Xie. Out-of-distribttion representation learning for time series classification. In International Conference on Learning Representations, 2023.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv preprint arXiv:2307.0472.1 2023.
Giambattista Parascandolo, Alexander Neitz, ANTONIO ORVIETO, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In International Conference on Learning Representations, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
A Philipp Schmidt, R Duerichen Reiss, and Introducing WESAD Kristof Van Laerhoven. A multimodal dataset for wearable stress and affect detection. In Proceedings of the International Conference on Multimodal Interaction, 2018
Hangwei Qian, Sinno Jialin Pan, and Chunyan Miao. Latent independent excitation for generalizable gensor-based cross-person activity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1192--11929, 2021.
Qi Qian, Shenghuo Zhu, Jiasheng Tang, Rong Jin, Baigui Sun, and Hao Li. Robust optimization over multiple domains. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4739-4746, 2019.
Joaquin Quiñonero-Candenn Masashi Sugiyama, Neil D. Lawrence, and Anton Schwaighofer. Dataset shift in machine learning. MIT Press, 2009.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAl Blog, 2018
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
Mohamed Ragab, Zhenghua Chen, Wenyu Zhang, Emadeldeen Eldele, Min Wu, Chee- Khong Kwoh, and Xiaoli Li. Conditional contrastive domain generalization for fault diagnosis. IEEE Transactions on Instrumentation and Measurement, 71:1-12, 2022. doi: 10. 20010991M.22022.155440000
Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In International Conference on Machine Learning, pages 1571-1578, 2012.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5409. PMLR, 2019.
Attila Reiss. PAMAP2 physical activity monitoring. UCI Machine Learning Repository, 2012. DOI: http://Aoooor///024443/////a
12

---

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. International Conference on Learning Representations, 2019.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Sai Vemprala, Aleksander Madry, and Ashish Kapoor. Unadversarial examples: Designing objects for robust vision. Advances in Neural Information Processing Systems, 34:15200-15284 2021.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33:1 15:11539-1151 2020.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out- ouut--f-distibuttinn generalization: A survey, arXiv preprint arXiv:2108, ftti::2018.133335. 13224, 2021.
Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Ng. Domain generalization for text classification with memory- supervised contrastive learning. In Proceedings of the 29th International Conference Computational Linguisties, pages 6916-6926, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621 133, 2020.
Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in Neural Information Processing Systems, 4, 1991.
Yan Wang, Zhixuan Chu, Xin Ouyang, Simeng Wang, Hongyan Hao, Yue Shen, Jinjie Gu, Siqiao Xue, James Y Zhang. Qing Cui, et al. Enhancing recommender systems with large language model reasoning graphs. arXiv preprint arXiv:2308. 10835, 2023.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 2022.
Zi Xu, Huiling Zhang, Yang Xu, and Guanghui Lan. A unified single-loop alternating gradient projec- tion algorithm for nonconvex-concar and convex-nonconcave minimax problems. Mathematical Programmiing, 201
2023.
Hao Xue and Flora D Salim. Promptcast: new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 2023.
Hao Xue, Bhanu Prakash Voutharoja, and Flora D Salim. Leveraging language foundation models for human mobility forecasting. Proceedings of the 30th International Conference on Advances in Geographic Information Systems, pages 1-9, 2022.
LI Xuhong,
Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning, pages 2825-2834. PMLR, 2018
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8980-8998 8987, 2022.
Boyu Zhang, Hongyang Yang, and Xiao-Yan Yang Liu. Instruct-fingpt Financial sentiment analysis by instruction tuning of general-purpose large language models. arXiv preprint arXiv:2306. 2655, 2023.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412. 2017.
13

---

Mi Zhang and Alexander A. Sawchuk. Usc-had: A daily activity dataset for ubiquitous activity recognition using wearable sensors. ACM International Conference on Ubiquitous Comput- ing (Ubicomp) Workshop on Situation, Activity and Goal Awareness (SAGAware), Pittsburgh, Pennsylvania, USA, September 2012.
Wenyu Zhang, Mohamed Ragab, and Ramon Sagarna. Robust Comain-ffree domain generalization with class-aware alignment. In ICASSP 2021 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 2870-2874, 2021. doi: 10.1109/ 2021.9413872.
Xuyang Zhao, Tianqi Du, Yisen Wang, Jun Yao, and Weiran Huang. Arcl: Enhancing contrastive learning with augmentation-robust representations. In International Conference on Learning Representations, 2022.
Zangwei Zheng, Xiangyu Yue, Kai Wang, and Yang You. Prompt vision transformer for domain generalization. arXiv preprint (rX1::2208.08914 2022.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained Im. arXiv preprint arXiv:2302. 1939, 2023.
14

---

Appendix
A Theoretical Proofs and Discussion
A.1 Proof of Proposition 22
Since h(0, q, 6) is convex function, have that
V Veh(O 4 )z+1)
0 0
h(e, q. 8) 2 h(0t+1 q-*1 1
+
v V 4 rz1
q q ++1
(27)
V 0 ++
5 8
According to theorem
combine with Eq.
15 and Eq. the new cutting plane will be generated
as
Voh(O'+1 q0+1 .
lc+1
h(0't1 -9 +1 1
voh(0'+1 g 8'+11
4+1
S.
(28)
Voh(O'+1 t1 80+11
8
From the inequalities, we can derive the coefficients a;, bi, G and d; as follows
Voh(Ott) g't1 1 0 )
b, Vih(ttti q glt1 8tt2),
C = (o glt1 84t1).
(29)
d, = M@t+1 qi+l 1
Voh(0' g 1 )
-V,h q r+ 18tt1
Vsh(O 1 1 TT - -E,
which concludes the proof.
A.2 Proof of Theorem
A.2.1
Background
For sample distribution Ps on inputs space A' with binary labeling function and hypothesis h, the error (or risk) defined as follows
Es(h, f) ExwPs [[)((xx=))))( - f())]]
(30)
For simplicity, we use the shorthand (h) es (h, fs) for source risk and er(h) (h, fD) for target risk, where fs and fo represent the labeling function of source and target domain, respectively,
To bound the target error, following David et al. (2010). we define the H-divergence. Given source distribution Ps and target distribution Px over input space X, let H. be a hypothesis class on X. The H-divergence between Ps and PPr is
dæ (Pss, IP)) sup
- Pre, Pre)]
(31)
where J, {xx E X M(x) = 1, H). In addition, for a hypothesis space H. the symmetric difference hypothesis space dHAR is the set of hypotheses. And the dHaR in [Ben-David al. 2010 is defined as
dHAH PP) 2 sup
PIewPs [h(x) + h((x)] Pr [h(x) * h'(x)]l- h.
(32)
al. 2010 Let H be hypothesis space of VC dimension d. For each E {1...., K,, let Ds. be a labeled sample of size 3.m drawn from Ps. and labeled according to function fs.- h E H is the empirical minimizer of & (h) for fixed weight vector a on these samples and hi minkeR er (h) the target error minimizer, then for any 8 E (0,1), with probability at least -
Theorem 6. (Modified from Theorem 4 in BBe--Davil
ch( < eT (h+) I 0r (2)r + duan (Ps., Pr)) + +2
(EN, af
(d -log() 2m
)
(33)
where >, mingcH (h) + es. (h))
15

---

A.2.2 Proof
According to the definition of dHAH in Eq. 32 we have
dHAH (Ps..PT)
sup h.WE
PræNPs,
(h(x)) + 1 [()) PReNPy [h(x) + (()]]
sup (ProwPs, [h(x)) # N (x)] - ProwFo [ + N (a)))
A.N'EH
+ (PrawBe (m) # 11' (a)) Pr (1() # h (a) )
(34)
sup PrENPS [h(x) + h' (=)] PTa-Po (h(x) + h (=))]
A.NEH
sup PPra~Po [h(x) # (a)] - Pr [b(x) + (x))
h.
dHAH (Iss Pc) + dHHH (Pc, PT)
Since Pa = [Pa Pa = E, a.Ps,, Σ, a: 1, a; 2 0 Vij and Pc E Pas we can obtain that
dHAH (Ps.PP)) = dHAH Pss. Σ, ayl's,
dHAH Σ ayPPs Σ, ag
Σ, aydusH (Ps.,
(35)
dHAH (Ps.: Ps.)) max dUAH Ps;)
Combine with Eq. 34) and 35| we have
dHaH (P's, Pr) I max HHA (Ps,, dRan (Po,Pr)
(36)
(
Substitute Eq. 36 into Eq.
333
we obtain that
(r)) S cr (hT)
+
aj
2A: + max dHaH (ss.,s,) + duau (Pc,Pr)
1.
(37)
(
dl [dl(((2m) - log(0)
+2
i=1 BB
2m
Since A minneH (Er( es, (h)) (Th))) (1-)= + es, (h*), by setting À 2 ages, (h*) and a? Bi
C(o,, m, d) = 22 22/222 (EM
d [d [ 1o)(1) yields the proof.
2m
A.2.3 Discussion
Theorem provides theoretical framework for estimating performance on a new target distribution. The first term, (T(ht), represents the target error under the ideal hypothesis hy. The second term, A 2= N (Gees;()) (h), aggregates the combined error over all source distributions weighted by as and can be minimized via supervised loss with labels. The third term, dHAH (Pcc PT), measures the distributional discrepancy between composite source distribution and the target distribution. The fourth term, max,,; dHaR (P's, .P,,, quantifies the maximum discrepancy between any two source distributions. The last term, C'ô. m, d), is statistical term which depends the confidence level 8, sample size m., and VC dimension d.
The tri-level Icarning framework proposed in Eq. 10p aims to minimize the third term, dRaH FT). and the fourth term, max,,, dxAH (P.s, s. These two terms correspond to the group-level and sample-level uncertainties, respectively. Below, we discuss how these two terms align with the motivation for our tri-level optimization.
For the term dHAN (Pc. the goal of time series OOD generalization is to learn model that generalizes well to unseen domain distributions, which makes direct optimization infeasible due to the unavailability of target dataset. To minimize this discrepancy, we can only enlarge the set Pc. Specifically, we manipulate 5-perturbations applied to individual samples in the third level of tri-level learning framework. By optimizing these perturbations, explore broader range
16

---

of variations within each source domain, which potentially minimizes the term duan (Pc, Pr). enhancing the robustness of learned representations.
For the term maxi,j dHAH(P (Ps,, s,, Psy), the second-level optimization in our tri-level framework adjusts the weights oę that define the mixture of source distributions Pc.. By dynamically modifying these weights based on the 'worst-case' distribution, we minimize the term maxi,j d Ps,). ).
Our approach not only enhances representation invariance across diverse domains but also improves the model's resilience against variations within individual samples. The effectiveness of this tri-level framework is rooted in the interdependence between the problems at each level; adjustments in one level influence the conditions and outcomes the others. This demonstrates the necessity of our tri-level learning optimization, as it requires coordinated strategy that simultaneously considers sample-level, group-level, and parameter-level dynamics.
A.3 Proof of Theorem
First, the first-order Taylor expansion of f2(0, q, d') at the point (0, 6) is obtained as follows
[2 q', 8) = /2(0, q': .5) + Vo.2(0.0) 5)" (0 0) + Vol2(ō, o') 5) Since T₂ 1, therefore, have:
8).
(38)
5) - V f2 ())
(39)
Combine with Eq.
38 and 39
we can obtain that
(0 = 16 V. d' (2(0, + Vol2 0) 06 -5)
% 1/4 (2
8 + V 0. -V, 8)0 V./Vo/2(O.g'' 8)8
(40)
-1g V. Vof2 8)0 V V.fa 0 8)
where C' d/ Mg V f2(00 q' 8)+ v0 d' 0 Vsl2(0, o' 8)8 is affine function. Therefore, (0. 6) is a convex function. According to preserve convexity [oyd and Vandenberghe 2004]. h(0, q. 5) llq 600 5| is convexity, which concludes the proof of theorem
A.4 Proof of Theorem
Assume that in the th iteration, a new cutting plane is added, and the selected point (0'+1 q1+1, always lies within the region Rý formed by the cutting plane set Ct, we have
R$ 2 Ri 2 ... ......... RR-
(41)
Let H denote the feasible region of problem in Eq. 17 at the 8th iteration, and R represent the feasible region of problem Eq. [5) then it follows that
7t
2 71 2 H.
(42)
Let F(0', ((''''''000
5*) denote the optimal value of problem in Eq. 17 at the thh iteration, and fi represent the optimal value of the problem Eq. (15). Based on equation (42), we can obtain
F (0" gd, 8d)) < F (0', q¹, 5*) S.
(43)
It's seen that the sequence ( {F(0", ,') is monotonically increasing, As Ti + 0c, fi monotoni- cally converges to a certain fixed value. is worth mentioning that ( q. 5) = -(0, 8)|l is a convex function. Since the sublevel set of convex function is convex, the feasible region of problem in Eq. (17) is convexity. This implies that the iterative procedure, by continuously adding a cutting plane, is progressively converging the optimal value fi of the problem as referenced in Eq. 15
A.5 Proof of Theorem S
To begin with, we introduce a fundamental lemma that is pivotal for the subsequent analysis.
17

---

Lemma 1. For Σÿ= 90 (0', 0².8':<3) in -== 96 ,0.''; in - (0' 8 89: they are unbiased and bounded, that is,
شه (0' q' 8 :g)] = Voli (0', q'. 8".
E {c
- Σ Do (0', q' 8:; ()) / S S + =
(44)
E
{<5}
(ct}
2ー= 9y (0' ,','''''
m
8'; [1:] v (0', g" 000
| س Dn (0', q', 8 (5) 11] S 5 + o
(45)
E [c:}
[(}
m プド= go (0', q', 2''00% (9:) V85i (0,, ,0'
||
E 98 (0', o', 86 (/) 1|" S 0g c
(46)
E [c:
Here, [st]]]] denotes the expectation with respect to set of variables {ci, ... Sm
A.5.1 Proof of Lemmal i
Taking the variable @ as example, according to Assumption 4 for all i 1, n, we have
E (c) Σラ= g0 (0', d', 5"; [C)] 1 プラ= Bez [go (0', g. 8'; ct)] voF (0' 0'' 8")
(47)
Based on Assumption
5 the variance of FOF q' 5¹) is bounded, from which we can deduce
1 fー go (O'.q', 8"; [
[s3}
1 ETT (0', g' 8'; (9) - V (0', g', 00,,,)/]] + Vo (0', of', 8')1
=
-E(sh
(48)
Eys Be, [1000 (0 (:.. ()) +
oi af
The proofs of Eq. 45 and Eq. 46 follow similar logic. Thus, we complete the proof of Lemma
Combine with lemma now proceed derive Theorem 5 Under Assumption 3 that the gradient of F is Lipschitz continuous, for t t,, it follows that
F (0'+1 gtt1_ 8
< F (6 q", 5²) + (V (0', -')5') 00+1 -00) + (V&F (0', ""'.q',0 ,²)5') gg+t1 q¹)
+ ( (0', q', 8¹) 8e+1 5²) + 2 (1|0 0'l + g q² + 6 5'1(2)
L
(0' q', 80) T10 VoF (0' q' 8¹) 1 Σx 90 (0', d. ;;;)))
m
Th v 0', d. 8¹)
Σm 91 0 d. :::
F&F (0' q'
m
96 @0' q" 8" 5%:5)
176
+ L(na)"
m
1=1 9o (0', 8':))
2
I m
2 94 (0', (0','''':'')
+ 2
2(
Σー 96 (0'
(49)
18

---

Taking the expectation of both sides of Equation
49) with respect to {ci. ... Sm b we can obtain
B [s: [F (0'+1
.9" _ ,0(+1)]
F (0', q' 8¹) -o0l (5))
(oo (0', .q',
1
Σ j=1 90 (0', q 8 c))
m
-, (c)
4 (0', q', 8')
m Σ j=1
1
9q (0' q. (c)
(0((())
VoF (0', q', 8¹)
1
98 (0' q' :c:c))
m
L (1o)
[(()
1
m
90 (0' q. (t)
2
m
L (nn))
18 (c))
1
m
86 (2)
2
L (ms)
1
2
+ 2
IE (c5)}
98 00' q. 8':() j=1
772
(1)
F (0', q', 8*)
V F (0', q' 55)
|1
170 V F (Ø', ..' 5'1) f1g - |V F (0',
-
L (no)
o
(mg)
)
L
(17) 1
o3 + 0
2 ai
+
02
+
2
m
2
m
(50)
The inequality (i) is based lemmal 1 Taking the total expectation of both sides of Eq.
500
we have
E
[F (0 q 42 00+1
s E F
0 q', 8')] 7o ||V (0', ."..g'811
B
V of (0r,
q" "'' ["=]
(na)² L
0²)
L (na)²
E ||V Ff (0' o
a
(51)
+
0 + m
2
m
L ( 2
+ 2
m
where El-) denotes the expectation over all terms. Summing Eq. obtain
51 from ( t= 11 to / = Ti 1, we
Th "." (")1 )
E
F
71-1 110
<
E
F (0 _0"88']]
E
Vo F (0', 2- 8''1
110
Σ
ΣT-1
Ti-1 1=
E || 1
( 1811
Ti E
V F (0', .', 11]]
T1
L (no)²
of
16
aj
t=t1
2
m
21 L (1a))
02 + oe
+ -m
Ti-1 L (n
0g
+
t=t
m
m
=R
[F (0"r
,²,,8')]
Σ
T T1-
E [|VoF 1"1""
- 11g
T- E || |
F (0'
2 6'))
1=T
Ti-1 IE
F/F F (0', (0'.4'= 8*) 111 + Σ
Ti
(9 + 22
1222))
1/
(=)1
1=y1
L
of n 0 + 0303)
2m
(52)
Let 170 = f7y = 178 -
Combining with Eq. 52 we have that
Ti -1
(=t, E
V F p".q'. 8")
1
+ +|
(0' '''..''''' 8')
V (0'.g''00) 8*) [²]
< F (T g',87'
F* +
-
L
(Ti
- -1)
Σ T:-1
Σ 1=1
a?
(53)
2
1=
+ 2m
L
[Ti - t1) 1/2
Ti-1 1
3
0,2
t=
i=1
19

---

Combining the definition of e-stationary point described in Definition
with Eq. 53 we have
Σ Ti-1
Σn t=11
|| (0', ,' | (0', q' 8¹) V (0', q", 8' 1'
E
S F 0th ,qt1 80) li + (Ti(e) 41- Σ 21-
2 Σ of
(54)
L (Ti(e) 11)- Σ -
2m
6,
that is,
(i(() ~ 4 + (3 + +0 +03)
(55)
(c F(g"i , F2)²
5t1)
According to Eq. (55)
obtain
Ti(e) 2
(56)
Hence, it can be concluded that there exists a (e()) such that || =E [ (0', g' 8²)]|²
I (0', o' I| &&F (0", .0) [2]
which concludes the proof of Theorem
B Datasets
B.1 Datasets Information
HHAR[Blunck et al., 2015] collected activity data from 9 subjects engaging in activities, using smartphones that capture 3D) accelerometer data from various positions. PAMAP[Reiss, 2012 encompasses physical activities data from 9 subjects, recorded using wearable sensors monitoring physiological and movement metrics. WESAD (Philip Schmidt al. 2018 focuses on stress and affective state detection from 15 subjects, employing wearable sensors for ECG, EMG, respiration, and temperature under varied conditions. SWELL Kolddjk et al. 2014] recorded stress responses work environment from 25 participants using ECG, EDA, and heart rate sensors during typical office tasks under stressors. USC-HAD Zhang and Sawchuk 2012] comprises detailed motion and orientation data from 14 subjects performing various activities, captured via a MotionNode device with high sampling rate. DSADS DSADS[Barshan Barshan and Altun 2013] consists of 19 activities recorded from 8 subjects at Bilkent University, using body-worn sensors on torso and limbs, with data segmented for detailed analysis. Table 2 shows the information of the datasets used in experiments.
Table 2: Dataset information.
Dataset
Classes
Dimension
subjects
samples
HHAR
4
73420
PAMAP
13
40
9
31984
WESAD
4
63180
SWELL
2
16
52944
USC-HAD
4
40244
DSADS
19
45
4
17520
B.2 Domain Setting
The domain setting is summarized in Table This setting done balance the number of samples and classed across different domains.
B.3 Data pre-procssingg
For all datasets, we configure the window size as 128 and the step size as 64, resulting in a 50% overlap between two adjacent time series samples. Each sample is standardized using the formula i ش= where and represent the mean and standard deviation of the dataset, respectively. It's
20

---

Table Domain setting for HHAR, PAMAP and WESAD dataset.
Dataset
Domain A
Domain
Domain
Domain
HHAR
2
3
PAMAP
2
3
4
WESAD
1,2
3,5
4,6,9
7,8
SWELL
1,2,4,5
6,7.9.10
12,13.14.16
17,18,21,24
USC-HAD
2,3,4
4.5.6
1.7,9.10
11,12.13,14
DSADS 1,2
3,4
5,6
7,8
important to note that the and used here is specific to each domain, rather than the entire dataset. This approach intended to maximize the distinction between data from different domains.
C Network Architecture and Hyperparameters
Our baseline experiments were conducted using network architecture consisting of 10-layers dilated convolutions network. The dilation rate for each layer is set to 2*, where A: is the layer number. We used the same kernel size of 3 across all layers. Optimization was performed using the Adam optimizer with weight decay of 3 x 10-4 For all baseline experiments, we set the batch size to 256 and the learning rate 0.002. The training was set run for a maximum of 50 epochs. All the methods are implemented with PyTorch([Passe et al. 2019 version 1.7.1 an NVIDIA GeForce RTX 4090 graphics card.
In the TTSO fine-tuning experiments, we employed GPT-2 as the language model. Fine-tuning was performed in two stage: In the first stage, the learning rate for the large model the input embedding, it was set at 0.001. In the second phase, we adjusted the learning rate for the large model to 5 10 aiming to further refine the model's performance. During the evaluation phase, we froze the parameters of the language model, only fine-tuning the classifier with learning rate of 0.003 adapt the specific classification tasks. The batch size was consistently set at for all experimental stages.
x 10-4, and for
D Details of Fine-tuning
The structure of LLM Fine-tuning with TTSO is illustrated in figure 3a and 3b components involved are follows:
The primary
Supervised Loss
And & Nor
Contrastive
Classifier
Add & Nome
Foed Farvad
Feed Forword
And & Nome
Language Model
Language Model
Adll & Namn
Multsliead Atseticn
Vartiiliaa Anenisee
Input Embodding
Input Embodding
(a) First Stage: Alignment Fine-tuning
(b) Sccond Stage: Downstream Fine-tuning
Figure 3: Structure of LLM Fine-tuning with TTSO, illustrating the two-phase approach starting with alignment fine-tuning followed by downstream fine-tuning, adapted specifically for time series out-of-distribution generalization tasks.
Input Embedding: The first is the input embedding, where raw time series is transformed into embedding space that is amenable for processing by the language model. In experiments, the input embedding is a linear layer. As shown in Figure 3 the time series (indicated by the dashed box below the waveform) is combined with positional encoding to form the input representation.
Language Model: This is the core part of the model, typically comprising multiple pretrained transformer encoder. This model is used for processing the input embeddings and producing advanced
21

---

feature representations for subsequent classification tasks. Note that, to retain the intrinsic information of the language model, only the parameters of layer normalization can be tuned.
Classifier: In the second stage of fine-tuning, the classifier tailors the language model to the specific time series classification task. It takes the advanced feature representations from the language model and fine-tunes the model for downstream tasks. We use a linear layer for the classifier, with cross entropy as the supervised loss.
Contrastive Loss: A contrastive loss function employed to enhance the discriminative of the representations in the first stage of the fine-tuning process. This loss function aims to ensure that the representations of similar time series samples are brought closer together the representations space, while representations of dissimilar samples are pushed apart. Specifically, during this stage, the contrastive loss acts as a guiding signal for the language model, encouraging it to learn representations that effectively capture the underlying patterns and distinctions within the time series data, thereby adapting the language model to time series data more effectively.
E
More Experiments
E.1 Additional Datasets
We conducte on more datasets to demonstrate the superior performance of our framework. The results summarized in Table
Table 4: Classification accuracy(%) SWELL, USC-HAD and DSADS datasets.
SWELL
USC-HAD
DSADS
ALL
Method
D
AVG
B
C
AVG
A
AVG
AVG
ADARNN
66.2
57.5
50.6
58.2
59.5
60.7
60.0
58.6
59.7
88.1
77.5
82.9
85.0
67.7
GILE
56.7
58.1
54.9
62.3
58.0
61.0
64.3
66.2
57.2
62.2
84.7
76.3
82.6
78.2
80.5
66.9
ERM
55.6
59.6
53.0
61.1
57.3
60.7
62.9
61.1
59.3
61.8
36.7
81.9
87.3
81.7
84.1
67.9
60.3
89.9
67.5
GrapDRO
61.9
60.8
51.5
59.1
58.3
62.6
61.1
67.6
58.3
63.0)
92.0
81.6
90.0
82.6
86.6
69)
ANDMask
$8.1
57.9
$2.0
63.0
57,8
$7.8
58.4
66.3
$7.6
60.0
89.7
79.0
$9.9
82.5
85.3
67.7
RSC
Mixup
57.5
61.5
$7.2
52.6
57.2
64.9
61.4
67.9
56.4
62.9
93.9
82.3
91.5
84.7
85.
69.4
VERS
$7.3
58.2
$1.7
57.5
56.2
58.6
$7.3
66.3
59.1
60.4
89.8
79.0
95.4
87.0
87.9
68.
DIFEX
55.7
61.0
61.6
57.2
58.9
59.8
56.5
67.2
56.7
60.1
37.8
82.8
83.2
86.2
68.4
DIVERSIFY
59.2
63.4
61.7
62.2
60.3
89.0
TTSO
65.2
65.7
66.5
59.1
64.6
63.1
61.2
68.0
66.0
64.6
93.0
88.9
91.7
87.8
90.4
23.2
TTSO"
60.5
68.7
72.4
63,5
66.3
62.1
63,5
65,2
63.0
63.3
92.0
89.9
92.2
89.5
90.9
73.4
E.2 Illustration of Sample-level and Group-level Uncertainties
1.60
1.55
MAC www.AVAAMMM
1.50
Features
VANNAN WWW
1.40
1.30
1.20
Time Steps
Figure 4: Sample-Level Uncertainty: Each line represents a window of time series data with the same label.
To illustrate the concepts of sample-level and group-level uncertainties, accelerometer data collected by the "samsungold 1' device from four users in the HHAR dataset.
the x-axis values from
In Figure 4 sample-level uncertainty is shown by plotting time series data from specific label (e.g.. "walking" ), where each line represents a different time window. The variations among these lines illustrate the inherent noise, which represents sample-level uncertainty.
22

---

10
Groups
A
B
Density
C
D
X-axis values. Figure 5: Group-Level Uncertainty: Histogram of 'x' axis values, with each color representing different group.
Figure 5 demonstrates the group-level uncertainty by displaying the distribution of x-axis values from the accelerometer across different groups (users). Each color represents distinct group, and each group's unique characteristics contribute the overall group-level uncertainty.
E.3 Ablation Study on LLM Architectures and Parameter Configurations
To further investigate the impact of various architectures and parameter settings of LLMs, we conducted additional ablation experiments that focused on different LLM architectures and parameter sizes (e.g., base model and large model). These experiments included encoder-only models (e.g.. BERT), decoder-only models (e.g., GPT-2), and which configurations yield the greatest benefits during fine-tuning. The results are summarized in the 5
models (e.g., BART) determine
Table 5: Performance comparison of different LLM architectures and parameter sizes across datasets.
Architecture
Version
HHAR
PAMAP
WESAD
AVG
Base
64.3
66.9
64.4
64.2
Encoder-Only (BERT)
Large
61.7
52.5
62.3
58.8
Basc
72.9
76.1
68.4
72.5
Decoder-Only (GPT)
Large
64.5
69.4
66.5
66.8
Base
57.3
65.4
64.2
62.3
Encoder-Decoder (BART)
Large
55.5
61.2
61.4
59,4
The results indicate that decoder-only architectures, specifically the GPT-2 base model in this experiments, achieve the best performance. However, increasing the number of parameters in all three architectures leads a significant drop in performance across architectures for time series OOD generalization.
75.0
72.5
Average Accuracy
67.5
65.0
62.5
10
12
Figure 6: The effect of varying the number of Transformer layers (k) average accuracy for OOD generalization across HHAR, PAMAP, and WESAD datasets.
To further explore how the number of parameters affects performance, we conducted experiments using GPT-2 models with varying numbers of Transformer layers on 3 datasets to evaluate their OOD
23

---

generalization performance. For these experiments, we utilized 20% of each dataset. As shown in Figure (in the attached PDF), the results demonstrate that optimal OOD generalization performance is achieved with a configuration of Transformer layers.
Based this findings, incorporate this optimal layer configuration with TTSO framework, yielding improved results as detailed in the table 6
Table 6: Performance improvements on different domains for HHAR, PAMAP, and WESAD datasets.
Target
A
B
C
D
AVG
HHAR
+1.4
+0.1
+0.8
+0.9
+0.80
PAMAP
+1.0
-1.4
+0.8
-0.3
+0.03
WESAD
+5.3
-3,5
+2.3
-0.1
+1.35
Limitation
TTSO is general framework for learning invariant representations across diverse domain distri- butions, currently discussed only for time series classification. This framework could be further enhanced by extending it to more time series OOD tasks, such as time series forecasting and anomaly detection. Additionally, distribution shifts occur not only in time series but also other machine learning domains like images [Deecke al. 2021 and text [Tn al. 2022 Applying our approach to these domains could further improve performance.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we clearly state the main contributions of our work. The tri-level learning framework, stratified localization algorithm and iteration complexity analysis is in Section 3.2 3.3 and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
The limitations of this work can be found in Appendix F
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
24

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: this paper, all theoretical results are accompanied by a full set of assumptions and complete proofs. These are clearly stated and numbered within the main text and are cross-referenced appropriately. Detailed proofs for major theorems are provided in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Thcorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our paper provides detailed descriptions of the experimental setup. Detailed information regarding datasets, domain setting, data pre-processing, network architecture and hyperparameters can be found in Appendix B.T B.2 B.3 and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
25

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While the data used in our study is publicly available, we are currently unable provide open access the code. However, have included detailed descriptions of the data access, preprocessing steps, model architecture, and experimental setup in the paper and Appendix. These details should be sufficient for others to reproduce the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ cttilss.cc/uubbccguuiic
CoodeSumeeeeeeeeeeeeee ssi aasionoPlicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
26

---

6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Our paper provides detailed descriptions of the experimental details. Detailed information regarding datasets, domain setting, data pre-procsssnng network architecture and hyperparameters can found in Appendix B.1 B.2 B.3 and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the mean accuracy and standard deviation across three runs with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the type compute workers (GPU) in the Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
27

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://eeurij: wwtpcccccpppctteeeceecccc http:////uuuuapcccacoaaaccca/aar
Answer: [Yes]
Justification: Our research adheres to all guidelines outlined in the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
28

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This paper does existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing experiments or research with human subjects.
29

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "QILZ3blxJj", "reviewer": "Reviewer_Z4zT", "summary": "Out-of-Distribution（OOD）generalization in ML emphasizes on improving model adaptability and robustness aginst unseen and potentially adversarial data. This paper explores OOD generalization for time series data with pre-trained Large Language Models and proposes a novel tri-level learning framework to handle the data distribution uncertainties. It goal is to address not only sample-level but also group-level uncertainties in the new dataset. The paper offers a theoretical analysis to justify the method and develops a cutting plane strategy for the tri-level optimization problem. It demonstrates guaranteed convergence. Extensive experiments on real-world datasets confirm the effectiveness and efficiency of the proposed method.", "questions": "Can the TSSO framework be employed to manage data uncertainties in other type of data, for example natural language data?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1) OOD generation is an important problem in ensuring the robustness and reliability of machine learning models. It becomes increasingly important in AI as machine learning models and systems are expected to be deployed in numerous real-world applications in the near future.\n2)The tri-level learning framework is grounded in robust theoretical principles and offers a fresh perspective for modeling and studying the OOD problem. This framework opens new avenues for investigating OOD challenges in time series data.\n3)The TTSO algorithm is proven to converge, with the authors also determining its convergence speed.\n4)Extensive experimental studies have been carried out to validate the effectiveness of the proposed methods.", "weaknesses": "While the paper is well organized overall, the clarity of this paper can be further enhanced via providing examples to illustrate the concepts of sample-level and group-level uncertainties in time series.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(W1)** While the paper is well organized overall, the clarity of this paper can be further enhanced via providing examples to illustrate the concepts of sample-level and group-level uncertainties in time series.\n\n**(Reply to W1)** Thank you for your insightful comments. Per your suggestion, we have provided an example using the HHAR dataset to illustrate the concepts of sample-level and group-level uncertainties in the attached PDF (global rebuttal). Specifically, we use the x-axis values from accelerometer data collected by the ‘samsungold_1’ device from four users.\n\nIn Figure 1, sample-level uncertainty is shown by plotting time series data from a specific label (e.g., 'walking'), where each line represents a different time window. The variations among these lines illustrate the inherent noise, which represents sample-level uncertainty.\n\nFigure 2 demonstrates the group-level uncertainty by displaying the distribution of x-axis values from the accelerometer across different groups (users). Each color represents a distinct group, and each group's unique characteristics contribute to the overall group-level uncertainty. \n\n**(Q1)**  Can the TTSO framework be employed to manage data uncertainties in other type of data, for example natural language data?\n\n**(Reply to Q1)** Yes, the TTSO framework we proposed is versatile and applicable to various data modalities. However, our focus in this work was specifically on time series data. Focusing on time series data allows us to provide a comprehensive analysis that might not be achievable if we were to cover multiple data types simultaneously.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Imm0vmAUKv", "reviewer": "Reviewer_BvSM", "summary": "The paper studies the problem of OOD generalization in time series tasks, building on recent observations that use data level uncertainties and group level uncertainties. This has been a successful way to build robust, transferrable representation. The paper additionally also includes an additional maximization for data augmentation that makes it the \"tri-level\" learning framework. The paper also theoretically analyzes generalization properties of the proposed algorithm. \n\nThe TTSO method is also used to fine-tune LLMs which is used for time series classification in OOD scenarios. In this part, it builds on some recent works that utilize pre-trained or fine-tuned LLMs in other domains, leveraging the superior feature learning capacities of these large frontier/foundation models.", "questions": "Please see above", "limitations": "Yes, it appears so.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The paper addresses an important problem to build robust techniques for time series data, which is under studied and often quite challenging due to the inherent noise in time varying datasets.\n* Using an LLM within a framework for generalization in time series appears to be a novel framing\n* The paper does extensive theoretical and empirical analysis to demonstrate the properties and performance of TTSO.", "weaknesses": "On the surface of it, the paper shows a good improvement over several commonly used methods in OOD generalization, improving on different benchmarks. However, many aspects of the paper are unclear:  \n* Two of the main aspects of the paper are not sufficiently well motivated in my opinion -- \n\t* (a) _OOD generalization in time series_: Why is this particular method effective for time series? as far as the assumptions made in the paper go, this is a general technique that can be applied to any arbitrary dataset/modality. While it is true that time series problems do not receive as much attention, that alone is not motivation enough. What assumptions in this paper restrict the use of TTSO for a broader set of modalities? If none, then how does this compare on other benchmarks?\n\t* (b) _LLMs for augmentation_: Why is the LLM necessary? The idea that LLMs can be used to produce different views of the sample is interesting, but the empirical performance appears to be too be marginal at best (+1.4% gain vs not using it from table 1). It doesn't seem to justify the additional computational burden for the small gain in performance. It would be helpful if the paper clearly articulates what the hypothesis is exactly here with regard to LLMs -- the ablation study in Fig 2 is a good start -- what kinds of LLMs benefit when fine-tuning? Is there a bias within the GPT-2 model used that is beneficial for these datasets? What about more advanced architectures/or LLMs trained on larger datasets, are they expected to give a bigger boost in performance or does it plateau?\n* I think the paper is interesting and has a lot to say, but i would recommend making the core hypothesis and idea cleaner as I have outlined above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(W1)** *OOD generalization in time series*: Why is this particular method effective for time series? as far as the assumptions made in the paper go, this is a general technique that can be applied to any arbitrary dataset/modality. While it is true that time series problems do not receive as much attention, that alone is not motivation enough. What assumptions in this paper restrict the use of TTSO for a broader set of modalities? If none, then how does this compare on other benchmarks?\n\n**(Reply to W1)** We appreciate your insightful comments. As pointed out by you, the OOD generalization for time series remains relatively under-explored. Our paper aims to bridge this gap by focusing on time series data, which allows us to provide a comprehensive analysis that might not be achievable if we were to cover multiple data types simultaneously.  However, please note its ability to be applied to other types of data as well is a testament to its versatility and robustness, rather than a limitation. \n\nIn addition, given the extensive body of existing work on OOD generalization in computer vision and natural language processing, addressing such an extension within the scope of a single paper is impractical. We hope this work can spur further research in this area and lead to a more comprehensive understanding of OOD generalization via this tri-level learning across various data modalities.\n\n**(W2)** *LLMs for augmentation*: Why is the LLM necessary? The idea that LLMs can be used to produce different views of the sample is interesting, but the empirical performance appears to be too be marginal at best (+1.4% gain vs not using it from table 1). It doesn't seem to justify the additional computational burden for the small gain in performance. It would be helpful if the paper clearly articulates what the hypothesis is exactly here with regard to LLMs -- the ablation study in Fig 2 is a good start -- what kinds of LLMs benefit when fine-tuning? Is there a bias within the GPT-2 model used that is beneficial for these datasets? What about more advanced architectures/or LLMs trained on larger datasets, are they expected to give a bigger boost in performance or does it plateau?\n\n**(Reply to W2)** Thank you for your valuable feedback. Your feedback is invaluable in enhancing the quality of our work. The inclusion of an LLM within the proposed TTSO framework is **optional** rather than **mandatory**. Specifically, the TTSO framework can be effectively applied to time series OOD generalization without the integration of an LLM. The consideration of LLM within the framework is due to its potential as an emerging research direction that warrants further exploration. In fact, previous studies have demonstrated that LLMs have been shown to be effective in transfer learning across various modalities[1], and pre-trained transformers can improve OOD robustness [2,3]. Additionally, in cases where computational complexity presents a significant challenge, the TTSO framework may be employed without the use of an LLM to tackle time series OOD generalization.\n\nTo further address your concerns regarding the hypothesis and efficacy of using LLMs in our framework, we conducted additional ablation experiments that focused on different LLM architectures and parameter sizes (e.g., base model and large model), including encoder-only (e.g., BERT), decoder-only (e.g., GPT-2), and encoder-decoder models (e.g., BART), to explore which configurations offer the most benefit during fine-tuning. The results are presented in the following table. \n\n| Architecture           | Version | HHAR | PAMAP | WESAD | AVG  |\n| ---------------------- | ------- | ---- | ----- | ----- | ---- |\n| Encoder-Only (BERT)    | Base    | 64.3 | 66.9  | 64.4  | 64.2 |\n|                        | Large   | 61.7 | 52.5  | 62.3  | 58.8 |\n| Decoder-Only (GPT)     | Base    | 72.9 | 76.1  | 68.4  | 72.5 |\n|                        | Large   | 64.5 | 69.4  | 66.5  | 66.8 |\n| Encoder-Decoder (BART) | Base    | 57.3 | 65.4  | 64.2  | 62.3 |\n|                        | Large   | 55.5 | 61.2  | 61.4  | 59.4 |\n\nThe results indicate that decoder-only architectures, specifically the GPT-2 base model in this experiments, achieve the best performance. However, increasing the number of parameters in all three architectures leads to a significant drop in performance across 3 architectures for time series OOD generalization.\n\nTo further explore how the number of parameters affects performance, we conducted experiments using GPT-2 models with varying numbers of Transformer layers on 3 datasets to evaluate their OOD generalization performance. For these experiments, we utilized 20% of each dataset. As shown in Figure 3 (in the attached PDF), the results demonstrate that optimal OOD generalization performance is achieved with a configuration of 8 Transformer layers. Based on this findings, we incorporate this optimal layer configuration with TTSO framework, yielding improved results as detailed in the following table. \n\n| Target | A    | B    | C    | D    | AVG   |\n| ------ | ---- | ---- | ---- | ---- | ----- |\n| HHAR   | +1.4 | +0.1 | +0.8 | +0.9 | +0.80 |\n| PAMAP  | +1.0 | -1.4 | +0.8 | -0.3 | +0.03 |\n| WEASAD | +5.5 | -3.5 | +2.5 | -0.1 | +1.35 |\n\n[1] One fits all: Power general time series analysis by pretrained lm (NeurIPS 2023)\n\n[2] How Good Are Large Language Models at Out-of-Distribution Detection? (arXiv 2023)\n\n[3] Pretrained Transformers Improve Out-of-Distribution Robustness (ACL 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ja5he565We", "reviewer": "Reviewer_tk3X", "summary": "The paper explores the challenge of OOD generalization in time series data using pre-trained LLMs. The authors propose a novel framework TTSO that integrates both sample-level and group-level uncertainties. They also introduce a stratified localization algorithm tailored to this tri-level optimization problem, theoretically demonstrating its convergence. The paper presents extensive experiments to validate the method’s effectiveness.", "questions": "1. How does the tri-level learning framework handle scenarios with highly imbalanced time series data?\n2. Can the proposed stratified localization algorithm be adapted for other types of data beyond time series?\n3. What are the specific computational requirements for implementing the proposed method on large-scale datasets?", "limitations": "NA", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The tri-level learning framework uniquely combines sample-level and group-level uncertainties, providing a comprehensive approach to OOD generalization.\n2. The paper includes a solid theoretical foundation, with analyses that justify the proposed method and its iteration complexity.\n3. The stratified localization algorithm offers a novel solution to the tri-level optimization problem, enhancing scalability and computational efficiency.\n4. Extensive experiments on real-world datasets demonstrate the effectiveness of the proposed method, showing significant performance improvements.\n5. The study leverages the advanced capabilities of LLMs in time series analysis, contributing to the emerging field of using foundational models for non-linguistic data.", "weaknesses": "1. The tri-level learning framework might be overly complex for practical applications, potentially limiting its adoption.\n2. The proposed method, especially the stratified localization algorithm, may incur high computational costs, which could be a barrier for large-scale applications.\n3. The paper could benefit from a more comprehensive comparison with other state-of-the-art methods in time series OOD generalization.\n4. There is a need for more discussion on the real-world applicability and potential limitations of the proposed method in various domains.\n5. Some sections of the paper are dense and challenging to follow, particularly the theoretical analyses, which might be difficult for a broader audience to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(W1 & W2)** The tri-level learning framework might be overly complex for practical applications, potentially limiting its adoption. The proposed method, especially the stratified localization algorithm, may incur high computational costs, which could be a barrier for large-scale applications.\n\n**(Reply to W1 & W2)** We appreciate the concern regarding the complexity of the tri-level learning framework. Time series OOD generalization is a challenging problem that necessitates a sophisticated and comprehensive approach. The TTSO framework is theoretically motivated by Theorem 2 (derived from [1]) and represents the **first work** to integrate both sample-level and group-level uncertainties under a tri-level learning framework for time series OOD generalization. This approach is crucial for effectively tackling the unique challenges of time series OOD generalization. \n\nAdditionally, the stratified localization algorithm is **more computationally** efficient compared to hypergradient-based tri-level optimization methods[2,3]. Furthermore, the decomposable nature of cutting planes provides a promising pathway for distributed implementations of TTSO (e.g., ADBO[4], AFBO[5]), allowing the algorithm to be effectively applied to large-scale applications.\n\n**(W3)** The paper could benefit from a more comprehensive comparison with other state-of-the-art methods in time series OOD generalization.\n\n**(Reply to W3)** We have included comparisons with two time series OOD generalization methods: DFDG [6] and CCDG [7]. The results is detailed in following tables. \n\nHHAR\n\n| Target| A    | B  | C  | D   | AVG  |\n| --------- | ---- | ---- | ---- | ---- | ---- |\n| DIVERSIFY | 73.7 | 64.2 | 78.9 | 71.2 | 71.8 |\n| DFDG| 71.2 | 65.8 | 74.1 | 70.4 | 70.3 |\n| CCDG| 73.0 | 63.2 | 77.3 | 72.4 | 71.5 |\n| TTSO*| 77.6 | 67.3 | 80.6 | 69.9 | 73.9 |\n\nPAMAP\n\n| Target| A    | B | C| D    | AVG  |\n| --------- | ---- | ---- | ---- | ---- | ---- |\n| DIVERSIFY | 74.0 | 84.0 | 56.5 | 72.9 | 72.0 |\n| DFDG| 73.1 | 80.5 | 59.2 | 70.2 | 70.8 |\n| CCDG| 72.3 | 84.8 | 56.6 | 72.1 | 71.5 |\n| TTSO* | 78.5 | 89.6 | 61.4 | 75.0 | 76.1 |\n\nWEASAD\n\n| Target    | A    | B| C | D  | AVG  |\n| --------- | ---- | ---- | ---- | ---- | ---- |\n| DIVERSIFY | 57.6 | 73.0 | 72.6 | 57.1 | 64.6 |\n| DFDG | 49.8 | 71.6 | 71.1 | 50.7 | 60.8 |\n| CCDG | 54.5 | 70.5 | 69.8 | 54.1 | 62.2 |\n| TTSO* | 59.5 | 71.9 | 77.3 | 65.0 | 68.4 |\n\n**(W4)** There is a need for more discussion on the real-world applicability and potential limitations of the proposed method in various domains.\n\n**(Reply to W4)** Per your suggestion, we have added more discussion of TTSO framework on the real-world applicability and potential limitations in our manuscript (Appendix G & F) as follows. \n\n1. Real-world Applicability. Our study on time series OOD generalization has significant potential in sensor-based applications, such as human activity and emotion recognition. For instance, our method can improve model robustness and accuracy against distribution shifts in healthcare, sports training, and abnormal behavior detection.\n2. Potential Limitations. While our method demonstrates significant potential in time series OOD generalization, the ideas we propose are versatile and can be extended to more general settings. However, this may pose a challenge since different types of data have distinct sample-level uncertainties, which require reformulating the third-level optimization problem to effectively manage these uncertainties.\n\n**(W5)** Some sections of the paper are dense and challenging to follow, particularly the theoretical analyses, which might be difficult for a broader audience to understand.\n\n**(Reply to W5)** We understand that the theoretical sections of our paper may be challenging for some readers. However, these analyses are essential as they provide the necessary foundation and justification for our method's robustness and effectiveness. Per your suggestion, we've added explanations to make these sections more accessible to a broader audience in Appendix A.\n\n**(Q1)** How does the tri-level learning framework handle scenarios with highly imbalanced time series data?\n\n**(Reply to Q1)** In this paper, our primary focus is on time series OOD generalization, not on handling highly imbalanced data. However, it's worth mentioning that the proposed framework is flexible and effective. By incorporating established techniques such as data resampling (e.g., oversampling minority classes or undersampling majority classes) and synthetic data generation (e.g., SMOTE), TTSO framework can be adapted to address data imbalance issues effectively.\n\n**(Q2)** Can the proposed stratified localization algorithm be adapted for other types of data beyond time series?\n\n**(Reply to Q2)** Yes, the TTSO framework we proposed is versatile and applicable to various data modalities. However, our focus in this work was specifically on time series data. Focusing on time series data allows us to provide a comprehensive analysis that might not be achievable if we were to cover multiple data types simultaneously. \n\n**(Q3)** What are the specific computational requirements for implementing the proposed method on large-scale datasets?\n\n**(Reply to Q3)** Currently, the proposed method has been tested on a setup with two NVIDIA RTX 4090 GPUs and an Intel i9-14th generation processor. For large-scale datasets, the computational requirements will scale accordingly. \n\n[1] A theory of learning from different domains (ML 2010)\n\n[2] A gradient method for multilevel optimization (NeurIPS 2021)\n\n[3] Betty: An Automatic Differentiation Library for Multilevel Optimization (ICLR 2023)\n\n[4] Distributed distributionally robust optimization with non-convex objectives (NeurIPS 2022)\n\n[5] Provably Convergent Federated Trilevel Learning (AAAI 2024)\n\n[6] Robust domain-free domain generalization with class-aware alignment (ICASSP 2021)\n\n[7] Conditional Contrastive Domain Generalization for Fault Diagnosis (IEEE TIM 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OeiEqbzFqU", "reviewer": "Reviewer_9773", "summary": "The paper presents a novel approach for time series out-of-distribution generalization via pre-trained large language models.\nThe authors introduce a Tri-level learning framework that combines sample-level and group-level uncertainties accompanied by a theoretic perspective. Furthermore, a stratified localization algorithm is proposed for the tri-level optimization problem followed by a theoretical convergence guarantee. The paper demonstrates performance gain in six real-world time series datasets, demonstrating the effectiveness of the method.", "questions": "Can this method be adapted for OOD time series regression and forecasting?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The method proposed by the authors is sensible and seems to perform well compared to other out-of-distribution generalization methods.\n\n2. Authors provide an in-depth analysis of their method and provide a convergence analysis.\n\n3. The paper conducts a fair experiment on six real-world time series datasets, showing superior performance in out-of-distribution generalization.", "weaknesses": "I do not see any major weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for recognizing our work. \n\n**(Q1)** Can this method be adapted for OOD time series regression and forecasting?\n\n**(Reply to Q1)** Yes, our proposed tri-level learning framework is designed to learn robust representations for time series OOD generalization, which means the learned representations could potentially be applied to other downstream tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Figure 1, Figure 2 and Figure 3 is in the attached PDF.

Meta-review: This paper focuses on time-series out-of-distribution generalization with LLMs, provides new methods that use uncertainties and localization, and gives theoretical analysis on the generalization properties and convergence. Reviewers appreciated the application of LLMs to the problem domain and found good improvements in the experimental section. Although the theoretical work may not be easily accessible to a wide audience, most of the reviewers appreciated the work to show convergence properties.

While the paper restricted its focus to time series and did not test on other modalities, reviewers noted that the methods did not seem to rely on any specific aspects of time-series problems, and in that sense could have been tested more broadly.

In the discussion phase, the authors showed new results using a variety of architectures and LLM sizes. Somewhat curiously, larger LLMs led to worse performance on the paper’s tasks. The authors are encouraged to think more about the possible causes of this observation and provide substantive discussion, as a practitioner could pick up this technique, think to use the most advanced LLM available, and find poor OOD generalization results.
Decision: Accept (poster)