Title: Unified Transferability Metrics for Time Series Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Time Series Analysis; Transferability Estimation; Pre-trained Model
Abstract: With the increasing number of time series pre-trained models, designing transferability evaluation metrics for time series has become an urgent problem to address. 
While transferability evaluation has been extensively studied in computer vision, we aim to address a critical gap by developing tailored metrics for time series analysis. 
In this paper, we introduce TEMPLATE, a transferability estimation framework specifically tailored for versatile time series analysis, comprising three complementary metrics: (1) Dependency Learning Score quantifies a model’s capacity to capture temporal dependencies. (2) Pattern Learning Score evaluates the representation quality in extracting discriminative temporal patterns. (3) Task Adaptation Score assesses cross-task generalization capability, enabling versatile time series analysis. TEMPLATE presents a versatile framework compatible with both classification and regression paradigms. Through comprehensive benchmarking across 5 distinct downstream tasks, our method demonstrates superior capability in identifying optimal pre-trained models from heterogeneous model pools for transfer learning. Compared to the state-of-the-art method ETran, our approach improves the weighted Kendall's $\tau_w$ across 5 downstream tasks by 35\%. The code is available at https://github.com/ooooooover/TEMPLATE.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the underexplored but important problem of evaluating the transferability of pre-trained models in time series tasks. The authors propose TEMPLATE, a framework that combines three complementary metrics: Dependency Learning Score, Pattern Learning Score, and Task Adaptation Score, to assess the capability of a model to transfer across tasks. The method is validated on 32 datasets across five types of downstream tasks (classification, short- and long-term forecasting, imputation, and anomaly detection), showing consistent performance gains over existing methods such as ETran and LogME.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Problem significance: Transferability estimation for time series pre-trained models is a crucial but overlooked problem, which this paper tackles directly.\n2. Methodological novelty: The paper proposes three task-agnostic and interpretable metrics based on temporal dependency, dominant pattern learning, and hierarchical adaptation.\n3. Extensive experiments: The study covers 5 tasks, 32 datasets, and 10 pre-trained models, with clear benchmarking against 8 prior methods.\n4. Thorough analysis: Ablation studies show complementary benefits of all metrics; time complexity analysis highlights efficiency and scalability.\n\nWeaknesses:\n1. The framework evaluates only static representations, without considering the interaction with fine-tuning dynamics (e.g., sensitivity to learning rate).\n2. While the trend decomposition is intuitive, it may not perform well on non-stationary series with structural breaks; this is not discussed in detail.\n3. The use of a manually-set λ coefficient for task-specific weighting (±1) may limit generality; adaptive or learned weighting would be more flexible.\n4. Anomaly detection performance on some datasets (e.g., SMD) is suboptimal, but the reasons are not fully explored in the main text.\n\nQUESTIONS:\n1. Have you considered learning the task-specific λ value adaptively rather than setting it manually based on task type?\n2. How does TEMPLATE perform on non-stationary time series with structural breaks or regime shifts? Would a dynamic trend extractor help?\n3. It would be informative to include model architecture ablations (e.g., Transformer vs CNN) to assess architecture-agnostic robustness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose three specialized metrics for evaluating the transferability of time-series pre-trained models, specifically designed for time series foundation models. These include: (1) the Dependency Learning Score, which quantifies a model’s ability to capture temporal dependencies; (2) the Pattern Learning Score, which assesses the representation quality in extracting discriminative temporal patterns; and (3) the Task Adaptation Score, which measures cross-task generalization capability. These evaluation metrics are applicable to both classification and regression cases. The authors validate the effectiveness of these metrics on five downstream tasks and demonstrate their superior performance compared to the state-of-the-art ETran method.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The authors address a problem of significant importance by developing specialized model transferability evaluation methods tailored for time series characteristics. \n- The proposed approach features computationally simple and efficient logic.\n- The paper is clearly written and accessible.\n\n### Weaknesses\n- Some formulas in the paper contain flaws, as referenced in the  **Questions** section below.\n- The metrics adopted in the experimental section are not entirely reasonable, as discussed in the **Limitations** section below.\n\nQUESTIONS:\n- The authors claim that \"The largest singular value typically corresponds to the primary temporal pattern in time series\" (line 180). Could you clarify the formal definition of temporal pattern in this context? What are its potential manifestations (e.g., trends, periodicity, or other temporal structures)?\n- There are issues with Equation (5). First, in the conventional sense, $\\mathrm{tr}(A)$ is not equivalent to the Frobenius norm of $A$. Additionally, here $H^1$ represents the output of the first neural network layer, while $H$ denotes the final layer's output. If their dimensions differ, then the computation of $H^1*H^T$ would be infeasible. Are there additional computational steps not explicitly stated here?\n- Regarding Equation (10), how are $\\min(S_{dl})$ and $\\max(S_{dl})$ computed? Based on the earlier description, $S_{dl}$ appears to be calculated from a batch of data. Does this imply that the min and max values correspond to the minima and maxima across multiple batches?\n- The UEA archive contains nearly 30 datasets, but only 9 are selected for the experimental section (line 274). What are the selection criteria? How did the metrics perform on the other datasets?\n\nLIMITATIONS:\nThe authors adopt Kendall's $\\tau$ as the evaluation metric for the proposed measurement method in this paper. The experimental results show relatively low $\\tau$ values, which implies that—with high probability—the truly optimal model may not be the one ranked highest by $\\tau$. From a practical standpoint, the core question we typically care about is: Which model is the best among a pool of candidates? Consequently, the accuracy rate of identifying the top-performing model could serve as a more meaningful evaluation criterion than $\\tau$ itself.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TEMPLATE, a novel framework for estimating the transferability of time series pre-trained models. TEMPLATE consists of three complementary metrics—Dependency Learning Score, Pattern Learning Score, and Task Adaptation Score—designed to address unique challenges in time series, such as cross-task heterogeneity and feature space mismatch. The framework demonstrates strong empirical performance on five mainstream time series tasks, outperforming prior transferability estimation methods such as ETran and LogME, while maintaining scalability via efficient approximations like power iteration.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.  The methodology is sound, with clear mathematical formulations and robust benchmarking across a wide range of datasets. The use of power iteration for scalability is a practical engineering contribution.\n2.  The proposed metrics are designed specifically for time series, filling a clear gap in existing literature focused primarily on vision tasks.\n3.  The paper is generally well-written with logical structure, clear figures, and detailed appendices.\n\n**Weaknesses:**\n\n1.  While the three proposed scores (Dependency Learning Score, Pattern Learning Score, and Task Adaptation Score) are intuitively motivated and empirically validated, the paper lacks a theoretical analysis or formal justification to support why these metrics correlate well with transferability across diverse tasks.\n2.  The final transferability score combines three normalized metrics with fixed equal weights (and a task-type-based sign for $\\lambda$). However, the choice of this fixed linear aggregation lacks theoretical grounding or empirical sensitivity analysis. There may be cases where certain metrics dominate or interfere depending on task characteristics.\n3.  The main quantitative evaluation focuses on Kendall's $\\tau_w$, which reflects ranking correlation but does not capture real-world model selection performance such as top-1 accuracy or top-k hit rate. Users in practice may care more about how often the top-ranked model is truly best-performing.\n4.  The performance of TEMPLATE lags behind ETran on certain anomaly detection datasets (e.g., PSM and SWaT in Table 6), suggesting a degree of sensitivity to dataset-specific characteristics or distribution shifts.  Since real-world time series anomaly detection often involves non-stationary data or rare events, further analysis of failure modes or robustness under distribution shift would strengthen the method's practical applicability.\n\nQUESTIONS:\n1. Have you considered learning the weights of the three metrics instead of using a fixed aggregation with task-specific sign? Would a learned or adaptive approach improve performance or robustness?\n2. Beyond Kendall's $\\tau_w$, how well does TEMPLATE perform in practical top-k model selection scenarios (e.g., top-1 accuracy or top-3 hit rate)?\n3. Performance on datasets like PSM and SWaT appears to lag behind ETran. Could you provide insights into what type of distribution shift harms TEMPLATE, and how it might be mitigated?\n4. Do the three sub-scores interact or overlap in what they measure? For instance, does high dependency learning score already imply good pattern learning?\n5. Can TEMPLATE be extended to or applied in zero-shot or few-shot learning scenarios, which are increasingly relevant in time series foundation model usage?\n\nLIMITATIONS:\nYes.  The paper discusses the limitations in handling datasets with severe distribution shifts or structural breaks, and acknowledges that this remains a challenging open problem.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes ​​TEMPLATE​​, a novel framework for evaluating transferability in time-series foundation models. Addressing the lack of tailored metrics for time series (unlike computer vision), TEMPLATE introduces three complementary scores: ​​Dependency Learning Score​​, ​​Pattern Learning Score​​ and Task Adaptation Score. Extensive experiments are conducted across across 32 datasets and 5 downstream tasks, and the result demonstrate the effectiveness of TEMPLATE.

STRENGTHS AND WEAKNESSES:
## Strengths
1. The paper dedicates transferability metric for time-series foundation models, addressing a critical gap in literature.
2. The proposed method is benchmarked rigorously across ​​5 distinct tasks​​ and ​​32 datasets​​, including multivariate and large-scale benchmarks (e.g., Traffic: 862 channels).
3. The proposed method avoids brute-force fine-tuning via power iteration and nuclear norm approximations, reducing computational overhead.

## Weaknesses
1. Reliance on SVD assumes principal components reliably represent transferable knowledge. This may not hold for noisy or non-stationary data, where lower singular values could be informative.
2. Dependency Learning Score​​ uses average pooling for trend extraction (Eq. 1), potentially oversimplifying complex temporal dependencies (e.g., multi-scale trends).

QUESTIONS:
See Weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02