Enforcing Interpretability in Time Series Transformers: A Concept Bottleneck Framework

Authors: Angela van Sprang, Erman Acar, Willem Zuidema
Primary area: interpretability and explainable AI
Keywords: Interpretability, Concept Bottleneck Model, Centered Kernel Alignment, Autoformer, Time Series Transformer

Abstract:
There has been a recent push of research on Transformer-based models for long-term time series forecasting, even though they are inherently difficult to interpret and explain. While there is a large body of work on interpretability methods for various domains and architectures, the interpretability of Transformer-based forecasting models remains largely unexplored. To address this gap, we develop a framework based on Concept Bottleneck Models to enforce interpretability of time series Transformers. We modify the training objective to encourage a model to develop representations similar to predefined interpretable concepts. In our experiments, we enforce similarity using Centered Kernel Alignment, and the predefined concepts include time features and an interpretable, autoregressive surrogate model (AR). We apply the framework to the Autoformer model, and present an in-depth analysis for a variety of benchmark tasks. We find that the model performance remains mostly unaffected, while the model shows much improved interpretability. Additionally, interpretable concepts become local, which makes the trained model easily intervenable. As a proof of concept, we demonstrate a successful intervention in the scenario of a time shift in the data, which eliminates the need to retrain.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730297230067

thread_type:
official_review

<thread_object>
creator: reviewer_m89b

<content>
summary:
This paper proposes a framework to enforce interpretability in time-series forecasting Transformers by adapting the Autoformer model with a concept bottleneck approach. The framework aligns the model’s representations with interpretable concepts, such as a surrogate AR model and time-based features, using Centered Kernel Alignment (CKA). This structure aims to make parts of the Autoformer model more transparent, allowing practitioners to interpret the model’s reasoning and make targeted interventions if needed. The paper demonstrates that the proposed framework maintains interpretability with a minimal performance trade-off across six time-series datasets.

strengths:
1. Novel Interpretability Framework: This work contributes a new direction in Transformer interpretability by combining Concept Bottleneck Models (CBMs) with the Autoformer, explicitly aligning model representations with interpretable concepts. 

2. Few Performance Trade-offs with Useful Intervention Property: Table 1 shows that while the Autoformer with bottlenecks generally has a slight performance trade-off, the interpretability improvements may be valuable in settings where transparency is essential. Additionally, the “Intervention” experiment (Lines 480-485) demonstrates a practical application of the framework, where a temporal shift intervention shows the model’s adaptability to new data distributions, a useful feature in evolving environments.

4. Potential Balance of Interpretability and Complexity: By modifying only a single layer to incorporate the concept bottleneck and aligning some heads with interpretable concepts, the framework achieves interpretability without overhauling the Transformer architecture. This approach makes the Autoformer’s components "easily intervenable," according to the authors, providing a possible solution for practitioners needing complex forecasting models with interpretable checkpoints.

weaknesses:
1. Limitations in Granular Interpretability: CKA encourages global alignment of the bottleneck representations with the predefined concepts, which may not capture fine-grained temporal patterns that are essential in many time-series applications. In the CKA analysis (Figure 3), alignment scores reflect similarity with concepts on a broad level but do not offer insights at specific time intervals or for anomalies. This setup could limit interpretability for users who need detailed, time-specific insights. Extending the interpretability framework to capture these localized patterns would make the model’s insights more actionable.

2. Interpretability Evaluation Metrics: The interpretability evaluation relies mainly on CKA scores and qualitative visualizations. Although CKA scores indicate alignment between model representations and interpretable concepts, they do not provide a full measure of "practical interpretability" from an end-user perspective. Incorporating metrics that measure interpretability in terms of clarity or usefulness for decision-making could make the framework’s impact clearer and more valuable.

3. Applicability Across Different Models: Although the framework is applied to the Autoformer model, extending it to other more performant Transformer-based time-series models would confirm its generalizability. While the authors mention this as a possible future direction (Lines 530-531), this limits the scientific contribution of the work.

4. Model diagrams (Figure 1 and 2) and the CKA scores (Fig 3) could be presented more clearly. They often require frequent referrals back to the text and legend.

questions:
1. Justification for Concept Bottleneck over Post-Hoc Methods: Could you clarify the specific advantages of using the concept bottleneck framework over post-hoc interpretability methods like SHAP, LIME, or attention-based visualizations in this time-series context? An expanded discussion would help in understanding the unique benefits of your approach for interpretability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636945610

thread_type:
official_review

<thread_object>
creator: reviewer_ztc3

<content>
summary:
The paper proposes a time series Transformer model to be more interpretable with concept bottlenecks using time features and simple autoregressive models as interpretable concepts.
* A training framework encouraging the similarity between transformer representations and pre-defined interpretable concepts using CKA.
* Application of Autoformer on the model was made and its performance was evaluated on 6 benchmark datasets.
* Demonstrate the capability of model intervention in case of temporal shifts.
* Extensive interpretation analysis supported by the visualization technique.

strengths:
* Novel application of CBM to time series
* Creative use of CKA for concept alignment
* Integration with Autoformer architecture
* Novel intervention capabilities

* Comprehensive experiments on 6 datasets
* Detailed ablation studies
* Visualization of learned concepts
* Intervention demonstration

* Comparable performance to baseline
* Interpretable predictions
* Intervention capabilities for temporal shifts
* Domain-agnostic approach

weaknesses:
* Single model architecture (Autoformer)
* Further research is needed to apply CBM to other types of predictive models
* Selection of interpretable concepts relies on heuristics
* Limited analysis of statistical significance
* No comparison to other interpretability methods

* Potential information leakage not fully addressed
* Limited analysis of concept quality
* No theoretical guarantees
* Trade-offs not fully explored


* In Table 1, the simple AR model outperforms the Autoformer with bottleneck in 4 out of 6 datasets. Since the AR model is inherently interpretable, these results may suggest that the proposed method is less effective than expected. The authors could consider adding more complex datasets to strengthen the experimental evaluation.

Suggestions for Improvement
* Compare with other interpretability methods; Add user studies with domain experts; Provide more complex intervention scenarios; Test on longer sequences
* Study concept quality metrics; Analyze computational overhead; Evaluate statistical significance; Investigate scaling properties
* Test with other transformer architectures; Explore more complex concepts; Add theoretical guarantees

questions:
* Is there a qualitative or quantitative comparison of the proposed method with other XAI techniques?

* The results of the intervention experiment are intriguing, but the purpose of this experiment remains somewhat unclear. Could the authors provide a more detailed analysis, discussion, or examples to clarify this?

* How were the specific interpretable concepts (AR model and time features) chosen? Were other concepts considered?

* How do you validate that the learned concepts are truly interpretable and meaningful? Have you conducted any user studies with domain experts?

* Why was Autoformer specifically chosen as the base architecture?
Would the approach work similarly with other transformer variants?

* What is the impact of bottleneck location on performance and interpretability? Was there a systematic study of different locations?

* How sensitive is the training to the CKA loss weight α? Are there guidelines for selecting this parameter?

* What is the computational overhead of the bottleneck compared to standard Autoformer? How does this scale with sequence length?

* How do you quantitatively evaluate the quality of interpretations? Are there metrics beyond CKA scores?

* How generalizable is the intervention approach to other types of shifts? What are the limitations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730658757337

thread_type:
official_review

<thread_object>
creator: reviewer_rmbn

<content>
summary:
The authors develop a concept bottleneck model for time-series forecasting with the objective of improving interpretability. Concept bottleneck models are a pre-existing approach to interpretability whereby the model aims to predict a set of concepts first, and then only uses the predicted concepts for the final forecast.

Starting from the Autoformer architecture, the authors introduce two types of bottlenecks (an autoregressive forecast and a time-of-day prediction). To ensure all information passes through the bottlecneck, they then ablate the residual connections. Finally the training loss is an interpolation of the standard loss + a score based on the similarity score CKA of the model’s representations and interpretable concepts.

strengths:
The strengths are as follows:

- The paper is well-written, and the idea is expressed clearly.
- The authors achieve what they set out to do: their model functions at the intended task.

weaknesses:
The weaknesses of the paper are as follows:

- In reviewing the performance results in Table 1, as the authors themselves acknowledge there is no significant improvement in performance (as is to be expected given the algorithm, this is of course not an issue). The paper however lacks a comparative analysis of this interpretability against other methods that also offer interpretable time-series prediction: does their approach outperform others in that space?

- Although the concept is intriguing, it feels somewhat derivative, essentially applying concept bottlenecks to time-series forecasting. One immediate concern is the relative lack of novelty. This may not be a significant issue if there were more extensive analysis of the components in their approach, yet the exploration remains somewhat limited. Specifically, other proxy tasks for the interpretable concepts could have been explored, as well as other components (e.g. bottleneck location, similarity metric used, transformer models...).

- The authors note that the AR model outperforms other approaches. This finding is not unexpected given prior work (e.g., [1]), but further analysis is warranted. The key unanswered question, in my view, is how much of the absence of performance degradation is due to the strong proxy task provided by AR (i.e. is their model performing as well as the unaltered baseline only due to the strong signal provided by the AR subtask).

- The dataset selection is somewhat limited. The authors mention that the time-series analyzed in this study had strong linear characteristics, which likely explains the AR model's performance. This could motivate the use of more complex datasets to verify if the findings hold more broadly.


References
[1] FreDo: Frequency Domain-based Long-Term Time Series Forecasting.

questions:
Please refer to the weaknesses section above for questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730835442293

thread_type:
official_review

<thread_object>
creator: reviewer_j9av

<content>
summary:
In an effort to develop more interpretable time-series forecasting models, the authors have combined a transformer-based architecture (Autoformer) with a concept bottleneck approach. The concepts do not correspond to any a priori annotations bur rather are derived either from an autoregressive model or from sample timestamps.  The authors encourage the network to "reason" using these concepts by adding an additional term to the loss function that captures the similarity between the model's internal representations and the precomputed concepts. The balance between prediction error and representational alignment (in the cost function) with the concepts is regulated through a single hyperparameter. Furthermore, the alignment scores with the different concepts (as captured by CKA) seem to make intuitive sense for many of the datasets (electricity usage and time of day for example). Overall, this is an interesting approach to a timely problem in the field. That said, there are some open questions about the approach that need to be addressed.

strengths:
- The goal of the paper, the presentation, and the implementation details are clear
- The approach does not require costly annotations for concepts and can (arguably) be applied to any time-series data

weaknesses:
- Looking at the qualitative results in Figure 9 and the summary in Table 1, this approach seems to do well for data that has a strong cyclical component (traffic and electricity). In fact, for all other datasets, the simpler AR model works best. How do you explain this? It seems like you get performance AND interpretability using an AR model, then why do you need a model with many more parameters? Maybe there are other datasets that could highlight the benefit of this approach (vs a simple AR based model) a little better? Perhaps I misunderstood something.
- It's hard to get an understanding of how the model is leveraging the concepts, especially since your results on hyper-parameter sensitivity (Table 5, Appendix F) are not the most intuitive; the first and second best settings of the alpha parameter are far apart (0.7 and 0.0). For pedagogical reasons, it might help to train the model on a synthetic dataset, constructed with the concepts (+noise) of your choice. Using a synthetic dataset might give the reader some more mechanistic intuition.

questions:
- Is the optimal setting for the hyper-parameter dataset specific?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732286302706

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Update paper: additional experiments on Vanilla Transformer

comment:
To address the common weakness mentioned by multiple reviewers (regarding application to only one Transformer architecture), we included a new Appendix to the paper where we apply the framework to a different Transformer architecture: the vanilla Transformer. The Appendix is included at the end of the document, and the new parts are written in blue.

These additional results confirm the conclusions from the Autoformer experiments, in particular that the framework can be applied to a time series Transformer without having any significant impact on the overall model performance, while providing improved interpretability. Similar to the Autoformer model, the vanilla Transformer performs better than the AR model for the ‘Electricity’ and ‘Traffic’ dataset. 

Since all time series Transformer architectures are derived from the vanilla Transformer, this deeper exploration with the framework highlights the general applicability of the framework. We kindly invite you to look at these additional results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732738939094

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional experiment: synthetic dataset

comment:
We would like to notify the reviewers that we have included new results on a synthetic dataset in Appendix I, following the suggestion by Reviewer J9av. We believe these additional experiments help in understanding how the model leverages interpretable concepts. In summary, by increasing the weight of the CKA loss, we show that the bottleneck components become increasingly more similar to the (ground-truth) interpretable concepts, which helps with the forecasting task. The new parts are written in blue.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel framework for interpretability in time-series forecasting by integrating the concept bottleneck approach with the transformer-based Autoformer architecture. Instead of relying on predefined annotations, the model derives interpretable concepts from surrogate autoregressive (AR) models or sample timestamps. A Centered Kernel Alignment (CKA)-based loss term encourages the model’s internal representations to align with these concepts. The authors evaluate their approach on six benchmark datasets and demonstrate that the method maintains comparable predictive performance while improving transparency and enabling targeted interventions, such as handling temporal shifts.

Strengths

+ The application of the concept bottleneck model (CBM) to time-series forecasting is novel and well-motivated.
+ The paper introduces a creative use of CKA to align model representations with interpretable concepts.
+ The proposed framework integrates seamlessly with Autoformer without requiring costly annotations.
+ Intervention capabilities, such as handling temporal shifts, are demonstrated.
+ The approach maintains interpretability with minimal performance trade-offs.
+ The methodology is clear, and the paper is well-written and easy to follow.
+ The method is domain-agnostic, showing potential for broader applicability to other time-series tasks.

Weaknesses

+ The AR model outperforms the Autoformer with bottlenecks in four out of six datasets, questioning the need for a more complex model.
+ There is no comparison with other interpretability methods (e.g., SHAP, LIME, or attention-based visualizations).
+ The selection of interpretable concepts is heuristic, and the quality of these concepts is not thoroughly analyzed.
+ The analysis of hyperparameter sensitivity, such as the α weight, is limited and produces inconsistent results.
+ The CKA-based alignment encourages global similarity but does not capture fine-grained or localized temporal patterns.
+ The datasets used are relatively simple, primarily reflecting cyclical behaviors, limiting the generalizability of results.
+ There is no systematic study of the computational overhead, bottleneck locations, or scalability of the approach.
+ Visualizations and CKA analysis could be presented more clearly to enhance interpretability.

Some concerns have been addressed by the authors during the rebuttal period.