Investigating Pattern Neurons in Urban Time Series Forecasting

Authors: Chengxin Wang, Yiran Zhao, Shaofeng Cai, Gary Tan
Primary area: learning on time series and dynamical systems
Keywords: urban time series forecasting, neuron detection

Abstract:
Urban time series forecasting is crucial for smart city development and is key to sustainable urban management. Although urban time series models (UTSMs) are effective in general forecasting, they often overlook low-frequency events, such as holidays and extreme weather, leading to degraded performance in practical applications. In this paper, we first investigate how UTSMs handle these infrequent patterns from a neural perspective. Based on our findings, we propose $\textbf{P}$attern $\textbf{N}$euron guided $\textbf{Train}$ing ($\texttt{PN-Train}$), a novel training method that features (i) a $\textit{perturbation-based detector}$ to identify neurons responsible for low-frequency patterns in UTSMs, and (ii) a $\textit{fine-tuning mechanism}$ that enhances these neurons without compromising representation learning on high-frequency patterns. Empirical results demonstrate that $\texttt{PN-Train}$ considerably improves forecasting accuracy for low-frequency events while maintaining high performance for high-frequency events. The code is available at https://github.com/cwang-nus/PN-Train.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729503140441

thread_type:
official_review

<thread_object>
creator: reviewer_t7mg

<content>
summary:
This work addresses the challenge of predicting low-frequency events (e.g., holidays, emergencies) in urban time series data, which are often overlooked by traditional models. The authors propose Pattern Neuron guided Training (PN-Train), a novel method that identifies and fine-tunes neurons specifically associated with low-frequency patterns. PN-Train incorporates a perturbation-based Pattern Neuron Detector (PND) to locate key neurons, and a Pattern Neuron Optimizer (PNO) to enhance their learning without compromising high-frequency event predictions.

strengths:
S1: PN-Train introduces a novel approach by using a perturbation-based detector to identify neurons associated with low-frequency patterns and fine-tuning these neurons to improve prediction accuracy. This is an innovative perspective, combining neuron-level analysis with practical forecasting tasks.

S2: By capturing and fine-tuning neurons under different patterns, PN-Train enhances the fairness of predictions for low-frequency events, reduces model bias, and helps in forecasting scenarios with sparse data.

S3: The experiments provide solid evidence of the existence of Pattern Neurons, with case studies offering visualized results that make the concept highly intuitive and easy to understand.

weaknesses:
W1: While the paper introduces neuron detection and optimization methods, it lacks in-depth theoretical explanation for why these neurons exhibit distinct behavior for low-frequency events. More analysis and discussion[1,2] on the working mechanisms of pattern neurons, such as potential differences in weight distribution or activation between high- and low-frequency events, is needed, along with supporting theoretical calculations.

W2: The paper highlights patterns for holidays and emergencies in the dataset, but in real-world applications, low-frequency events can be more random and harder to predict. Further discussion on how PN-Train handles unpredictable low-frequency events, such as concert events, sport events, or extreme weather like typhoon,  and then testing its generalization ability in more complex scenarios would be beneficial.

W3: There are some typos, such as inconsistent citation formatting in reference 766, the use of "$B" in Figure 5b, and inconsistency between "B" in Figure 5 and "D" in line 486. Please clearfully check for typos throughout the paper.


[1] Zhou Z, Huang Q, Wang B, et al. ComS2T: A complementary spatiotemporal learning system for data-adaptive model evolution[J]. arXiv preprint arXiv:2403.01738, 2024.

[2] Tang T, Luo W, Huang H, et al. Language-specific neurons: The key to multilingual capabilities in large language models. ACL, 2024

questions:
Please see W1-W3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630609136

thread_type:
official_review

<thread_object>
creator: reviewer_3uym

<content>
summary:
This paper firstly investigates how urban time series models (UTSMs) can be utilized to deal with the low-frequence events in urban management. It proposes a pattern neuron-guided training (PN-Train) method consisting of a perturbation-based detector and a fine-tuning mechanism. The experiments on a series of time series forecasting tasks show the superiority of PN-Train, which enhances performance in low-frequency events while maintaining high performance in high-frequency events.

strengths:
[1] A simple yet effective method is proposed to deal with the low-frequency patterns in urban time series data. It is of certain novelty to design a method from the neural aspect, making  

[2] Experiments on multiple datasets validate the existence of pattern neurons and the proposed PND stage successfully detects them. Overall, the empirical studies are solid. 

[3] The paper is well-written and easy to follow, with sufficient technical details on the pattern neuron detector, pattern neuron verifier, and pattern neuron optimizer.

weaknesses:
[1] It is better to have a more complete discussion about low-frequency events, e.g., how these events are handled in previous works (Related Works section), how low-frequency events should be technically defined, and how the evaluation metrics are established to verify the effectiveness of the methods. 

[2] Lack of comparison with baseline methods that deal with various unbalanced data distribution problems.

questions:
Q1. Is there any way to quantitatively define or model the frequency of events? 

Q2. Is there any theoretical principle that can support the neuron-based strategy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634249038

thread_type:
official_review

<thread_object>
creator: reviewer_xvj7

<content>
summary:
This work introduces PN-Train, a novel urban time series forecasting method that significantly improves prediction accuracy by identifying and fine-tuning pattern neurons associated with low-frequency events, such as holidays. PN-Train employs a perturbation-based detector to recognize these neurons and enhances them through a fine-tuning mechanism without compromising the representation learning of high-frequency patterns. Empirical results demonstrate that PN-Train considerably boosts forecasting accuracy for low-frequency events while maintaining high performance for high-frequency events.

strengths:
(1) The work proposes a perturbation-based detector that effectively identifies pattern neurons associated with low-frequency patterns.

(2) It designs a fine-tuning mechanism to enhance these pattern neurons without compromising the representation of high-frequency patterns. This design ensures high accuracy for both low-frequency and high-frequency events.

(3) The study provides empirical results that verify the effectiveness of the proposed method, demonstrating significant performance improvements.

weaknesses:
(1) The distinction between low-frequency and high-frequency events is primarily between holidays and non-holidays, which may not have substantial practical significance. Other baselines also improve accuracy for both low-frequency and high-frequency events simultaneously. Focusing on anomaly detection or traffic prediction under anomalous events might have been more meaningful.

(2) The absence of open-source code makes it difficult to reproduce the results.

(3) The study utilizes too few datasets. It would have been better to use datasets from the PeMS system, such as Large-ST [1], which are well-processed and cover longer time spans.

[1] Liu X., Xia Y., Liang Y., et al. LARGEST: A Benchmark Dataset for Large-Scale Traffic Forecasting. Advances in Neural Information Processing Systems, 2024, 36.

questions:
Refer to weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730766943531

thread_type:
official_review

<thread_object>
creator: reviewer_x92l

<content>
summary:
This paper proposes PN-Train, which detect and fine-tune pattern neurons to improve the forecasting performance on low-frequency patterns.

strengths:
- Improving forecasting performance on low-frequency events is interesting and challenging.
- The idea of pattern neuron is intuitive and reasonable.
- The proposed framework outperforms existing works.

weaknesses:
- The framework requires explicitly defined low-frequency patterns, i.e., holidays, and a dataset with low-frequency patterns is also required (as in Eq(3)). It is unclear whether the framework could handle implicit low-frequency patterns.
- In PNV, it will be better to test whether removing PNs leads to significant performance decreasing on high-frequency patterns, since PNs could be critical for all patterns rather than just low-frequency patterns.
- Minor errors. For example, in Fig.2, PNV seems to deactivate neurons which are not pattern neurons.

questions:
- In line2 Algorithm1, why do author filter out low-frequency samples?
- Can we allocate low-frequency patterns by categorizing neuron activations? 
- Is the framework able to handle two or more low-frequency patterns?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733239613970

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers and ACs,

We sincerely thank you for your thoughtful reviews and invaluable comments, which have greatly enhanced the quality of our submission. We are pleased that our additional clarifications and experiments have effectively addressed the concerns raised. Your insightful suggestions have been carefully incorporated into the revised version, with edits temporarily highlighted in **RED**.

We are delighted that the reviewers recognized the **novelty** of PN-Train (Reviewers XvJ7, 3uYm, T7Mg), which **combines neuron-level analysis with practical forecasting tasks** (Reviewer T7Mg) to **enhance accuracy for low-frequency events while maintaining high performance on high-frequency events** (Reviewers T7Mg, XvJ7). By **identifying and fine-tuning neurons associated with low-frequency patterns**, PN-Train **outperforms existing methods** (Reviewer x92L), supported by **solid empirical results** (Reviewers 3uYm, T7Mg) and **visualizations** (Reviewer T7Mg). The paper is also **well-written with sufficient technical details** (Reviewer 3uYm).

The reviewers also raised insightful and constructive concerns, and in response, we have made the following major revisions:
- **Clarifications (Reviewers XvJ7, T7Mg)**: We have emphasized the importance of low-frequency events in urban time series forecasting and clarified the critical role of accurate holiday forecasting in optimizing operations, reducing costs, and enhancing user experiences.

- **Related Work (Reviewers XvJ7, 3uYm, T7Mg)**: We have expanded the related work section to highlight the differences between PN-Train and prior studies on low-frequency events, continuous learning, fine-tuning, and neuron disentanglement.
- **Additional Experiments (Reviewers x92L, XvJ7, T7Mg)**: We have conducted further experiments on various low-frequency events, demonstrating that neuron-level fine-tuning significantly enhances predictions under different anomalous conditions (Appendix 6). We also validated PN-Train on additional datasets (Appendix 6).
- **Baselines (Reviewer 3uYm)**: We have added additional baseline methods to confirm PN-Train’s superior performance in forecasting under low-frequency events (Section 4, Appendix 6).
- **Code Release (Reviewer XvJ7)**: To ensure reproducibility, we have released our code, dataset, and checkpoints.
- **Visualizations (Reviewer T7Mg)**: We have included additional visualizations of pattern neurons to validate our assumptions and highlight our contributions (Appendix 7).

We are greatly encouraged by the positive feedback from all reviewers and delighted that the consensus places our paper above the threshold. We hope PN-Train provides a fresh perspective and inspires further exploration of neuron-level analysis in time series models.

Thank you once again for your invaluable suggestions, support, and organization.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a novel method consisting of a perturbation-based detector and a fine-tuning mechanism for time-series forecasting. The authors introduced neural importance - a novel interpretability mechanism for time-series forecasting task, applicable to both high-frequency and low-frequency events. The experiments on a series of time series forecasting tasks show the superiority of PN-Train, which enhances performance in low-frequency events while maintaining high performance in high-frequency events. 
The paper is also well-written.

There were concerns about identifications and details of high and low frequency patterns raised by some reviewers. There were only two datasets used. 

The authors have addressed the reviewers' concerns on the details on high and low frequency patterns, and have added experiments on a new dataset, in addition to the two datasets they have in the original version of the paper. These are now added to the appendix of the paper (from LargeST). I strongly suggest that the new experiments to be included in the main body of the paper, in addition to the two datasets, to illustrate the generality of the method.

Although this paper is borderline, given that almost all reviewers have raised their score, and all is unanimous with score 6, so this is still around borderline. In my view, the interpretability of time-series forecasting black box model here is the strongest contribution of this paper that's worth further discussions with others in the conference.