Trusted and Interactive Clustering for Time-Series Data

Authors: Chaoyu Gong, Yang You
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Time-series clustering, evidence theory, information fusion

Abstract:
Time-series clustering has gained abundant popularity and has been used in diverse scientific areas. However, few researchers take an information fusion perspective to combine information from the time and frequency domains to accomplish clustering, although these two domains offer distinct and complementary characteristics of time-series. Motivated by this issue, we propose a trusted and interactive model, which leverages evidence theory to combine time- and frequency-based clustering results produced by the corresponding contrastive learning module. After mathematizing clustering results from the two domains as mass functions, the uncertainty contained in these results can be quantified at the sample-specific level. The combined result thus promotes clustering reliability, and is optimized based on the pseudo-labels generated by k-means in an interactive learning paradigm. Both theoretical analysis and experimental results on 136 benchmark datasets validate the effectiveness of the proposed model in clustering performance. Extensive ablation experiments demonstrate the contribution of combining information from the time and frequency domains and using the interactive learning paradigm. The embeddings learned are also experimentally shown to perform well in other downstream tasks.

Decision: reject
Reviewer scores: 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1729113096753

thread_type:
official_review

<thread_object>
creator: reviewer_ujk1

<content>
summary:
The paper proposes an approach to cluster time series using a contrastive learning approach to learn meaningful representations.
To implement contrastive learning, time series augmentations based both on the time and frequency domain are used.
Rather than directly generating cluster assignments, the proposed model outputs parameters of a Dirichlet distribution. The latter are further processed to obtain the final cluster assignments.
A k-means algorithm is run from time to time during training to generate cluster assignments that guide the optimization.

strengths:
The idea of inserting contrastive learning within a probabilistic framework is interesting in principle. However, the proposed solution and implementation seem flawed.

weaknesses:
The most severe of my concerns is that one of the main contribution of the paper, which is the contrastive learning based both on time and frequency agumentation is not a novel contribution but is proposed in 

Zhang, Xiang, et al. "Self-supervised contrastive pre-training for time series via time-frequency consistency." Advances in Neural Information Processing Systems 35 (2022): 3988-4003.

The approach is basically the same and even some considerations/discussions are taken from that paper. Despite the paper from Zhang et al. is cited, the authors do not say that contrastive learning approach based on time and frequency augmentation comes from there. On the other hand, the authors make it look like is their own contribution.

My second concern is about the clarity of the presentation: the paper is rather confusing and difficult to follow. There are a lot of different concepts mixed together, often in a non-organic manner, and none of them seem particularly novel or well justified.

The whole part about the probabilistic framework, uncertainty quantification, and trustworthiness seems badly implemented. the authors say that their decoder produces parameters for a Dirichelet distribution. However, there are neither sampling procedures nor variational losses in the framework that justify the use of the probabilistic framework. It seems to me that the authors try to address this by introducing concepts such as "Subjective logic" and "Dempster’s rule", which I found rather confusing. In fact, I don't understand how much different it would be in practice to produce the cluster assignments directly from the FC decoder.

Finally, running a k-means while training a deep learning framework seems very unpractical and costly to implement, both in terms of time and space complexity. The fact that one should resort to such a solution hints that the proposed framework is unstable and unable to learn good cluster assignments.

questions:
- Why the work of Zhang et al. was not properly described, saying that they introduced a contrastive learning framework that ensures consistency between time and frequency augmentations?
- What performance would you get by producing directly the cluster assignments m_c from the FC decoder?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729685739897

thread_type:
official_review

<thread_object>
creator: reviewer_nlrp

<content>
summary:
The model named TIC (Trusted and Interactive Clustering) tries to develop a methodology for clustering by fusing information from two distinct views: time and frequency domain. With strong backing from evidence theory and mass functions, the method learns robust embeddings from each domain using the respective contrastive loss. The learned embeddings are combined and an additional interactive loss using K-Means is additionally used to improve the clustering performance.

strengths:
Introduction :
The second and third motivations of dynamically weighing the information from multiple views and combining them respectively for improving clustering results are novel directions.
Challenges associated with each motivation and the contributions from the work are clearly described.
Architecture diagram (Figure 1):
Well annotated and gives the overall idea of the methodology
Preliminaries :
Written with great clarity such that it is comprehensible to those who are not aware of evidence theory and mass function;
Focusing well on how they are fine-tuned towards the clustering and integration objectives.
Methodology:
Existing concepts, definitions, and theorems used are well-referenced, except in the "interactive learning module"
Notations and Problem Statement are very clear
The motivation for the design of the augmentation bank is well-written
Uncertainty quantification is well established with strong theoretical background
It is theoretically proven why the combined results are "trusted"
Figure 3 beautifully illustrates the "class collision" issue
Experiments :
Experimental evaluation on a large number of datasets against a variety of baselines for tasks of clustering and above (classification & anomaly detection) proves the effectiveness of the method
Statistical significance tests are used to prove that the proposed method is statistically different from other baselines
Excellent Ablation studies prove the significance of each loss component towards the model performance; A very detailed ablation study in the appendix for analyzing the contribution of each component further instantiates that the each of them is critical towards the model development
Figure 4b clearly illustrates how interactive loss contributes to better clustering
Figure 5 very well demonstrates the need to avoid frequent updates using k-Means and how the best frequency is chosen; The Subsequent explanation in "Hyperparameter sensitivity of t" is also well-written
Appendix :
A thorough literature study has been done as evident from Appendix A.1
Proofs for the propositions further confirm the "trusted" claim of the method
Baselines and datasets are well-described
Showing the effect of various perturbations in Table 9 conveys the idea of choosing the necessary techniques to alter the signals

weaknesses:
Introduction :
It is mentioned that "there have been few clustering works which already attempted fusing time and frequency information"; Hence I feel it is not appropriate to give the first motivation for this work as "incorporating frequency information to enhance the ability to detect clusters."; It would have been more appropriate if it was mentioned that attempting this direction motivates other goals of weighing the contributions from each domain and incorporating them for clustering, which are novel.
Architecture diagram :
I guess the update decision function from k-Means indicates the epoch at which pseudo-labels are regenerated. If so, a better representation of the same is desirable.
Preliminaries :
Is the variable "m" missing from the definition "A mass function, .. is defined as "?
What is meant by "vacuous" in the phrase "vacuous mass function"?
Methodology :
It can be the case that the perturbations from the augmentation bank could be minor. Then the perturbed signals will not be contrastive enough and their cosine similarity could be high. Hence the claim that the augmentations used in both the modules are "contrastive" is weak, unless there is a strong reasoning. A more detailed explanation is desirable on how the perturbations are done: whether it is the same for the entire dataset; during training or random throughout
Contrastive loss functions (Equations 3,4):
A major basis for the work is that two distinct views for the time and frequency domain are formed. Hence is it justified to use the same value for the hyperparameter "τ" in both cases?
In the numerator it is shown that the similarity function is applied to embeddings of positive pairs; whereas the denominator uses "HF (x_j)" instead of the corresponding embedding g_j while applying the similarity function to negative pairs. What is the rationale behind it given that both are equivalent?
What is the reasoning behind using indicator functions in the denominator to exclude similarities between positive pairs? Because the loss function is in similar lines that of sigmoid and also conventional normalization techniques where such an indicator function is not used. Is it the same as the "class collision issue" mentioned in Remark 2? Still, it is not intuitive why the indicator function in contrastive loss can avoid class collision in interactive loss.
The concept of pseudo-labels is not properly explained or referenced; without which the role of K-Means looks incomplete
Equation 11 of the Interactive learning module is under-referenced where techniques like digamma function, and Dirichlet strength are used; Also what is the background with which conventional cross-entropy is modified? Theoretical backing for deriving equations 12,13 seem missing
Though Figure 3 gives a nice illustration of the "class collision issue"; I feel the term is misleading; It is mentioned that the issue represents the scenario where all clusters consist of only one sample; How does it convey the term "collision"?
The claim that "our TIC method can also accommodate multi-variate time-series" is strong and important in time-series analysis. However, it is not mentioned how the method presently done for univariate data can be easily extended to the multi-variate case.
Experiments :
"Implementation Details" :
 it is written that the "softmax layer is replaced with the RELU to ensure that the network outputs are non-negative"; But is the statement valid, because the output of softmax is also strictly non-negative?
 It is mentioned that "2-norm penalty coefficient of 0.0005" is used. However, there is no mention of the 2-norm regularization being used in the methodology and the purpose it serves.
"Ablation Study" :
It is valid that removing the interactive loss component leads to performance degradation due to "losing the basic clustering information from K-Means". But it is not clear why it gives the worst performance. The reasoning for the connection between "class collision issue" and "positive pairs" is still vague.
Figure 4 Illustration :
I feel there is a typo as one of the embedding notations should include "Freq" instead of "Time".
It is true that the time and frequency embeddings of the same sample should come close in the embedding space, But it is not clear how the model affects the cosine distance with and without their combination in the illustration though the description was later found in the appendix. It would be appropriate if the appendix description is included in the figure illustration itself.
"Hyperparameter sensitivity": Usually sensitivity experiments are used to demonstrate the model's robustness in the context of varying values of a critical hyperparameter. Here there are many hypermeters used in the model out of which only "t" is chosen; Also the demonstration is regarding how the best value is chosen and not on model robustness;
I think there is a typo in "Notions and problem formulation". Is it "Notations and .."?
Appendix :
Time Series Anomaly Detection baselines are not recent, state-of-the-art such as MTGFlow, TranAD, and GDN and are not evaluated using benchmark datasets such as SWaT, WADI, SMAP; So the results, though favorable to the proposed method cannot be considered as a breakthrough for this downstream task.
Figure 7 shows that the embeddings form distinct clusters towards the last epoch. But a strange observation is that the cluster distinction is not apparent till epoch 70 i.e. the first 2 plots essentially show the samples being distributed among two clusters; 3rd plot (epoch 70) shows 3 clusters and by the last plot (epoch 99) they become well distinct. Is there any justification for the same?
What is the unit of measure of the running time reported in Table 5?

questions:
As above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730228488639

thread_type:
official_review

<thread_object>
creator: reviewer_whwj

<content>
summary:
The authors propose a novel time series clustering method rooted in evidence theory. They approach relies on two embeddings, one in the time and one in the frequency domain. The resulting embeddings are used to parameterize a Dirichlet distribution which is then used to compute a mass function over the inputs (once for the time and once for the frequency domain). The mass functions are combined in an interactive loss which considers pseudo labels from a k-means clustering on the time series embeddings making sure to align the D. distribution with the k-means results. The authors provide evidence that their method results in superior clustering performance and good performance in secondary tasks such as anomaly detection and time series classification. Lastly, an ablation study is performed which sheds light on the importance of the involved loss terms.

strengths:
The authors provide an exhaustive study and evaluation of the proposed method. They compare the method to various competing methods and dive deep into the mechanisms of their approach (ablation study). Additionally, the method consists of various moving parts which makes it challenging to describe each component in a comprehensive manner. However, the authors did a great job in doing so. While the notation feel overloaded at times, the overall flow is quite good. Provided figures support this flow nicely.

weaknesses:
Statistical significance: The authors indicate statistical significance in Table 2. However, it seems they are not correcting for multiple hypotheses. Given the number of comparison methods and data set, I'd recommend adding a Critical Difference Plot[1] to underline statistical significance.
Algorithm efficiency: It is unclear how efficient the algorithm is. It would be great to see for each data set how long training takes and also how long training/inference of the competing methods take (e.g., adding a respective table).
Preliminaries: The background on Evidence Theory seems rather short. In particular, it is hard to get an intuition for the relationship between it and clustering. In particular, I am not sure what "trusted" means in this context? It is just a higher probability mass we're assigning to a certain sample?

1. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7, 1-30.

questions:
1. You create a set of augmentations for an individual time series, why do you just pick one for training?
2. Can you confirm that all data sets have a train/val/test split and they were used accordingly for hyperparameter search?
3. Why not write $g_j^{Time}$ in Eq. 3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731713244207

thread_type:
official_comment

<thread_object>
creator: reviewer_whwj

<content>
title:
Updating Score

comment:
After considering the work by Zhang et al., 2022 ("Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency"), I lower my score to 3: "reject, not good enough". While I still see some contributions beyond this paper, I agree with reviewer UJk1 that the way this work is referenced is insufficient and intransparent.
</content>
</thread_object>

</thread>


Metareview: