Spatio-temporal Heterogeneous Federated Learning for Time Series Classification with Multi-view Orthogonal Training

Authors: haishuai  wang
xiang  zhang
jiajun  bu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 09:33:43

Abstract: Federated learning (FL) is undergoing significant traction due to its ability to perform privacy-preserving training on decentralized data. In this work, we focus on sensitive time series data collected by distributed sensors in real-world applications. However, time series data introduce the challenge of dual spatial-temporal feature skew due to their dynamic changes across domains and time, differing from computer vision. This key challenge includes inter-client spatial feature skew caused by heterogeneous sensor collection and intra-client temporal feature skew caused by dynamics in time series distribution. We follow the framework of Personalized Federated Learning (pFL) to handle dual feature drifts to enhance the capabilities of customized local models. Therefore, in this paper, we propose a method FedST to solve key challenges through orthogonal feature decoupling and regularization in both training and testing stages. During training, we collaborate time view and frequency view of time series data to enrich the mutual information and adopt orthogonal projection to disentangle and align the shared and personalized features between views, and between clients. During testing,  we apply prototype-based predictions and model-based predictions to achieve model consistency based on shared features. Extensive experiments on multiple real-world classification datasets and multimodal time series datasets show our method consistently outperforms state-of-the-art baselines with clear advantages.

Main paper: 
 related work 2.1 federated learning for time series: 
 Recently, many deep learning methods have been adopted to process distributed sensitive time series data monitored by private sensors. Federated learning, as a significant decentralized deep learning paradigm, has been widely used to analyze these time series data. In [39] an attention mechanism-based convolutional neural network-long short-term memory (AMCNN-LSTM) model is proposed for industrial anomaly detection. FedTSC [36] presents a two-level secure mechanism for time series classification with secure feature extraction and secure model training. They emphasize security and interpretability. EFDLS [65] focuses on a multitask time series classification setting and proposes a knowledge distillation-based framework. MetePFL [7] proposes a prompt learning method for weather forecasting with foundation models. However, it holds a strong hypothesis that each client can fine-tune a foundation model, which is not universally practical in mobile computing scenarios for time series. FLAMES2Graph [69] also targets the interpretability of time series classification. They adopt a Multivariate Highly Activated Period (MHAP) evolution graph aggregation for better interpretable performance. It is notable that these works mostly focus on the general performance of time series classification or forecasting, ignoring the challenges of the time series data modality itself, such as the time and frequency feature and non-stationarity that this paper focuses on. Also, they did not consider the challenge resulting from the implementation of federated learning but adopted time series as a general downstream task.Feature-skewed federated learning focuses on improving the customized capability of clients' local models under diverse feature spaces or domains held by clients. As pFL methods for label skew, model decoupling [6, 20] and prototypical methods [55, 56] are also widely adopted for feature-skewed FL. FedBN [33] preserves the batch normalization layers, aggregating other parameters to overcome feature concept shifts among clients. AlignFed [78] performs a reverse aggregation strategy of decoupled personalized federated learning for label skew. They craft personalized feature extractors and generalized classifiers. FedPCL [56] adds contrastive loss as an incremental approach of FedProto [55] . The pre-trained parameters are also considered to improve the initial performance of FL. DFL [42] aims to disentangle shared and private features by multi-branch feature extractor and applies mutual information loss to optimize the training. Differing from model decoupling, FedFA [77] and FedRDN [67] focus on the data itself, applying data augmentation to enrich data representations. With expanded feature representation, models are more robust to different underlying distributions across clients. However, previous works mainly emphasize the distribution shifts in the computer vision context. The spatial feature heterogeneity of time series data is under-explored.Test-time adaptation, which has been widely investigated in the fields of computer vision [17, 45, 52, 57] , aims to improve model performance on unknown target domain data during the test time. FedTHE [20] as a pioneer work, explore the ID and out-of-distribution (OOD) situations that clients will encounter during federated deployment. They decouple the original model into a global head and personalized head and perform robust cross-entropy. They also enforce feature space alignment via prototypes. FedICON [54] adopts contrastive learning to FedProto [55] , making the predictions of different augments of a sample more stable to achieve robust updates during test time. These works concentrate on the domain shifts and common corruptions of image processing, but we extend to the unique distribution drifts of time series.Feature distributions of time series in the real world are often dynamic and change over time, especially for scenarios such as weather forecasting and clinical detection. Many works have focused on it. Non-stationary [40] proposed a new transformer that contains Series Stationarization and De-stationary Attention modules. Dish-TS [11] introduces a dual-conet framework to learn the distribution of input and output spaces separately, naturally capturing the distribution differences between the two spaces. OneNet [76] uses reinforcement learning to train two networks to balance temporal correlation and cross-variable dependence. RevIN [27] normalizes the input time series to fix its distribution in terms of mean and variance, then returns the output to the original distribution. [26] first propose test-time adaption in unsupervised time series anomaly detection. They utilize trend estimates and normal instances based on the model prediction itself for model updates. EQUATION)where 𝛿 is the threshold and 𝑑 (•) is the distance metric to evaluate the distributions' divergence, such as KL divergence.Optimization objective. We denote 𝜽 as the basic model architecture with two-part: feature extractor 𝑓 parameterized by 𝜽 𝑓 and prediction head 𝑐 parameterized by 𝜽 𝑐𝑙𝑠 . The typical FL considers a generalized optimization problem, to minimize the empirical risks:EQUATIONIn contrast to traditional generalized FL, pFL aims to improve every client's personalized models. Clients pursue adapting to each local data distribution collaboratively. The client-wise personalized learning objective can be formulated as:EQUATIONwhere clients' personalized objectives can be computed by loss function and reach a better performance to tackle the cross-client feature skew and test time feature skew. 
time series frequency features: 
 The Discrete Fourier Transform (DFT) is often applied to transform signals in the time domain into the frequency domain. With different domains, the cognitive perspective of the same thing also changes. From a new view of the frequency domain, more representations can be revealed. For each dimension of a time series sample 𝑥, the DFT process can be denoted as:EQUATIONwhere 𝑋 [𝑘] is the 𝑘-th frequency component in the frequency domain, 𝑥 [𝑡] is the 𝑡-th data point in the time series, 𝑇 is the time step (total number of data points), 𝑘 is the current frequency index, 𝑖 is the imaginary unit. 
method 4.1 overview: 
 In this section, we describe the proposed novel federated learning method: FedST. We consider two main processes: federated training and federated inference. In the training phase, to tackle the feature distribution shifts across clients, we adopt an orthogonal factorization way. By projecting features to orthogonal subspace, we align the shared feature between time and frequency view, and local and global representations. The communication between clients and the server includes spatial model aggregation and global prototype aggregation. Considering the feature distribution shift in the test time of time series data, we craft an adaption objective to fine-tune the representation and make prediction more confident. Figure 4 visualizes the training procedure of both phases.Time-Frequency Feature Encoder. as a global knowledge, written as:EQUATIONwhere the set of global prototypes can be obtained as {𝒑 𝑇 𝑐 } 𝑐 ∈ C , and frequency prototype {𝒑 𝐹 𝑐 } 𝑐 ∈ C can also be derived. Orthogonal Space Factorization. We first recall the definition of orthogonality, given two vectors 𝑣 and 𝑢:EQUATIONwhere the product of two vectors is zero, then we derive they are orthogonal. Given a vector space V, we can define two orthogonal subspaces V 𝑠 and V 𝑝 . Thus, any vector in space can be disentangled into two orthogonal subspaces via projection matrixes 𝑃 𝑖 . Inspired by previous literature [4, 12, 38] , for shared subspace and personalized subspace, we ensure the orthogonality of the projection matrix:𝑃 ⊤ 𝑠 𝑃 𝑠 = 𝐼, 𝑃 ⊤ 𝑝 𝑃 𝑝 = 𝐼, 𝑃 ⊤ 𝑠 𝑃 𝑝 = 0, 𝑃 ⊤ 𝑝 𝑃 𝑠 = 0.Given a intermediate layer 𝑙 of feature extractor 𝑓 𝑙 with 𝐿 hidden layers, the optimization objective of gradients can be decomposed as:EQUATIONwhere 𝐷 𝑘 denotes a diagonal matrix that embodies the Jacobian matrix associated with the pointwise nonlinearity. 𝑃 𝑖 refers to the projection matrix, 𝜙 𝑃 denotes the product of 𝑃 𝑖 and 𝑙-layer parameters. Due to the orthogonal gradient updates, we can project features into orthogonal subspaces and disentangle them. After receiving the output of time-frequency encode, each view embedding 𝒉 𝑇 𝑖 , 𝒉 𝐹 𝑖 are projected by a non-linear multilayer perceptron (MLP) projector, into two orthogonal embeddings: personalized feature embeddings 𝒉 𝑇 ,𝑝 𝑖 , 𝒉 𝐹,𝑝 𝑖 and shared feature embeddings 𝒉 𝑇 ,𝑠 𝑖 , 𝒉 𝐹,𝑠 𝑖 . View-wise Orthogonality Loss. Inspired by [73, 75] , we convey wealth to mining the mutual information among different time series views and enhance representation training by pulling in shared features and pushing away personalized features. In the orthogonal latent subspace, we adopt a contrastive way to achieve this objective, written as: L 𝑇 𝑣𝑜𝑙 = - 1 |D 𝑘 | ∑︁ 𝑥 𝑖 ∈ D 𝑘 log exp(|𝒉 𝑇 ,𝑠 𝑖 • 𝒉 𝐹,𝑠 𝑖 |/𝜏) 𝑥 𝑗 ∈ B exp(|𝒉 𝑇 ,𝑠 𝑖 • 𝒉 𝐹,𝑠 𝑗 |/𝜏)where B denotes a batch in the learning process, 𝜏 refers to a temperature parameter to control the tolerance. The corresponding contrastive loss for frequency view L 𝐹 𝑣𝑜𝑙 can be obtained in the same way. Within a single view, we also enforce the projection in orthogonal subspace to minimize the angular between shared feature and personalized feature. We define the private orthogonal loss for an unimodal as:EQUATIONwhere we are able to formulate frequency view loss L 𝐹 𝑝𝑜𝑙 as well. In this way, orthogonality between these two features can be strengthened. The local encoder and projector possess the capacity to acquire invariances and variances of shared and personalized features.Domain-wise Orthogonality Loss. As discovered in [12, 54, 70] , the domain gap between clients is vital for feature-skewed FL. Learning to decouple the global feature and personalized feature remains to be investigated. In this work, we continue to adopt orthogonal subspace projection for global knowledge and local embedding. We utilize global prototypes as global knowledge and project them into orthogonal subspace as 𝑷 𝑇 ,𝑠 𝑐 and 𝑷 𝑇 ,𝑝 𝑐 , as well as frequency view 𝑷 𝐹,𝑠 𝑐 , 𝑷 𝐹,𝑝 𝑐 . We facilitate each sample to be invariant in the shared feature space, and orthogonal in the personalized feature space. We perform cross-client orthogonality constraint with the loss function as below:EQUATIONWe summarize the total orthogonal loss for each view {𝑇 , 𝐹 } as:EQUATIONThe general optimization objective cross-entropy loss to minimise empirical loss is formulated as follows:EQUATION)Finally, the overall loss function is denoted as:EQUATIONVanilla Fedavg [44] aggregates clients' models based on local dataset size as𝜽 (𝑡 +1) = 𝑘 ∈ [𝐾 ] | D 𝑘 | | D | 𝜽 (𝑡 )𝑘 . Averaged aggregation strategy fits the goal of generalization, not for personalization. For labelskewed pFL, model decoupling has been widely adopted, where the classifier is regarded as the source of model shifts. So, a shared feature extractor with a personalized classifier stands out as a popular structure [6, 8, 20, 70] . In contrast to label skew, a reversed aggregation is commonly implemented with a personalized feature extractor with a shared classifier [12, 42, 54, 78] for feature skew. In this work, we aggregate the classifier 𝜽 𝑐𝑙𝑠 and preserve the timefrequency encoder 𝜽 {𝑇 ,𝐹 } 𝑓 and projector to fit the feature-skewed personalization task, as previous work 𝜽(𝑡 +1) 𝑐𝑙𝑠 = 𝑘 ∈ [𝐾 ] | D 𝑘 | | D | 𝜽 (𝑡 ) 𝑐𝑙𝑠,𝑘 .In addition, clients also upload local time-frequency prototypes {𝒑 {𝑇 ,𝐹 } 𝑐 } 𝐶 𝑐 for global knowledge fusion [55, 56, 78] , denoted as: EQUATIONIn the federated inference phase, training data is no longer approachable. The feature distributions of coming test samples show dynamic shifts due to the non-stationary time series. In this case, gradually updating the trained model with unsupervised data can help the model adapt to the changing feature concept. Prototypical Orthogonal Adaption. Given an unlabeled test sample 𝑥 𝑖 during adaption, we can obtain the projected feature 𝒉 𝑠 𝑖 , 𝒉 𝑝 𝑖 as a shared feature and personal feature for both time and frequency view. We maintain the pre-trained global prototypes {𝒑 𝑐 } 𝑐 ∈ C as a prototypical classifier. Instead of computing cosine or Euclidean similarity to get the nearest prototype as a pseudo label, we produce the pseudo label via a minimized angler of the shared feature in the orthogonal subspace:EQUATIONwhere 𝜎 (•) is the normalized prediction by a softmax function. We align the predictions of the prototype (in the orthogonal subspace) with the predictions of the model to ensure that the extracted feature distribution of the current test data should be consistent with the feature distribution of all previous data of the same class. Thus, the loss to maintain consistency and align representation is defined as:EQUATIONwhere we omit the notation for time and frequency views, the total loss is calculated by the sum of L 𝑇 𝑝𝑜𝑎 and L 𝐹 𝑝𝑜𝑎 . The process mentioned above is undergoing for both views. By minimizing the entropy, the model is able to generate more accurate representations of samples in the target domain. We then update the prototypes with new data, gradually fitting the dynamic distributions. 
experiments 5.1 experimental setup: 
 Datasets. We use five real-world time-series datasets for evaluation, which are commonly used in domain adaption and other feature drifts works for time series [15, 37, 47] , involving human activity recognition HAR [13] , HHAR [51] ,WISDM [28] and Electroencephalogram (EEG) signal data: Sleep-EDF [14] ,Epilepsy [2] . The summary is given in Table 1 .Baselines. We take four groups of methods as baselines. (1) typical FL and pFL for label distribution shifts: FedAvg [44] , Fed-Prox [32] , FedDyn [1], FedRoD [6] , FedProto [55] (2) pFL for feature distribution shifts: FedBN [33] , AlignFed [78] , FedFA [77] (3) pFL for test time adaption: FedTHE [20] , FedICON [54] (4) FL tailored for time series: FLAMES2Graph [69] .Data distributions. We consider dual feature distribution shifts.• Spatial feature heterogeneity: For spatial feature heterogeneity, we adopt the general feature-skewed setting in pFL. We treat each entity in time series data as a domain, the [3, 15, 46] . We mix the test set across clients. Each client holds a test set from another domain.Implementation Details. For different dataset, we set the number of clients 𝐾 based on # of subjects (entities) in Table1, along with doubled. However, in Epilepsy, the default entity size is 500, which is not realistic for a common federated learning scenario. So we randomly extract 100 and 200 entities for experiments. Each experiment is run for 100 communication rounds. Because our main contribution is not encoder and backbone model design, for fairness, we follow the existing work [15, 50, 69] , using the CNN network to train time series data for five epochs in each round. We set the batch size to 128. We use the SGD optimizer with a momentum of 0.5 and a learning rate of 0.01.Results on spatial heterogeneity and natural temporal heterogeneity. We follow the previous works, using an averaged personalized accuracy to evaluate the performance. Table 2 shows the performance of each baseline. We find that FedST achieves the best results compared to the federated learning approaches in all benchmarks. Concretely, FedST outperforms at least 2.37% accuracy improvement on the Epilepsy dataset, and obtains 14.53% accuracy improvement on Sleep-EDF, which is the largest dataset. It indicates the superiority of our model in dealing with large-scale datasets. Regarding baseline performance, we find that general pFL methods that keep personalized models locally, such as FedProx and FedProto, will have a significant impact on their performance. Prediction head decoupling methods like FedRoD and FedTHE, can usually achieve better results than FedAvg, but the improvement is limited. Flame2Graph, tailored for time series data, but does not consider spatial-temporal feature skew, our method still obtains the advantage.Results on spatial heterogeneity and synthetic temporal heterogeneity. To further evaluate the robustness under feature distribution shifts in the test time, we craft the synthetic temporal heterogeneity of feature distributions in Table 4 . Facing the unseen domain feature at the inference face, in terms of clients' average accuracy, FedST has much better FL adaptability than stateof-the-art pFL methods in a test-time adaption federated learning setting. Compared with FL methods equipped with test-time adaption, FedTHE and FedICON, FedST improves prediction accuracy by 12.05% and 11.33% on HAR with multiple clients. We can find that our method still performs well in various baselines, as the performance of other baselines has dropped significantly due to unknown feature drifts during testing.To validate the effectiveness of each training strategy in FedST, we compare it with several ablations. Mainly, our ablations include: removing whole orthogonal training module L 𝑜𝑝𝑙 , removing viewwise orthogonal training L 𝑣𝑜𝑙 and L 𝑝𝑜𝑙 , removing domain-wise orthogonal training L 𝑑𝑜𝑙 , removing test-time adaption L 𝑝𝑜𝑎 and only considering one view selecting from time and frequency. The results shown in Table 3 turn out that each part of our approach plays an important role and they work together to achieve better performance. Notably, we notice that orthogonal training has the most significant impact on the improvement. Without orthogonal projections, there is a huge drop in performance of 8.82% to 16.88%. We also find that domain-wise orthogonal constraint and view-wise orthogonal constraint equally contribute to the overall performance, where a minor advantage of domain-wise loss indicates the necessity of decoupling global and local knowledge.In our method, the hyperparameter 𝜆, 𝜆 1 , 𝜆 2 are used to control the strength of the different orthogonal regularization loss. 𝜆 controls the overall orthogonality constraint, used to balance with the model's predicted cross-entropy loss. As shown in Figure 5 , the larger the value of 𝜆, the more the model focuses on the orthogonality of shared features and personalized features. Through experiments, we find that keeping around 1 for regularization is more appropriate. The 𝜆 1 , 𝜆 2 are used to control the orthogonality between views and the orthogonality between clients. We found that the orthogonality of features between clients is more sensitive and plays a dominant role in feature-skewed federated learning. We also studied the contrast temperature of contrastive learning 𝜏.Higher temperature values will increase the output of the model and be more diverse, and the steady-state model also requires a temperature of around 1.Our method leverages time and frequency domain information to enhance training. In real applications, many time series data are multi-modal, rather than having only one modality in the original time domain. We extend our method to the multi-modal domain.When we calculate features or losses in the time and frequency domains, we add other modalities in the same way. We compare our ACIDS PAMAP2 method with baselines on two commonly used multi-modal datasets ACIDS and PAMAP2 [49] . From the results shown in Figure 6 , we find that it consistently outperforms state-of-the-art baselines with significant advantages, ranging from 3% to 6%, and proved to be valuable in the multi-modal time series domain. 
conclusion: 
 In this paper, we introduce FedST a federated learning framework for time series that addresses both inter-client and intra-client feature shifts. FedST considers both time and frequency view features. To tackle spatial feature heterogeneity, we apply an orthogonal training paradigm to disentangle the features between views and clients. For temporal feature heterogeneity caused by nonstationarity of time series, we update the model with prototypes in orthogonal subspace to enhance prediction confidence. Experimental results on five datasets demonstrate FedST effectiveness. 
federated training phase: 
 Server1 IntroductionNowadays, machine learning (ML) is undergoing a paradigm shift from cloud data centers to distributed edges [19, 29] . With the development of mobile Internet of Things (IoT) [61] and multimedia computing platforms [16, 24, 48, 72] , a large amount of valuable time series data is recorded by distributed smart devices or entities. Time series data refers to a series of data points or observations arranged in chronological order. It has been widely used in anomaly detection [23, 26] , activity recognition [13, 74] , weather forecasting [7] , healthcare diagnosis [59, 66] , recommendation system [21, 22, 68] , and emotion analysis [30] . One of the major research topics is time series classification (TSC) by deep neural networks (DNN) [41, 58] . The data is based on WIDSM [28] However, due to the sensitivity and privacy issues of users' data, the model training in a centralized data way faces challenges in practice. Traditional centralized machine learning methods require all data to be concentrated on a central server for training, which not only increases the overhead of data transmission but also raises potential risks of privacy leaks.To tackle the challenges of data confidentiality and communication efficiency, federated learning (FL) [1, 25, 32, 44, 60, 62, 71] emerged as a promising distributed training paradigm. Federated learning can collaboratively train from massive multi-source data without exchanging the original data, retaining data ownership [34] while reducing the communication burden [43] . Specifically, edge devices preserve their private data locally, and federated learning mainly realizes training a robust model by aggregation and distribution of local models through multiple rounds of communication.Although FL shows promising prospects in data collaboration, it still faces the challenge of data heterogeneity in practice [44] . Time series data collected by sensors often come from different individuals and institutions, where the client's local data is non-independent and equally distributed (non-IID).Non-IID data can lead to model drifts and catastrophic forgetting of global knowledge in federated learning, which further leads to model performance degradation and slow convergence. Data heterogeneity in federated learning can be categorized into label space heterogeneity and feature space heterogeneity. Much work has focused on the problem of non-IID in the label space heterogeneity among clients [1, 6, 31, 32, 35] . Compared with the more obvious label distribution deviation caused by class imbalance, the feature skew data in federated learning is more hidden and more difficult to overcome [5, 33, 77, 78] .Although general feature distribution skew has been explored in FL, it becomes more challenging when applied to time series data beyond image processing [5, 33, 77, 78] , i.e., the data exhibit dual feature heterogeneity in both spatial and temporal perspectives.Challenge 1: Spatial feature heterogeneity. In real-world deployment, the time series data collection methods of different devices and individuals vary greatly, resulting in cross-client feature skew [10] . As shown in Figure 2 , two samples share the label space but exhibit heterogeneity in feature space. In the FL context, each client may become a domain, and the complex dependence of time steps makes it difficult to extract invariant features [3] , hindering the cross-domain generalization ability of FL models. From another perspective, time series have two perspectives: time view and frequency view. Domain shift may occur in both time view and frequency view. It is also possible to shift the time feature while leaving the frequency feature relatively unchanged, as shown in Figure 3 . Explicitly modelling time and frequency domain features becomes more challenging.Challenge 2: Temporal feature heterogeneity. The nonstationarity of time series data has been an under-addressed challenge [11, 27, 40, 76] . Since the time series data is observation data discretely sampled by sensors at a certain recording frequency, its future data (horizons) may show different feature distributions from the past observations (look-backs) [11, 27] . In the FL, it reflects in the distribution shifts between training sample features and testing sample features [18, 45, 52, 57] . The feature drifts of test samples make the original model unable to perform ideally on the changing dynamics. Accessing the original data will cause huge computational costs for complete retraining of the model. How to realize test time adaption for robust FL deployment is under-explored [20, 54] .Encountering the dual challenges of spatial-temporal heterogeneous federated learning, we need to introduce a training scheme: personalized federated learning (pFL) [6, 8, 9, 31, 32, 55, 63, 64] which is tightly associated with dual feature-skewed FL and test time adaption. Differing from vanilla generalized FL, pFL aims to enhance the capability of clients' personalized models during the local test phase. From the perspective of spatial feature heterogeneity, a well-collaborated generic model finally needs to fit each local test set's domain. Thus, previous works about featureskewed FL were often presented as a uniform challenge along with pFL [5, 33, 42, 54, 56, 78] . Besides, some pFL works mainly focus on label distribution shifts [6, 8, 31, 32, 53] . From the point of view of temporal feature heterogeneity, clients' time series data shows concept drifts during test time. The optimization goal of pFL is that the trained model performs better on each client's test samples, which coincides with the test-time adaption objective of fine-tuning the model online to improve test-time performance.To overcome these challenges, in this paper, we propose FedST, a two-stage updated federated learning framework to solve spatialtemporal feature heterogeneity. The framework mainly consists of two modules: orthogonal training and consistency testing. In the training phase, we propose an orthogonal decoupling of cross-view and cross-client representations to extract and utilize shared and private features. In the testing phase, we align the predictions of the prototype and the predictions of the model based on the orthogonal subspace, minimizing uncertainty and fine-tuning representations.Specifically, for the feature-skewed FL, previous work has discovered the importance of decoupling personalized knowledge and global knowledge. Multi-model methods have been used to disentangle different features in images [12, 42, 70] for promoting local personalization. For the distributed feature shifts of time series, we focus on not only the decomposition of cross-domain features between clients but also the decomposition of view-wise features of time series. Time series data can be converted into multimodal information, and supplementary information from different views can be used to effectively analyze time series data [73, 75] . Multiview learning can extract extra information [38, 73, 75] Corresponding polar coordinates of frequency features show that frequency features are more domain-invariant than time features in some cases. The data is from WISDM [28] .series than single-view in the latent space, as exhibited in Figure 3 . We hope that the optimization objective is to improve the orthogonal between shared features and private features of the same view, the private features between different views, and private features between clients. Therefore, we can capture the private features and shared features between views, and the personalized knowledge and global knowledge between clients through the orthogonal space of factorization. We adopt contrastive learning and prototype-based methods to achieve our cross-client and cross-modal orthogonal training. In the federated inference phase, the visited model learned a feature representation ability that fits the local domain with global knowledge. However, since the feature concept of the time series drifts over time, the model may not be able to generate accurate representations for newly exposed samples. We leverage aligning the predictions of the prototype and the predictions of the model based on orthogonal subspaces to maintain model consistency and representation quality. We summarize our contributions as follows:• We consider a unique spatial-temporal feature skew challenge tailored for time series in the real-world application of federated learning. • We propose FedST which leverages orthogonal latent space to disentangle client-level and view-level features, and adopts prototype-based consistency for test time adaption. • Extensive experimental results, FedST has achieved superior empirical performance better than SOTA solutions for pFL, feature-skew FL and FL for time-series. 
 
 Appendix:  


Decision: The paper proposes FedST, a two-stage federated learning framework that effectively tackles spatial-temporal feature heterogeneity in time series data. It demonstrates superior experimental performance but lacks discussion on privacy protection and communication efficiency, making it a borderline accept.

Reviews: review: paper_summary: paper_summary: The paper presents FedST, a federated learning framework specifically designed to address the challenges of spatio-temporal feature heterogeneity in time series classification. The authors focus on the dual spatial-temporal feature skew that arises due to the dynamic nature of time series data collected by distributed sensors. The proposed FedST, leverages both time and frequency views of time series data and employs an orthogonal training approach to disentangle and align shared and personalized features across clients and views.
strength_weakness: strength_weakness: 1. The posposed FedST introduces a unique method to handle the complexity of time series data in FL by using a dual-view (time and frequency) orthogonal training mechanism, which tackles both spatial and temporal feature heterogeneity, which are critical issues in real-world applications.

2. This paper have conducted extensive experiments on multiple real-world classification datasets and multimodal time series datasets. The consistent outperformance of FedST against state-of-the-art baselines demonstrates the effectiveness of the proposed method.
limitations: limitations: 1. The orthogonal training approach, while effective, may introduce additional complexity into the FL process. The paper could benefit from a discussion on the computational overhead and scalability of the proposed method.

2.  FedST's scalability to a large number of clients and high-dimensional time series data is not thoroughly discussed. The overhead of maintaining orthogonality across a massive distributed system may become a bottleneck, potentially limiting the framework's effectiveness in extremely large federated settings.
paper_summary: paper_summary: Despite the general exploration of feature distribution skew in Federated Learning (FL), applying it to time series data becomes more challenging due to the dual feature heterogeneity exhibited in both spatial and temporal dimensions. Spatial feature heterogeneity refers to the cross-client feature skew caused by differences in data collection methods among different devices and individuals, while temporal feature heterogeneity refers to the non-stationarity of time series data, meaning that the feature distribution of future data may differ from that of past observed data.

To address these challenges, this paper proposes FedST, a two-stage federated learning framework designed to tackle spatial-temporal feature heterogeneity. The framework primarily consists of two modules: orthogonal training and consistency testing. During the training phase, the paper introduces orthogonal decoupling of cross-view and cross-client representations to extract and utilize both shared and private features. In the testing phase, the paper minimizes uncertainty and fine-tunes representations by aligning prototype predictions with model predictions based on orthogonal subspaces. Through this approach, FedST effectively handles feature skew in time series data within federated learning applications.

In summary, the contributions of this paper include:

1.Introducing the unique challenge of spatial-temporal feature skew in time series data for real-world federated learning applications.

2.Proposing the FedST framework, which leverages orthogonal latent space to decouple client-level and view-level features and employs a prototype-based consistency approach for test-time adaptation.

3.Demonstrating through extensive experimental results that FedST outperforms existing state-of-the-art solutions in personalized federated learning, feature-skewed FL, and time series data FL.
strength_weakness: strength_weakness: 1. Addressing Spatio-Temporal Feature Heterogeneity Challenges: The paper introduces the FedST framework, which specifically tackles the heterogeneity of spatial and temporal features in time series data in real-world applications. Such feature heterogeneity often leads to performance degradation and slower convergence in federated learning.

2. Personalized Model Training: FedST employs personalized federated learning (pFL) methods to enhance the performance of client-specific models during local testing, allowing them to better adapt to the unique data distribution and requirements of each client.

3. Multi-Perspective Feature Extraction: The FedST framework improves model generalization and robustness by extracting and utilizing shared and private features from cross-perspective and cross-client representations through orthogonal training and consistency testing.

4. Superior Experimental Performance: Extensive experimental validation demonstrates that FedST outperforms existing state-of-the-art solutions in personalized federated learning, feature-shift federated learning, and time series federated learning, proving its effectiveness and advantages.
limitations: limitations: 1. Lack of discussion on privacy protection

2. Whether this method will transmit more parameters, leading to a decrease in communication efficiency
paper_summary: paper_summary: This paper proposes a federated learning method called FedST to address the challenge of dual spatial-temporal feature skew in time series classification tasks. The key ideas are:
1) During the training stage, collaborating time view and frequency view of time series data to enrich the mutual information and adopting orthogonal projection to disentangle and align the shared and personalized features between views and between clients.
2) During the testing stage, apply prototype-based predictions and model-based predictions to achieve model consistency based on shared features.
The proposed FedST method is evaluated on multiple real-world time series classification datasets and shows consistent performance improvements over state-of-the-art federated learning baselines.
strength_weakness: strength_weakness: 1) The paper identifies the unique dual spatial-temporal feature skew challenge tailored for time series in the real-world application of federated learning.
2) The orthogonal feature decoupling and regularization techniques are designed to effectively handle the challenges posed by spatial and temporal feature drifts.
3) The method is comprehensively evaluated on multiple real-world time series datasets, demonstrating its effectiveness.
limitations: limitations: The paper focuses on time series classification, while there could be other important time series tasks (e.g., forecasting, anomaly detection) that also suffer from the spatial-temporal feature skew issue in federated learning.
paper_summary: paper_summary: This paper proposes a FedST framework to handle the spatial and temporal heterogeneity in federated learning for time series. It mainly utilizes an orthogonal feature decoupling scheme across different views and clients. Extensive experiments on five datasets demonstrate its effectiveness. However, this paper only include single modal data (time-serises), for the suitability it should be b) unimodal/unimedia in nature but of sufficient interest to the multimedia community. Because the author try to consider multi-view method to improve they proposed model.
strength_weakness: strength_weakness: S1. The logic of the paper is clear, and the overall presentation is good.

S2. The background and motivation are clearly explained.

S3. The proposed method shows impressive accuracy improvement on the five datasets.
limitations: limitations: W1. Some typos. E.g.

   -Line 103: "(non-IID)" lacks space
   - Line 288: "concepts shits among clients". Shift?
   - Line 291: "labe skew"
   - Line 342: "{(...}"; "denots"
   - Line 399  Need to capitalize at the beginning title
   - Line 403 Eq.3 should be "."
   - Line 492: "a initial global prototypes"
   - Equation (7): "i=1"
   - Line 844: "𝜆, 𝜆1, 𝜆2is"

W2. The experiment section needs a running efficiency report and a comparison against the baselines.

W3. Please list the specific dataset name for Fig.2 and 3

W4. What exactly are the time encoder and frequency encoder? More explanation is needed in the paper.

W5. The concepts of Local-Global Vision, Time-Frequency Vision, and Shared Subspace Features Similarity in Fig.4 are very abstract; it’s completely unclear just from the figure, and more explanation is needed.

W6. According to Sec.4.3, the federated aggregation process in Figure 4 seems to be divided into two parts; one is the aggregation of local prototypes. What does the aggregation process on the right represent? As far as I know, the author should only upload the weights of the classifier for aggregation. Why describe it with a whole model? Have the author tried the effects of uploading all parameters for aggregation?

W7. I think the challenge the paper proposed of Spatial feature heterogeneity may not be a novel challenge. For example, we have the HHAR dataset, where if the paper consider Subjects as domains, we can analogize that a domain is a hospital, and each hospital has up to six types of case (classes) data. Since data cannot be shared, we use federated learning to solve the problem of data non-sharing, which is exactly the goal for which federated learning was proposed. From the description of Challenge 1, this doesn’t seem like a new problem.

W8. For Eq.10, I can't understand the meaning of \phi_{P{s,p}}, P_{s,p}, and W_{k+1}. I didn’t find the explanation of these notations.

W9. For the ablation experiments, it seems that the frequency view contributes very little to the model, could a single view complete the entire model design? You should also add experiments w/o L_{vol} and w/o L_{pol}.

W10. The hyperparameter experiments section should include experiments on how to set the parameter \delta.

W11. You need to add some tables and figures to show the non-IID distribution of the dataset since improving federated algorithms under non-IID conditions is the main highlight of your work.

W12. Actually Sec.5.5 should be the main part of the paper's experiment since the purpose of ACM MM is to focus more on multi-modal work