Vi2ACT:Video-enhanced Cross-modal Co-learning with Representation Conditional Discriminator for Few-shot Human Activity Recognition

Authors: kang  xia
wenzhong  li
sanglu  lu
yimiao  shao
  vi2act
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 09:47:26

Abstract: Human Activity Recognition (HAR) as an emerging research field has attracted widespread academic attention due to its wide range of practical applications in areas such as healthcare, environmental monitoring, and sports training. Given the high cost of annotating sensor data, many unsupervised and semi-supervised methods have been applied to HAR to alleviate the problem of limited data. In this paper, we propose a novel video-enhanced cross-modal collaborative learning method, Vi2ACT, to address the issue of few-shot HAR. We introduce a new data augmentation approach that utilizes a text-to-video generation model to generate class-related videos. Subsequently, a large quantity of video semantic representations are obtained through fine-tuning the video encoder for cross-modal co-learning. Furthermore, to effectively align video semantic representations and time series representations, we enhance HAR at the representation-level using conditional Generative Adversarial Nets (cGAN). We design a novel Representation Conditional Discriminator that is trained to assess samples as originating from video representations rather than those generated by the time series encoder as accurately as possible. We conduct extensive experiments on four commonly used HAR datasets. The experimental results demonstrate that our method outperforms other baseline models in all few-shot scenarios.

Main paper: 
 introduction: 
 Human Activity Recognition (HAR), as an emerging research field, has garnered widespread academic attention [28, 40, 42] . HAR aims to analyze and understand individual or group behaviors, habits and other behavioral characteristics. With the rapid development of smartphones [16, 27] and wearable devices, sensor technology has been widely employed. The diverse and rich data generated by sensors, such as accelerometers [12, 42] , gyroscopes, and magnetometers [21] , provide detailed behavioral information, enabling researchers to comprehensively and precisely unveil human behavioral patterns in various environments. Its applications span across various domains, including healthcare [16] , smart home [13] , environmental monitoring [5] , among others.The development of HAR has undergone several stages, initially focusing on manual representation extraction. It later entered the era of machine learning, employing traditional algorithms such as support vector machines and decision trees. In recent years, with the rise of deep learning technology, especially the successful application of convolutional neural networks (CNN) [28, 30, 31] , recurrent neural networks (RNN) [4, 22] and Transformer [37, 40] , HAR has achieved end-to-end representation learning, greatly improving the accuracy of recognition.In deep learning-based HAR, a significant drawback is the cost and complexity associated with acquiring labeled data. Training supervised learning models requires a substantial amount of labeled data, and manually annotating datasets is a time-consuming and expensive task.Recently, Xia et al. [40] proposed TS2ACT to achieve few-shot HAR. It uses label text to search for human activity images via a search engine. Then it adopts a pre-trained CLIP [29] image encoder to jointly train with a time series encoder using contrastive learning, where the time series and images are brought closer in representation space if they belong to the same activity class. Although using image representations for contrastive learning can effectively enhance time series representations, it is still challenging to distinguish temporally opposite activities such as "opening a door" and "closing a door" due to the lack of temporal information in images. Moreover, the lack of a one-to-one correspondence between time series data and their labeled image classes means that straightforward application of contrastive learning may lead to inaccuracies in data alignment.To address the aforementioned issues, we propose a novel videoenhanced cross-modal co-learning method, Vi2ACT, to achieve few-shot HAR. We propose utilizing video semantic representations to enhance the time series encoder instead of image semantic representations. Initially, we employ a text-to-video generation model to create label-related videos, thereby generating semantically richer video content. Subsequently, we use these videos along with their corresponding labels to train a video encoder. Second, we design a novel Representation Conditional Discriminator. This module is trained to assess samples as originating from video encoder rather than those generated by the time series encoder. Through the adversarial training, action representations and video semantic representations can be aligned more efficiently.To summarize, our main contributions are as follows.• We propose a novel cross-modal co-learning approach Vi2ACT to strengthen few-shot HAR. To obtain a large number of video semantic representations, we introduce a novel data augmentation approach that leverages a text-to-video generation model to generate label-related videos. Subsequently, we fine-tuning the video encoder for cross-modal co-learning. • We design a novel Representation Conditional Discriminator that guides the time series encoder to extract class-specific semantic representations through specific class conditions, thus improving HAR performance. To our knowledge, this is the first attempt to enhance HAR from the representation level using conditional Generative Adversarial Nets. 
related work 2.1 supervised human activity recognition: 
 Early HAR research primarily focused on manual representation extraction and traditional machine learning techniques such as SVM [8] and KNN [33] . Consequently, it often required domain expertise and was limited to capturing surface-level features.In contrast, deep learning models have made significant strides in autonomously learning hierarchical representations from raw sensor data. Among the various deep learning architectures, CNNs [25, 28] , RNNs [4, 22] and Transformers-based [9, 34] method have exhibited promising performance. For instance, Zeng et al. [43] design a CNN-based HAR method to capture the local correlations and scale invariance of signals. Challa et al. [4] propose a CNN-BiLSTM model which enables learning of both local representations in sequential data and long-term dependency relationships. Conformer-HAR [17] introduces the Conformer, which is the state-of-the-art (SOTA) model in the field of speech recognition.In order to alleviate the problem of scarcity of labeled data, semisupervised, unsupervised learning, and particularly few-shot learning methods have been widely used in HAR.Semi-supervised HAR [3, 23] primarily utilizes labeled data to train an encoder, which is then used to assign pseudo-labels to a large amount of unlabeled data for further encoder training. To tackle the constraints posed by limited labeled data and class imbalance, Chen et al. [6] propose a pattern-balanced semi-supervised method. Tang et al. [36] utilize confidence to identify the most relevant samples and propagates labels to the most confident samples.Unsupervised HAR [11, 41] typically employs a two-stage pipeline, involving unsupervised pretraining followed by fine-tuning on labeled data. ColloSSL [14] leverages natural transformations in the sensor data from multiple devices to perform contrastive learning, and learns a robust encoder. Ma et al. [24] propose a multi-task learning framework integrating CNN-BiLSTM autoencoder, K-means clustering, and classification tasks to achieve unsupervised HAR.Few-shot HAR methods aim to recognize novel activities with only a few or even one labeled sample. TS2ACT [40] uses label text to search for human activity images, and then jointly trains a time series encoder with the CLIP [29] image encoder to achieve fewshot HAR. RF-CM [39] proposes a general cross-modal framework which achieves few-shot radar-based HAR by using a large number of public WiFi data.Due to the high cost associated with collecting sensor data, recent studies have utilized vision as an auxiliary modality to assist models in extracting representation from sensor data. For instance, IMU-Tube [18, 19] proposes to generate virtual sensor data from videos by applying pose estimation techniques to infer the 3D motion of the human body in the video. This approach typically relies on starting with a large number of videos, as only a segment of IMU signal can be extracted from a single video. Additionally, since this method often depends on the accuracy of pose estimation algorithms, it makes the method highly sensitive to occlusions of the human body. In contrast, our method does not require pose estimation; instead, it utilizes a video action recognition model to directly learn embeddings for HAR from entire video segments.In addition, large language models (LLM) have been used to achieve zero-shot HAR. HealthLearner [20] explores the potential of LLMs as general few-shot learners based on time-series health data. They discover that with minimal in-context tuning, LLMs could ground digital time-series data from wearable and clinicalgrade sensor devices , significantly improving zero-shot inference and supervised baselines in tasks such as activity recognition, calorie computation, and atrial fibrillation classification. HARGPT [15] processes raw IMU data by inputting it into an LLM and uses roleplaying and "think step-by-step" strategies for prompting, successfully identifying human activities from the raw IMU data. We denote the collection of sensor data samples as 𝑆 = {𝑠 1 , 𝑠 2 , ...𝑠 𝑛 }, where each 𝑠 𝑖 is derived by segmenting the original signal into fixed-size segments. Each sample 𝑠 𝑖 ∈ R 𝐿×D is characterized by a multivariate time series, where 𝐿 represents the length of the sample, and D is the channel dimension, containing all x, y, and z-axis from the accelerometer, gyroscope, and magnetometer. 
cross-modal video representation generation: 
 Due to the limited availability of labeled data, we propose to use text-guided video generation to supplement the data deficiency. Initially, we fill the label text (e.g., "jogging") into pre-defined prompt templates (e.g., "A man is [class]") to construct textual prompts (e.g., "A man is jogging"). Subsequently, these prompts are fed into a text-to-video generation model to create videos relevant to the labels. We utilize an open-source video generation model Mod-elScopeT2V [38] to generate large amounts of semantically richer videos. Following this, we employ a pre-trained video encoder to acquire all the video representations. Suppose that video 𝑋 𝑐 is a randomly selected video from the videos relevant to the class 𝑐. We use the video encoder F (•) to obtain video representations 𝑓 𝑉 𝑐 ∈ R 𝑑 byEQUATIONThe time series encoder (denoted by 𝐺 (•)) is intended for recognizing and encoding correlations within sensor data at various time steps. However, unlike words in a sentence, individual time steps in time series data lack inherent semantic meaning. Therefore, extracting local semantic information becomes crucial for meaningful analysis. Previous HAR methods often directly applied Transformer to time series data. In contrast, we enhance locality and capture subsequence-level semantic information by aggregating time into subsequences, as shown in Figure 2 (a). Specifically, for an input time series data 𝒔 ∈ R 𝐿×D , we first segment each input into patches Patching(𝒔) ∈ R 𝑁 ×𝑙 ×D of equal size using a sliding window of width 𝑙 with a 50% overlap, where Patching(•) denotes the patching operation and 𝑁 is the number of patches. To address any shortfall in the last patch, we duplicate the last value and pad it to the end of the original sequence. Then the input data dimensions are mapped to a 𝑑-dimensional Transformer latent space 𝒔 𝑝 ∈ R 𝑁 ×𝑙 ×𝑑 through a convolutional backbone consisting of 1D-CNN layers and SiLU [10] activation functions.EQUATIONWe employ a standard Transformer [37] as our backbone network, consisting of several Transformer blocks. We take the average along the length dimension of 𝒔 𝑝 ∈ R 𝑁 ×𝑙 ×𝑑 and adding learnable positional encodings W 𝑝𝑜𝑠 ∈ R 𝑁 ×𝑑 to get the input 𝒔 𝑑 = Avg(𝑠 𝑝 ) + W 𝑝𝑜𝑠 , where 𝒔 𝑑 ∈ R 𝑁 ×𝑑 denote the token which EQUATION) Here, Self-Attention(•) denotes multi-head self-attention, which efficiently simulates interdependencies between positions in the time series, capturing critical temporal relationships. The Feedforward Neural Network FFN(•) enhances model expressiveness by introducing non-linearity to better represent inputs at a higher level. The normalization operation LayerNorm(•) helps alleviate the issue of internal covariate shift, enhancing independence between representations, thereby improving the robustness and training efficiency of the network.Finally, we take the average of the output from the last Transformer block 𝒛 𝑑 ∈ R 𝑁 ×𝑑 as the input time series representation 𝑓 𝑇 ∈ R 𝑑 . Therefore, for an input 𝒔 ∈ R 𝐿×D , we can obtain its representation through the encoder 𝐺 (•) by.EQUATION)To enhance the capability of the time series encoder in capturing richer semantic representations, we align video representations and time series representations within the same representation space. However, simply using Contrastive Learning [7] or Mean Squared Error loss may lead to inaccuracies in model alignment. To address this issue, we introduce a Representation Conditional Discriminator (denoted by 𝐷 (•, •)) to implicitly align video representations and time series representations.The Time Series encoder and the Representation Conditional Discriminator respectively play the roles of the generator model 𝐺 and the discriminator model 𝐷 in the Conditional Generative Adversarial Network (cGAN). cGAN [26] is a method for training generative models. The generator 𝐺 is trained to produce outputs that the discriminator 𝐷 cannot distinguish from "real" data. The discriminator 𝐷 is trained to evaluate samples as accurately as possible, determining whether they originate from the training data or are generated by 𝐺. In our framework, the Time Series encoder is utilized to map 𝑠 𝑐 to 𝑓 𝑇 𝑐 , where 𝑠 𝑐 represents a time series sample belonging to the action category 𝑐. The discriminator 𝐷 (𝑓 , 𝑐) outputs a scalar indicating the probability that representation 𝑓 , under the condition of class 𝑐, originates from the video rather than 𝐺. The Time Series encoder 𝐺 (•) learns to deceive the discriminator. The discriminator 𝐷 is trained to classify between fake tuples {𝑓 𝑇 𝑐 , 𝑐}, {𝑓 𝑉 𝑖 , 𝑐|where 𝑖 ≠ 𝑐} and real tuples {𝑓 𝑉 𝑐 , 𝑐}, as shown in Figure 1 . Both the generator 𝐺 and the discriminator 𝐷 are trained simultaneously. Specifically, the parameters of 𝐺 are adjusted to minimize log(1 -𝐷 (𝐺 (𝑠 𝑐 ), 𝑐)), while the parameters of 𝐷 are tuned to minimize log 𝐷 (𝑓 𝑉 𝑐 , 𝑐) and log(1 -𝐷 (𝑓 𝑉 𝑖 , 𝑐)). The objective of the Representation Conditional Discriminator can be expressed as:EQUATIONRepresentation Conditional Discriminator adopts a fully connected network with multiple residual blocks as its backbone, as shown in Figure 2(b) . Layer normalization is applied to the discriminated representations within each residual block. To distinguish representation categories within discriminator, class-specific condition 𝑐 projections are introduced in each residual block. The inference process of the discriminator at block 𝑡 is shown in Equation 6𝐶 = 𝑊 (𝑖 ) 𝑡 (SiLU(Layernorm(𝑐))) + 𝑏 (𝑖 ) 𝑡 𝐹 = 𝑊 ( 𝑗 ) 𝑡 (SiLU(Layernorm(𝑓 𝑡 -1 ))) + 𝑏 ( 𝑗 ) 𝑡 𝑓 𝑡 = 𝑊 (𝑘 ) 𝑡 (SiLU(Layernorm(ConCat(𝐹, 𝐶)))) + 𝑏 (𝑘 ) 𝑡 (6)Ultimately, the output of the last block undergoes LayerNorm [1] , SiLU [10] , and a linear layer to serve as the final output, i.e., 𝐷 (𝑓 , 𝑐).After extracting the representation of the time series using the encoder, we employ a representation classifier 𝑔(•) to derive the final classification results. The representation classifier comprises three fully connected neural network layers with SiLU activation and is trained with labeled samples. Assuming 𝒚 represents the label, and ŷ = 𝑔(𝑓 ) is the probability distributions predicted by the classifier, the training objective of the representation classifier is to minimize the classification error by employing cross-entropy loss.EQUATION3.6 Train the Vi2ACT model 3.6.1 Fine-tuning the Video Encoder. Although a pre-trained video encoder can effectively extract action representations from videos, directly using it for video representation extraction may result in suboptimal performance due to significant disparities between the pre-trained and generated videos. Therefore, we initially fine-tune the pre-trained video encoder using the generated videos. Simultaneously, we optimize the above Representation classifier, employed as the video encoder's classifier, through cross-entropy loss. This approach eliminates the need to train a representation classifier from scratch while effectively classifying semantic representations.In particular, for the generated video 𝑋 𝑖 , we use the following loss function for fine-tuning.For the generated video segments 𝑓 𝑉 𝑐 = F (𝑋 𝑐 ), we utilize crossentropy loss to simultaneously optimize the video encoder and representation classifier through the following formula. EQUATION, 𝑐)) + E 𝑠 𝑐 ,𝑐 [log(1 -𝐷 (𝐺 (𝑠 𝑐 ), 𝑐))] + 𝜆E 𝑠 𝑐 ′ ,𝑐 ′ log(1 -𝐷 (𝐺 (𝑠 𝑐 ′ ), 𝑐 ′ )) (9)Here, the hyperparameter 𝜆 ∈ (0, 1) is employed to achieve a balance between labeled and unlabeled data during training.  
evaluation: 
 In this section, we extensively evaluate the proposed model based on several publicly available HAR datasets.We consider four publicly available datasets that cover a wide variety of device types and activity recognition tasks in different environments. Table 1 provides a summary of the statistical information for all four datasets. For data preprocessing, we only use the acceleration, gyroscope and magnetometer sensing signals to form the time series. Then we split the time series into segments of equal window size as input. Each segment contains 500 data points with 50% overlap. Following the literature [32, 40] , we randomly selecting 80% of the users to construct the training set, and the rest for testing. The validation data is the unlabeled data in the training set, which is used for generating pseudo-labels and fine-tuning the model hyperparameters.We compare the proposed method with several state-of-the-art HAR solutions including both supervised and semi-supervised methods.The following are two fully supervised baseline algorithms.• TCN [2] . TCN combines the best practices such as dilated convolutions and residual connections with causal convolutions for autoregressive prediction. • ConformerHAR [17] . ConformerHAR introduces the stateof-the-art (SOTA) model Conformer in the field of speech recognition. Furthermore, they improved the performance by incorporating CNN layers that excel at extracting local representations effectively.We also compare our method with mainstream semi-supervised and unsupervised HAR algorithms as follows.• FixMatch [35] . FixMatch is a semi-supervised learning framework. It generates pseudo-labels for large amounts of unlabeled data and then use these pseudo-labels, along with the original labeled data, to train the model. • SimCLR [7] . The idea of SimCLR is to learn useful features through contrastive learning. The algorithm augments input data with random transformations (such as flipping, cropping, rotating, etc.) and uses a shared neural network to extract features. It then uses a contrastive loss function to encourage features between similar classes to be closer together, and features between dissimilar classes to be farther apart. After unsupervised training, we only keep the backbone network and freeze its parameters. We further train a two-layer fully connected network as a classifier using the same amount of labeled samples. • Self-supervised Learning (SSL) [32] . Self-supervised learning predicts time series transformations by training eight proxy tasks. After self-supervised training, they freeze the model parameters and use the labeled data to train a neural network classification head for HAR.• Multi-Task Deep Clustering (MDC) [24] . Multi-task deep clustering uses an autoencoder to get the feature. The Kmeans clustering algorithm is then applied to divide the dataset into different groups for generating pseudo-labels. Finally, a DNN is further trained for HAR.We implement the proposed Vi2ACT using Python and Pytorch. Following the literature [40] , we employ 6 Transformer blocks with the number of heads and the dimension 𝑑 set to 16 and 512, respectively. We employ the Adam optimizer for both fine-tuning the video encoder and model training stages. During the fine-tuning stage, the learning rate for the video encoder F is set to 0.000001, while the learning rate for the representation classifier 𝑔 is set to 0.0005. In the model training stage, the learning rates for the time series encoder 𝐺 and Representation Conditional Discriminator 𝐷 are both set to 0.0005. The hyperparameter 𝜆 for balancing the training of labeled and unlabeled data in Eq. 9 is set to 0.01 by default.In the hyperparameter analysis section, we will provide a detailed discussion of our hyperparameter settings. The experiments were conducted on a personal computer equipped with Intel(R) Core(TM) i7-12700 CPU @4.90 GHz 20 cores, an NVIDIA GeForce RTX 3090Ti graphics card and 32GB RAM.We compare Vi2ACT with existing work on four commonly used HAR datasets, as shown in Table 2 . Vi2ACT exhibits a substantial advantage over other baseline models in all scenarios with limited samples. Even when utilizing the entire set of labeled samples, competitive results are consistently achieved. Furthermore, Vi2ACT significantly outperforms the other two supervised algorithms, TCN and ConformerHAR. Additionally, Vi2ACT outperforms other semisupervised and unsupervised representation learning approaches, indicating its superior ability to leverage video representations for enhancing the model's acquisition of semantic representations compared to alternative semi-supervised learning methods. Simultaneously, Vi2ACT addresses the gap between unsupervised pretraining tasks and downstream inference tasks. From the experimental results using fully dataset, Vi2ACT demonstrates its efficacy both in effectively mining semantic representations from temporal data with a limited number of labeled samples and in further enhancing the semantic classification capability of the encoder with a substantial number of labeled samples.The accuracy for each class is illustrated in the Figure 3 . Vi2ACT demonstrates remarkable performance across the three datasets, achieving up to 98% accuracy in specific classes and even reaching 100% accuracy. Furthermore, Vi2ACT exhibits enhanced ability in discriminating between "Walking downstairs" and "Walking upstairs, " thereby effectively mitigating the drawbacks associated with image semantic representation guidance.To validate the effectiveness of our proposed Representation Conditional Discriminator 𝐷, we compare its performance with directly using the Euclidean distance (i.e., replacing 𝐷 with mean square error loss function) and contrastive learning (i.e., replacing 𝐷 with contrastive loss function). The results are summarized in the Table 3. It can be observed that due to the lack of strict one-to-one correspondence between time series and video, our method can more effectively align these two modalities.We present three prompts templates for Vi2ACT in Table 4 , which include "a video of [class] ", "an action video of [class]" and "a man is [class]". We evaluate the model's inference performance across videos generated with different prompts. For each prompt, we conduct 10 evaluations using all labeled data and compute the average accuracy. The difference in prompt will also lead to performance variance, among which "a man is [class]" has the best performance. Therefore, we use the prompt "a man is [class]" in our model training.We conduct experiments with different hyperparameters, and the results are shown in Tables 5 6 7 and 8 . It is worth noting that a deeper Representation Conditional Discriminator does not necessarily lead to better performance, as excessively deep networks may cause overfitting. Additionally, the performance of Vi2ACT improves as the number of generated videos increases, and the model performance almost converges when using 500 videos. This validates our proposal of using rich video representations to enhance the time series encoder. Finally, setting 𝜆 to 0.01 achieves a good balance between labeled and unlabeled data. 
conclusion: 
 In this paper, we propose a video-enhanced cross-modal co-learning method, Vi2ACT, to achieve few-shot HAR. We introduce a new data augmentation approach that utilizes a text-to-video generation model to generate class-related videos. Subsequently, a large quantity of video semantic representations are obtained through fine-tuning the video encoder for cross-modal co-learning. Furthermore, to effectively align video semantic representations and time series representations, we design a novel Representation Conditional Discriminator to enhance HAR at the representation level. Finally, we conduct extensive experiments to demonstrate the effectiveness of our method. 
 
 Appendix:  


Decision: All reviewers think the paper has proposed a novel method, where using t2v model to create data is innovative. Reviewers have some concerns on the quality of the generated videos, and some details in experiments. After the author’s rebuttal, most of the concerns have been resolved. The AC recommend accept for this submission.

Reviews: review: paper_summary: paper_summary: This manuscript proposes a method of data augmentation using text-to-video generation models to complete human activity recognition tasks with sensor time-series data. The authors have designed a novel conditional representation discriminator, trained with the idea of conditional generative adversarial networks, to accurately evaluate samples. They demonstrate the superiority of their method compared with several state-of-the-art approaches on four commonly used HAR datasets.
strength_weakness: strength_weakness: 1，The method section clearly introduces the overall structure proposed by the authors, with a distinctive idea of training a time-series data encoder using conditional generative adversarial networks.

2，The overall structure of the paper is complete and the narrative is clear.
limitations: limitations: 1，It is recommended to check if there is any repetition in the content of the article itself, such as the repeated description of TS2ACT in Section 1 and Section 2.3.

2，Most of the content of the article overlaps with the content of the TS2ACT paper, especially the experimental section, which is almost identical with only minor modifications in the data. It is recommended to rewrite it to reduce the overlap rate.

3，In Section 3.3, it is recommended to indicate that the reference image is Figure 2 in the introduction of the time-series encoder.

4，The discriminator discriminates which input features are time series and which are video generations given the known category c, thereby optimizing the generator training to make the generated features more consistent with category c. Therefore, isn't the overall effectiveness of the entire model particularly susceptible to the quality of the generation model?

5，It is recommended to cite and introduce the specific text-to-video generation model used in the paper.
paper_summary: paper_summary: This paper presents Vi2ACT, a video-enhanced cross-modal collaborative learning method designed to improve Human Activity Recognition (HAR) in few-shot scenarios. The authors use a text-to-video generation model for data augmentation, generating class-specific videos to enrich the training dataset. Then, the video encoder is fine-tuned to capture video features. Representation Conditional Discriminator is also introduced to distinguish between real and generated videos. The experiments conducted on four major HAR datasets show that Vi2ACT surpasses existing baseline models.
strength_weakness: strength_weakness: - The data augmentation by using video generation models for few-show action recognition is a promising strategy. 
- The experiment results in few shot settings outperform the existing methods with a noticeable margin. 
- Extensive ablation studies are conducted to showcase the effectiveness of the methods.
limitations: limitations: - As the data augmentation is from pretrained video generator, and the videos are used to train classifier to produce pseudo-label, how did the authors control the quality of the videos?
- In Table 7, the number of generated videos of more than 500 should be examined as well.
- Besides ModelScopeT2V[33], the authors should also examine the effect of video generators.
paper_summary: paper_summary: This study propose a video-enhanced cross-modal co-learning method to achieve few-shot HAR. Specifically, the proposed framework applied video semantic representations to enhance the time series encoder for HAR problem.
strength_weakness: strength_weakness: 1. The manuscript is easy to understand and follow.

2. The novelty of using video representation to guide time series HAR task is enough.
limitations: limitations: 1. Since the proposed method is mainly for cross-modal task, can the authors provide the feature representation visualization results  extracted from multidimensional data to validate the effectiveness of the proposed method?

2. Since this study is focused on few-shot HAR problem, the authors are encouraged to reorganize the related work section. For instance, section 2.1 can be shorten and section 2.2 and 2.3 can be extended. There are several recent few-shot studies [1-3] for HAR problem. In addition, current community start to applied LLMs for crossmodal task [5-6] . The authors are encouraged to select and discuss them in the related work to have a more comprehensive literature review. 

[1] Wang, X., Liu, T., Feng, C., Fang, D., & Chen, X. (2023). RF-CM: Cross-modal framework for RF-enabled few-shot human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 7(1), 1-28. 
[2] Wang, X., Yan, Y., Hu, H. M., Li, B., & Wang, H. (2024). Cross-modal Contrastive Learning Network for Few-Shot Action Recognition. IEEE Transactions on Image Processing.
[3] Yuan, H., Chan, S., Creagh, A. P., Tong, C., Acquah, A., Clifton, D. A., & Doherty, A. (2024). Self-supervised learning for human activity recognition using 700,000 person-days of wearable data. npj Digital Medicine, 7(1), 91.
[4] Jain, Y., Tang, C. I., Min, C., Kawsar, F., & Mathur, A. (2022). Collossl: Collaborative self-supervised learning for human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6(1), 1-28.
[5] Leng, Z., Bhattacharjee, A., Rajasekhar, H., Zhang, L., Bruda, E., Kwon, H., & Plötz, T. (2024). IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition. arXiv preprint arXiv:2402.01049.
[6] X. Liu, D. McDuff, G. Kovacs, I. Galatzer-Levy, J. Sunshine, J. Zhan, M.-Z. Poh, S. Liao, P. Di Achille, and S. Patel, “Large
language models are few-shot health learners,” arXiv preprint arXiv:2305.15525, 2023.

3. Format issue. Figure 1 is confused based on the five main components listed in line 230. Please redesign the overall framework for better understanding. In addition, there are some inconsistency in Table 2 regarding bolding formats. Please proofread the manuscript carefully.
paper_summary: paper_summary: The paper  presents a novel approach for human activity recognition (HAR) using few-shot learning. The authors propose a method called Vi2ACT, which enhances HAR by using a video-enhanced cross-modal collaborative learning approach. This method includes several key components: 1) Text-to-Video Generation. They use a text-to-video generation model to create videos relevant to different activity classes, providing semantically rich video content. 2) Cross-modal Co-learning. The generated videos are used to train a video encoder. This video encoder then collaborates with a time series encoder to enhance the representations used for HAR. 3) Representation Conditional Discriminator. They introduce a novel discriminator that distinguishes whether a representation originates from video data or time series data, helping to align the two modalities more effectively.
strength_weakness: strength_weakness: The authors present a highly novel and theoretically sound approach to enhancing human activity recognition using few-shot learning. Its strengths is as follows:

1) The approach of leveraging video data, rather than just image data, addresses the limitation of temporal information that is often missing in static images, thereby providing a richer dataset for model training.

2) The use of a text-to-video generation model to create class-related videos is an innovative approach in the context of human activity recognition (HAR). This method generates semantically rich video content that supplements the limited labeled sensor data typically available for HAR.

3) The application of conditional GANs to enhance representation learning in HAR is technically sound and leverages the strengths of adversarial learning to improve model robustness and accuracy.

4) The proposed cross-modal co-learning framework, which involves simultaneous training of video and time series encoders, is a novel concept. This method effectively integrates different types of data to enhance the learning process.
limitations: limitations: 1. The performance of the model heavily relies on the quality of the videos generated by the text-to-video model. If the generated videos are not of high quality or do not accurately represent the activities, it could negatively impact the model's performance. The paper does not provide a detailed analysis of the quality of the generated videos or how variations in video quality affect the overall performance of the HAR system.

2. The paper does not discuss the computational efficiency of the algorithm. The proposed algorithm incorporates various techniques such as video feature extraction, GANs, and contrastive learning, and we cannot determine how much additional computational overhead this will introduce. Considering that HAR algorithms are often run on lightweight devices, I believe it is essential to evaluate the computational speed.

3. Regarding the third contribution, I believe that extensive experimental validation is a basic requirement for any research work. Unless you have made a special contribution, such as proposing a new dataset or a new evaluation metric, experimental validation should not be considered a core contribution. Writing it this way would appear unprofessional.