Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 04:51:52

Abstract: Skeleton-based action recognition has garnered significant attention due to the utilization of concise and resilient skeletons. Nevertheless, the absence of detailed body information in skeletons restricts performance, while other multimodal methods require substantial inference resources and are inefficient when using multimodal data during both training and inference stages. To address this and fully harness the complementary multimodal features, we propose a novel multi-modality co-learning (MMCL) framework by leveraging the multimodal large language models (LLMs) as auxiliary networks for efficient skeleton-based action recognition, which engages in multi-modality co-learning during the training stage and keeps efficiency by employing only concise skeletons in inference. Our MMCL framework primarily consists of two modules. First, the Feature Alignment Module (FAM) extracts rich RGB features from video frames and aligns them with global skeleton features via contrastive learning. Second, the Feature Refinement Module (FRM) uses RGB images with temporal information and text instruction to generate instructive features based on the powerful generalization of multimodal LLMs. These instructive text features will further refine the classification scores and the refined scores will enhance the model’s robustness and generalization in a manner similar to soft labels. Extensive experiments on NTU RGB+D, NTU RGB+D 120 and Northwestern-UCLA benchmarks consistently verify the effectiveness of our MMCL, which outperforms the existing skeleton-based action recognition methods. Meanwhile, experiments on UTD-MHAD and SYSU-Action datasets demonstrate the commendable generalization of our MMCL in zero-shot and domain-adaptive action recognition. Our code will be publicly available and can be found in the supplementary files.

Main paper: 
 introduction: 
 Human action recognition is an important task in video understanding. The action conveys central information like body tendencies and thus helps to understand the person in videos. In pursuit of precise action recognition, diverse video modalities have been explored, such as skeleton sequences [6, 19] , RGB images [13, 61] , text descriptions [49, 56] and depth images [54] . In particular, the graph-structured skeleton can well represent body movements and is highly robust to environmental changes, thereby adopted in many studies using graph convolutional networks (GCNs) [7-9, 12, 30, 36, 50] , it also demonstrates strong scalability and real-time capabilities when deployed on edge devices. However, these skeleton-based methods [19, 39, 53, 65] (Fig. 1 (a) ) only using skeletons during training and inference stages will restrict recognition performance due to the inherent defects of skeletal modality. For instance, the skeleton modality lacks the ability to depict detailed body information (e.g. appearance and objects) and has difficulty in fine-grained recognition when dealing with similar actions.A good approach to addressing the aforementioned skeletonbased limitations is to introduce complementary multi-modality for action recognition. The typical process [1, 10, 13, 23, 61] involves employing two networks to separately model the skeletons and other modalities (e.g. RGB images and depth images), then some ensemble techniques are applied to merge the predictions from We apply the MMCL to GCN backbones and observe its capability to aid in better modeling. Specifically, we show the overall accuracy using joint modality on the NTU120 XSub benchmark and the top-8 actions with the highest improvement. multiple modalities. These multimodal-based methods (Fig. 1 (b) ) can make up for the defects of a single modality and provide more comprehensive information for fine-grained action classification by leveraging the complementary nature of multimodal data. Nevertheless, they suffer from the drawbacks of requiring significant inference resources and appear less efficient when deployed on edge devices due to the use of multi-modality in both training and inference stages(Fig. 1 (b) ). The aforementioned skeleton-based and multimodal-based methods naturally lead to a question: How to better leverage the complementary nature of multi-modality while retaining the efficient inference with single-skeleton modality?Motivated by the above question, we propose a multi-modality co-learning (MMCL) framework for efficient skeleton-based action recognition, which enhances the model's performance and generalization via multi-modality co-learning, while only using the concise skeleton in the inference stage to preserve the efficiency (Fig. 1 (c) ). Due to the effectiveness of multi-modality co-learning during the training stage, our proposed MMCL acquires enhanced robustness and generalization when applied to optimize mainstream GCN backbones, exhibiting improved performance in overall accuracy and specific actions as shown in Fig. 2 .In our MMCL framework, we incorporate the multi-modality co-learning into action recognition and provide instructive multimodal features based on the multimodal LLMs [14, 27, 28, 66] for skeletons. The proposed MMCL is shown in Fig. 3 , which will use skeleton, RGB and text modalities during the training stage, while only using the skeleton in inference to achieve efficiency. Given that RGB features [10, 61] effectively compensate for the skeletons, while existing multimodal LLMs exhibit strong generalization on real RGB images and achieve success in various visual tasks [15, 63, 67] , we adopt the RGB images as the input for the multimodal LLMs to help better model the skeletons. To mitigate the limitation of some existing multimodal LLMs (e.g. BLIP [28] , BLIP2 [27] , MiniGPT-4 [66] and LLaMA AdapterV2 [14] ) that cannot directly process video streams, we combine frames from the video stream into RGB images that include temporal information. Besides, to better harness the potent modeling capabilities of these multimodal LLMs for intrinsic features in images and text, we input the RGB images with text instructions into the Feature Refinement Module (FRM) for generating instructive text features conducive to action recognition. These instructive text features from multimodal LLMs will further refine the classification scores output by the fully connected layer and enhance the model's robustness and generalization in a manner similar to soft labels. Meanwhile, to better utilize the introduced RGB images with temporal information in LLMs, our MMCL uses a deep neural network to obtain RGB features in the Feature Alignment Module (FAM). Likewise, inspired by CLIP [42] , our MMCL uses the contrastive learning [56, 64] to better assist in modeling skeletons.Our contributions are summarized as follows: • 
related work 2.1 skeleton-based action recognition: 
 Deep learning-based methods [37, 57, 59 ] have achieved high success using CNNs [21, 22, 26, 32, 55] and RNNs [2, 25, 29, 35, 40, 44, 47, 62] in skeleton-based action recognition. To address the adverse effects due to view variations and noisy data, Liu et al. [37] visualized skeleton sequences as color images and fed them into a multi-stream CNN for deep feature extraction. Wang et al. [47] proposed a two-stream RNN to model the temporal dynamics and spatial structure of skeleton sequences, which breaks through the limitations of RNN in processing raw skeleton data. GCNs can effectively process structured data, thereby being adopted by many researchers in skeleton-based action recognition. For instance, Yan et al. [59] is the first to use GCNs for skeleton-based action recognition by proposing the ST-GCN to learn spatiotemporal features from skeleton data. Chen et al. [6] improve the design of GCNs by proposing a channel-wise topology refinement graph convolutional network (CTR-GCN), which effectively aggregates joint features in each channel. The methods mentioned above are limited by the drawbacks of only using skeleton modality.S R Y Y F G F G F C F C GCN GCN Backbone S M Symbols F G → Skeleton Feature F C → RGB Feature S M → Classification Score S R → Refinement Score Y → Action Lable I → RGB Image Q → Text Instruction F text → Text Feature M → Learnable Matrix I Q M F textBenefiting from the emergence of various multimodal datasets and improvements in computing resources, research about multimodalbased action recognition [1, 10, 13, 23, 61] has become popular. To address GCNs are subject to limitations in robustness, interoperability and scalability, Duan et al. [13] proposed a novel PoseConv3D to use both RGB heatmap and skeleton modalities for robust human action recognition. Das et al. [10] proposed a video-pose network (VPN) to project the 3D poses and RGB cues in a common semantic space, which enabled the action recognition framework to learn better spatiotemporal features exploiting both modalities. Commonly used modalities for multimodal-based action recognition include skeletons, color images, depth maps, text and point clouds. Methods [10, 11, 13, 20, 41, 54, 61] of integrating these multiple modalities have been shown to achieve better performance by leveraging multimodal features. Unlike the above methods that use multimodal data in both training and inference stages, our MMCL introduces multimodal data for multi-modality co-learning in the training stage and only uses the concise skeleton to keep efficient in inference.The powerful generation ability of large language models (LLMs) enables them to be transferred to different tasks effectively. Prompt learning and instruction learning are capable techniques of adapting different pre-trained LLMs to different tasks, thereby being applied in action recognition as an auxiliary strategy by many researchers. Wang et al. [49] proposed an ActionCLIP to directly uses class labels as input text to construct cues for video action recognition. Xiang et al. [56] used LLMs based on action labels to provide prior knowledge for skeleton-based action recognition. Different from the above methods that use simple text modalities based on action labels and unimodal LLMs, our MMCL is based on advanced multimodal LLMs for multi-modality co-learning and sets the text instructions according to the skeletal defects. Besides, these methods [49, 56, 58] that explicitly generate text features based on action labels are not advantageous for unknown actions. With the emergence of multimodal LLMs such as BLIP [28] , BLIP2 [27] , MiniGPT-4 [66] and LLaMA AdapterV2 [14] , receiving multimodal input can often generate more robust and richer text information. Inspired by this, our MMCL framework inputs RGB and text modality into multimodal LLMs and obtains the output text features for refining the classification score of the skeleton. These refined scores will enhance the model's robustness and generalization in a manner similar to soft labels. Benefiting from the powerful generalization capabilities of multimodal LLMs, the soft labels obtained by MMCL can be transferred to domain-adaptive action recognition. 
methodology: 
 In this section, we present our proposed Multi-Modality Co-Learning (MMCL) framework in detail. MMCL aims to enhance skeleton The person is holding a cup in one hand and a fork in the other. They are wearing black pants and a black shirt…The movement in the image shows the woman holding a cup of coffee in her left hand and drinking from it…Please describe the {target} .A woman standing in front of a white wall.The image shows a room with a woman standing in the middle of the room. The woman is wearing a green shirt and black pants. She is standing in front of a white wall ...The person is a woman who is wearing a black shirt and pants. She is holding a white coffee mug in her hand.(a) representation learning with complementary and instructive multimodal features based on the powerful generalization capabilities of multimodal LLMs. Meanwhile, MMCL is orthogonal to the backbone networks and thus can be coupled with various GCN backbones and multimodal large language models. Below we will introduce the proposed MMCL in five parts.(b) 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1Due to the unique advantages of GCNs in modeling graph-structured data, the GCN is prevailing for skeleton-based action recognition.In our MMCL framework, we also adopt GCN as the backbone network to model the skeleton modality and the GCN blocks are composed of a graph convolution layer and a temporal convolution layer. The normal graph convolution can be formulated as:EQUATIONwhere 𝐻 𝑙 is the joint features at layer l and 𝜎 is the activation function. D ∈ R 𝑁 ×𝑁 is the degree matrix of N joints and W l is the learnable parameter of the l-th layer. A is the adjacency matrix representing joint connections. Generally, the A can be generated by using static and dynamic ways. The A is generated by using datadriven strategies in dynamic ways while it is defined manually in static ways. [6] introduces A into graph convolution in the form of S = (s id , s cp , s cf ) and s id , s cp , s cf indicate the identity, centripetal and centrifugal edge subsets respectively. [24] proposes an HD-Graph to introduce A into graph convolution and uses a six-way ensemble strategy to train the GCN.In our MMCL framework, we use skeleton modality, RGB modality and text modality during the training stage, while only using the concise skeleton modality during the inference stage. The skeleton data is collected through Kinect sensors and it is a natural topological graph, which is denoted as G = {V, E}. V = {v 1 , v 2 , • • • , v 𝑁 }EQUATIONwhere 𝜀 denotes a detector. Inspired by the top-down pose estimation, we use a person detector 𝜀 to filter out most ambient noise and focus on the human body. Meanwhile, we resize the extracted person frame to save computing and storage resources by Γ. This detection and cropping method significantly filters environmental noise, while minimizing the loss of spatial information and has been widely applied in [33, 61] . Inspired by the uniform sampling strategy in [13] , we uniformly sample m samples from n person frames to reduce the temporal dimension byU 𝑚 𝑛 . ⊕ 𝑚 = [𝑓 1 ⊕ 𝑓 2 ⊕• • •⊕ 𝑓 𝑚-1 ⊕ 𝑓 𝑚 ]means to concat m person frames along the temporal dimension to form the RGB image with temporal information. EQUATIONDue to the CNNs have unique advantages in processing Euclidean data (e.g. RGB images), the FAM utilizes a deep convolutional neural network Θ to model high-level features from RGB image I. Before extracting RGB features, the FAM will resize and normalize the image to speed up the convergence of the model. In order to achieve alignment with the global skeleton features, the RGB features extracted by CNN will go through the MLP operation M 𝑎𝑙𝑖𝑔𝑛 . This alignment operation facilitates subsequent contrastive learning between two features. Contrastive learning is popular and widely used in deep learning, especially playing a key role in multimodal learning, which aims to learn the common features between similar instances and helps to distinguish similar human actions. Our MMCL conducts contrastive learning based on the framework [68] between the RGB features F 𝑖 𝑐 output by FAM and the skeleton features F 𝑖 𝑔 extracted by GCN backbone, while calculating the contrastive loss by:EQUATIONEQUATIONEQUATIONwhere 𝜃 is the cosine similarity and 𝜏 is a temperature parameter. Note that the skeleton features extracted by the GCN backbone will first go through a MeanPooling layer to obtain the global features. Since the purpose of contrastive learning is to learn common features between similar instances, and the common features between RGB videos and skeletons of the same action are the motion dynamics of this action, so the FAM can promote the learning of motion dynamics from skeleton. The second column represents the prediction of models for the currently visualized sample and the third column represents the accuracy for all samples within the currently visualized category. We selected four difficult action samples that are prone to prediction errors in CTR-GCN, which all belong to categories involving objects or are highly relevant to hands and is difficult to distinguish objects from skeleton diagrams. Our MMCL set text instructions based on skeletal defects and generated instructive features through LLMs to guide the model to focus on the modeling of human hands and objects, thus leading to significant accuracy improvements in these difficult action samples. Then the token features of the multimodal LLM decoder will be retained.EQUATIONEQUATIONIn terms of implementation, the FRM unifies text features obtained from different input samples into the same dimension n through 𝜙. Meanwhile, n learnable matrices M are multiplied with text features F 𝑡𝑒𝑥𝑡 ∈ R 𝑛 in each feature channel i and optionally refine the classification scores S M . These refined scores S R in Eq. 8 will enhance the model's robustness and generalization in a manner similar to soft labels. Intuition suggests that introducing more detailed text instructions would result in broader and more comprehensive action descriptions, which may be more helpful to the action recognition task. However, inputting complex text instructions into the MiniGPT-4 and BLIP models tends to generate more irrelevant content, which can introduce noise features to some extent. Therefore, our MMCL adopts a brief and direct text instruction based on skeletal defects, aiming to demonstrate the effectiveness of this text prompt and our MMCL framework.In our FRM, we focus on modeling the objects that human holds since the skeleton cannot recognize objects and selectively enhance and refine the classification scores for samples with or without objects. In fact, this focus can be easily transferred to different situations and adapted to different LLMs and text instructions by adjusting the learnable matrices during the training stage. Our FRM effectively demonstrates how to transfer multimodal LLMs to action recognition in a novel and general way. In Eq. 9, the refined classification score S 𝑅 and the true label Y of action samples will be used to calculate the refinement loss by:EQUATIONwhere N is the number of samples in a batch and Y 𝑖 is the one-hot presentation of the true label about action sample i. Actually, the FRM of our MMCL trains the network by refining the classification scores to generate pseudo-labels and use text features to supervise the model, which was also shown to be effective in [31] .In our proposed MMCL, we use Cross-Entropy (CE) loss as the classification loss. Our MMCL uses contrastive loss L 𝐶 and refinement loss L 𝑅 as the auxiliary loss to constrain the training of the whole network, thereby introducing multimodal features into skeletonbased action recognition. The complete training loss function is defined as:EQUATIONwhere 𝜆 1 and 𝜆 2 represent two hyper-parameters and L 𝑐𝑙𝑠 is the classification loss.It is easily observed that the distribution of training data derived from public datasets differs from that of testing data collected in real world or other datasets, leading to a domain gap between the training (source) and testing (target) inputs. Considering this situation, our MMCL introduces multi-modality co-learning based on multimodal LLMs to enhance the robustness and generalization of the model. When dealing with skeletons from other domain datasets or real world, our MMCL conducts data interpolation to align with by 40 different performers. We follow the evaluation using two benchmarks provided in the original paper [43] . NTU-RGB+D 120 [34] is derived from the NTU-RGB+D dataset. A total of 114,480 video samples across 120 classes are performed by 106 volunteers and captured using three Kinect V2 cameras. We also follow the evaluation using two benchmarks as outlined in the original research [34] . Northwestern-UCLA [48] comprises 1494 video clips spanning across 10 distinct categories. Our evaluation process aligns with the protocol in [48] . To demonstrate the generalization of our MMCL in zero-shot and domain-adaptive action recognition, we also conduct experiments on the UTD-MHAD and SYSU-Action datasets. UTD-MHAD [4] consists of 27 different actions performed by 8 subjects. Each subject repeated the action for 4 times, resulting in 861 action sequences in total. SYSU-Action [18] comprises 12 distinct actions performed by 40 participants, culminating in a collection of 480 video clips. Implementation details: All experiments are conducted on two Tesla V100-PCIE-32GB GPUs. We use SGD to train our model for a total number of 110 epochs with batch size 200 and we use a warm-up strategy [16] to make the training procedure more stable. The initial learning rate is set to 0.1 and reduced by a factor of 10 at 90 and 100 epochs, the weight decay is set to 4e-4. We use CTR-GCN [6] as the GCN backbone and use the InceptionV3 [45] to extract deep RGB features in FAM. We use YoloV5 [46] as the person detector for video frames and uniformly sample five frames from person frames to form the RGB image with temporal information. We extract instructive text features from RGB images and text instruction based on the BLIP [28] in FRM. We introduce four skeleton modalities in [6] and the six-way ensemble in [24] to achieve multi-stream fusion. In concrete implementation, we utilize the HD-Graph [24] to replace the natural connectivity of human joints, thereby altering the initialization of symbol A in Eq. 1. 
comparison with state-of-the-art methods: 
 Notably, our MMCL is the first to introduce multimodel features based on multimodel LLMs for multi-modality co-learning into skeleton-based action recognition, which achieves better performance by assisting previous GCN backbones and can be easily migrated to more advanced backbone networks. In Table 1 , we demonstrate the accuracy improvement for some action categories when applying the proposed MMCL in different GCN backbones on the NTU120 X-Sub benchmark. We also compare our MMCL with the state-of-the-art methods on the NTU 60, NTU 120 and NW-UCLA datasets in Table 2 . On three datasets, our model outperforms all existing methods under nearly all evaluation benchmarks.In Table 2 , on the X-Sub benchmark of the NTU 120 dataset, our MMCL using the same ensemble outperforms the baseline CTR-GCN [6] , which shows that introducing multimodal features based on MMCL into the backbone will achieve better performance. On the X-Set benchmark of the NTU 120 dataset, compared with VPN [10] and TSMF [3] that use both the skeleton and RGB modalities at the inference stage, our MAL performs better in both recognition accuracy and inference cost by only using the skeleton at the inference stage. Compared with the method [56] that use simple text modalities based on action labels and unimodal LLM, our MMCL achieved better recognition accuracy on three datasets.4.2.1 Effectiveness in different GCN backbones. In this section, we conduct ablation experiments on the X-Sub benchmark of NTU 120 dataset by applying MMCL to the strong CTR-baseline (CTRS-GCN) and the CTR-GCN. In Table 3 , we investigate the effectiveness of MMCL across different skeleton modalities and GCN backbones. By introducing the MMCL into CTR-GCN, an accuracy improvement of 0.78% and 0.98% is achieved on the joint and bone modalities respectively. The results in Table 3 show that our MMCL leads to improvements across different skeleton modalities and backbones. Besides, our MMCL can help the backbones improve the recognition accuracy of hard actions as shown in Fig. 6 .In Table 4 , we demonstrate the effectiveness of the FAM and show its robustness by using different CNNs. When the FAM is applied to CTRS-GCN and CTR-GCN, the recognition performance is improved. We argue that the MMCL enhances the modeling of skeleton features by performing contrastive learning between the RGB features extracted by the FAM and the global skeleton features output by the GCNs. In Table 5 , we also show the effectiveness of the FRM and its robustness by using different multimodel LLMs. When the FRM based on different LLMs is applied to the baseline network, the recognition performance is improved. An interesting observation here is that the performance of FRM based on MiniGPT-4 is worse than those based on Blip. We speculate that when faced with the same text instructions, MiniGPT-4 tends to focus on describing a lot of clothing-related information and the background (e.g. the color of clothes), which inadvertently introduces unnecessary textual noise. 6 present the accuracy of our MMCL under different hyperparameter settings. In Table 7 , we also show the parameters and computation cost required by MMCL for inference when using a single action sample. In Table 8 , we compared the recognition accuracy using different adjacency matrix on the NTU120 X-Sub benchmark. Here, replacing the naturally connected adjacency matrix with HD-Graph resulted in higher accuracy. In Table 9 , we explore the recognition accuracy when different text instructions are employed based on skeletal modality defects. The results in Table 9 suggest that the FRM based on different text instructions improves the recognition performance of the baseline model. Simultaneously, they also demonstrate that our FRM can be transferred to different reasonable text instructions as needed and can generate instructive textual features through multimodal LLMs to aid in the modeling of the baseline model.In Table 10 , we utilized a subset of UTD-MHAD dataset [4] and SYSU-ACTION dataset [18] to investigate the generalization of MMCL and LLMs. We utilize weights trained on the joint modality of NTU 120 X-Sub benchmark for zero-shot action recognition and use interpolation to extend the joints to fit the models' input as shown in Fig. 5 . In zero-shot recognition, all samples from two datasets have never been involved in training and they directly use a previously well-trained model for action recognition. Therefore, the accuracy of zero-shot recognition can effectively reflect the model's generalization and transferability. In Table 10 , when the backbone CTR-GCN is trained with MMCL, the top-1 and top-5 accuracy for actions from different domains are 39.17% and 76.67%, while the baseline CTR-GCN trained without MMCL are 27.50% and 51.67% in SYSU dataset. Compared to the baseline CTR-GCN and HD-GCN, our MMCL shows a positive improvement in both Top-1 and Top-5 accuracy. We believe that MMCL benefits from the RGB modality and multimodal LLMs, thus demonstrating commendable generalization and robustness when facing other datasets in different domains. Besides, when introducing text features generated by multimodal LLMs to refine the zero-shot scores based on the Eq. 8, we are surprised to find an improvement in both recognition accuracy. This indicates that soft labels based on multimodal LLMs can be transferred to the domain adaptive action recognition. We argue that multimodal LLMs exhibit strong generalization capabilities, as they also can generate robust and effective text features when presented with RGB images in different domains. 
conclusion: 
 We present a novel Multi-Modality Co-Learning (MMCL) framework for efficient skeleton-based action recognition, which is the first to introduce multimodal features based on multimodal LLMs for multi-modality co-learning into action recognition. The MMCL guides the modeling of skeleton features through complementary multimodal features and maintains the network's simplicity by using only skeleton in inference. Meanwhile, our MMCL is orthogonal to the backbone networks and thus can be coupled with various GCN backbones and multimodal LLMs. The effectiveness of our proposed MMCL is verified on three benchmark datasets, where it outperforms state-of-the-art methods. 
 
 Appendix:  


Decision: Pros:
- The paper is the first to introduce multi modal LLMs in skeleton-based action recognition. 
- The experiments are sufficient and have demonstrated the effectiveness of the proposed method.
- The paper is well written and easy to understand.

Cons:
- The contrastive learning method employed in this paper lacks novelty.
- The performance improvement is marginal

Reviews: review: paper_summary: paper_summary: This paper introduce a novel framework named multi-modality co-learning(MMCL) to achieve efficient skeleton-based action recognition. By incorporating LLMs to generate textual descriptions of images for co-learning in training, this method empowers the mainstream GCN module to produce robust feature representations. Only skeletons GCN module is used for inference due to the effective co-learning. Extensive experiments shows the effectiveness of the MMCL framework and its efficacy.
strength_weakness: strength_weakness: 1. The MMCL framework is the first to introduce multi modal LLMs in skeleton-based action recognition.
2. The experiments are sufficient and have demonstrated the effectiveness of the proposed method.
limitations: limitations: 1. The writing of methodology needs to be improved. For example, the VQA module and  in eq.7 should be introduced in detail. And the formula of  in eq.10 is not mentioned.
2. The sec 3.5 mentions the MCCL have the ability of domain-adaptive, but there lacks of corresponding experiments to prove it.
3. Lacks of qualitative results to prove the effectiveness of multi-modality co-learning.
paper_summary: paper_summary: This paper proposes Multi-Modality Co-Learning (MMCL) for skeleton-based action recognition . It leverages a Feature Alignment Module (FAM) aligning RGB video features with skeleton data through contrastive learning, and a Feature Refinement Module (FRM) utilizing RGB images and textual instructions to refine classification scores.
strength_weakness: strength_weakness: 1. The paper is clear and easy to understand.
2. The figures in the paper are well-designed and aid in comprehension.
3. The experiments are comprehensive, and the results demonstrate the effectiveness of the proposed method.
limitations: limitations: 1. While it is acknowledged that multimodal learning is a recent trend, the reliance on dataset creation and acquisition for RGB video modality in skeleton-based action recognition is not sufficiently addressed. Considering recent generative methods such as diffusion, which can effectively expand skeleton data. This limitation affects the scalability of the method for skeleton-based action recognition tasks.
2. The main components of the MMCL model seem to rely on existing methods, lacking innovation. For instance, the core innovation of Multi-Modality Co-Learning, primarily achieved through contrastive learning, the design draws heavily from the approach outlined in reference [56] LST.
3. Figure 2 does not effectively showcase the superiority of the model. Since the eight categories depicted in Figure 2 are already selected as the top-performing results, it doesn't provide necessary insights into the model's performance.
paper_summary: paper_summary: This study introduces MMCL, a framework for skeleton-based action recognition. The proposed MMCL uses large language models (LLMs) as auxiliary networks during training. This co-learning approach leverages both skeleton and multimodal data. MMCL relies solely on skeleton data. The framework consists of two modules: one that aligns visual features from video frames with skeleton features, and another that uses LLMs to generate instructive text features from RGB images, temporal information, and text instructions. These features refine classification scores, enhancing the model's robustness and generalization. Experiments are conducted on NTU RGB+D, NTU RGB+D 120 and Northwestern-UCLA benchmarks, also the UTD-MHAD and SYSU-Action datasets for zero-shot and adaptive action recognition.
strength_weakness: strength_weakness: The paper explores the multimodal contrastive learning methods for the research on skeleton-based action recognition. . 

The experiments are conducted with multiple datasets, NTU RGB+D, NTU RGB+D 120 and Northwestern-UCLA, also the UTD-MHAD and SYSU-Action datasets. 

The experiment part showcases sufficient ablation study and analysis of the proposed methods in different perspectives.
limitations: limitations: The contrastive learning method employed in FAM is based on the framework presented in [68].  However, the paper lacks a clear explanation of the differences between the proposed method and [68].  A detailed description of the technical contributions specific to FAM is necessary to understand the novelty of the contrastive learning approach within this context.

The FRM relies on visual question answering (VQA) as the text-based task. Exploring alternative approaches, such as video captioning, referring, or summarization, maybe improve this module. Additionally, investigating the use of multi-tasking for feature alignment, rather than relying solely on VQA, is a possible plus.

The paper focuses on using Graph Convolutional Networks (GCN) as the backbone architecture.  While this is a reasonable choice, hwo about exploring the performance and effectiveness of applying the proposed methods (FAM and FRM) to other backbone types, particularly transformer-based architectures. This would provide insights into the generalizability of the approach.

Table 2 shows that MMCL achieves only marginal improvements compared to existing methods that use multi-modal data during training and pose data during inference. This raises concerns about the overall effectiveness of the co-learning approach.

A clear baseline for the proposed model is still missing. Table 2 lacks information on the results achievable by using both pose data for training and inference. Establishing this baseline is essential for evaluating the true impact of approach.

The paper does not explore the performance of MMCL when using multi-modal data during the inference stage. Understanding how the model behaves when incorporating both modalities throughout the process would provide valuable insights into its overall efficiency and effectiveness, particularly regarding potential improvements or trade-offs compared to the proposed co-learning approach.
paper_summary: paper_summary: This paper presents multi-modality co-learning (MMCL) framework for skeleton-based action recognition by adopting additional RGB video and text modalities. During training, RGB features are aligned with skeleton features to facilitate contrastive learning.
strength_weakness: strength_weakness: The paper is well-written. 

The proposed method achieves the best results on NTU 60, NTU 120, and NW-UCLA datasets, albeit not significantly higher than SkeletonGCL or Koopman.
limitations: limitations: Using RGB modality to assist in the learning of skeleton modality can be considered somewhat innovative, but the significant difference between RGB modality and skeleton modality may mean that simply extracting features and aligning them may not improve the experimental results. 

The author's claim of Multi-Modality Co-Learning simply involves stacking RGB videos, text, and skeleton modalities, extracting features and aligning them in a straightforward manner. The method lacks significant highlights and is not implemented in a sufficiently detailed manner.

While there are no grammatical errors in the writing, the writting can be further improved, for example, placing experimental results in the Method section.