2M-AF: A Strong Multi-Modality Framework For Human Action Quality Assessment with Self-supervised Representation Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 04:58:01

Abstract: Human Action Quality Assessment (AQA) is a prominent area of research in human action analysis. Current mainstream methods only consider the RGB modality which results in limited feature representation and insufficient performance due to the complexity of the AQA task. In this paper, we propose a simple and modular framework called the Two-Modality Assessment Framework (2M-AF), which comprises a skeleton stream, an RGB stream and a regression module. For the skeleton stream, we develop the Self-supervised Mask Encoder Graph Convolution Network (SME-GCN) to achieve representation learning, and further implement score assessment.  Additionally, we propose a Preference Fusion Module (PFM) to fuse features, which can effectively avoid the disadvantages of different modalities. Our experimental results demonstrate the superiority of the proposed 2M-AF over current state-of-the-art methods on three publicly available datasets: AQA-7, UNLV-Diving, and MMFS-63.

Main paper: 
 introduction: 
 Action quality assessment (AQA) models have witnessed a surge in recent years due to their importance in various applications [1, 2, 7, 12] . Compared with traditional tasks such as classification or segmentation [5, 8] , the AQA task presents more challenges. Specifically, human actions for the AQA task usually have the same background and action type but significantly different degrees of completion, which cause the networks for conventional action tasks with inadequate discrimination power in the AQA task [23] .To address this challenge, researchers have explored improvements in feature capture [19, 23] and regression modules [28, 29] . However, achieving a breakthrough on challenging datasets remains elusive, which showcases the limitations of RGB modality. RGB data contains rich environmental and colour information, but lying in the limited representation of spatial relationships and sensitivity to external factors, the RGB methods are hard to capture the important features of human actions. The result reflected in the AQA task is that the RGB modality struggles to achieve better performance in some environmentally irrelevant actions (e.g. gym, figure skating) because it is not only highly redundant in its features, but also more sensitive to subtle motion variations. To solve this problem, recent methods [28, 29] had to use complex regression structures, which led to complex calculations. Based on the above problem, we consider the introduction of another modality to ameliorate the defects of RGBs.As a comparison to the RGB modality, the skeleton modality showcases different advantages for video tasks, including its intuitive spatial representation and robust feature specificity [6, 27] . Futhermore, the skeleton-based models offer significantly fewer parameters and faster computational speed than RGB ones [4] . But the flaw in skeleton data is that all environmental information is negelected, some of which may be crucial (e.g. splashes in diving sports). Therefore, designing a method to harness the advantages of both modalities will be effective. However, previous mainstream AQA researches seem to have little interest in skeleton as it seems not perform as well as RGB in unimodal states. Consequently, the purpose of our work is to design a competitive skeleton-based network and further build a multi-modality, simple and efficient AQA framework.In this paper, we propose a Self-supervised Mask Encoder Graph Convolution Network (SME-GCN) to achieve representation learning for skeleton data. SME-GCN separately preserves the dynamic and temporal characteristics of skeleton data by developing two masking strategies to achieve an additional emphasis on the motion and obtain high-quality contrastive sequences. Building upon these masked data, SME-GCN then employs a siamese network and contrastive loss for self-supervised learning. By simply attaching a regression structure, SME-GCN improved the performance of the skeleton model on the AQA task, which reflects the excellent utilization of skeleton characteristics.For the feature regression in action-based multi-modality methods, a common idea is to share the information in one or more layer(s) [13, 15] . These methods tend to introduce more extra computation in multi-modality calculations. In the AQA task, since almost each action has at least one modality capable of achieving good performance, we consider the feasibility of a new fusion approach that trains each sample to automatically select a more appropriate result from two modalities. Therefore, we design Preference Fusion Module (PFM) to choose the modality which is more suited for different action samples. Combining our SME-GCN, PFM and a simple RGB stream (I3D), we propose the Two-Modality Assessment Framework (2M-AF). It is worth noting that multiple backbones and regression losses can be configured within the modular 2M-AF, which exhibits the flexibility of our work. Our experimental results show that our 2M-AF has significantly surpassed the previous state-of-the-art methods on AQA-7, UNLV-Diving and MMFS-63 datasets.The key contributions we make are as follows: 1. We propose a skeleton-based self-supervised representation learning network named SME-GCN which incorporates a unique masking strategy designed specifically for the characteristics of skeleton data. To the best of our knowledge, we are the first to apply self-supervised representation learning to the AQA task and achieve performance comparable to mainstream RGB approaches.2. We propose a simple and efficient PFM module which can effectively avoid the disadvantages of different modalities and enhance the quality of multi-modality fusion without excessive hyperparameter tuning.3. By combining our SME-GCN, PFM module and an RGB stream, we propose a Multi-Modality Action Quality Assessment Framework called 2M-AF. Our experimental results show that 2M-AF outperforms the State-of-the-art methods on three competitive datasets, AQA-7, UNLV-Diving and MMFS-63. 
related works 2.1 action quality assessment: 
 RGB-based methods are the primary approaches used in current AQA tasks. The traditional methods [19, 21, 23, 26] utilize feature extractors like C3D or I3D to obtain features from videos and the final scores are generated through an LSTM or MLP regress module. However, due to the high requirements of feature discriminability in the AQA task, the previous methods have failed to achieve satisfactory results. Thus, feature regress modules emerge as a new research branch. C3D-AVG-MTL [20] adopts multi-task learning, simultaneously performing assessment, text generation and recognition tasks, but mutual interference by multi-task leads to an undesirable performance. CoRe-based methods [28, 29] utilize contrast learning to reduce the difficulty of scoring by comparing samples of the same class. However, training contrastive learning methods relies on additional category annotations and extra storage spaces. Recently, attention mechanisms have been utilized to highlight assessment-related information in features. TSA-Net [24] proposed a spatial-temporal attention module called Tube to reduce the detrimental effects of redundant information. Overall, researchers have done considerable work on RGB data, but they have still not made ideal progress, which has led us to consider adding extra modality information to the AQA model.Self-supervised representation learning aims to obtain high-quality features from unlabeled samples. Early methods generated features through pretext tasks [16, 25] . Then, self-supervised learning rapidly progressed and achieve remarkable performance, which is comparable to supervised learning. Some methods [3, 11] utilize contrastive learning by constructing pairs of samples to learn the correlations between samples and enable the backbone network to acquire rich discriminative representations. Building on this foundation, in the past two years, some methods [9, 32] have started to explore the application of self-supervised learning in skeleton-based action recognition but fail to consider the unique spatial-temporal relationship of joints. After that, MAE [10] , which adopts pixel masking to reconstruct images using encoder-decoder structures, lets researchers realize the importance of masking. As an improvement, PSTL [30] proposes a spatial-temporal masking strategy to generate pairs of skeleton samples. Overall, the masking strategy is primarily aimed at incorporating variability information while preserving key features. However, due to the complexity and atypical nature of AQA actions, masking at the skeleton dimension can easily compromise the fine details of the actions themselves. Therefore, the transfer learning of methods like PSTL is limited in terms of performance for the AQA task, which promotes us to build a self-supervised network more fitting our assignment. 
methods 3.1 self-supervised mask encoder graph convolution network: 
 As Figure 1 (4) shows, the original skeleton sequence will be transformed into two new sequences using the frame-level masking strategies. Then, the two sequences will be processed by the encoder with shared weights. Finally, the model will achieve self-supervised learning using contrast loss. Masking strategy. A 3D human skeleton sequence can be denoted as 𝑥 ∈ R 𝐶 ×𝑇 ×𝑉 , which has C channels, T frames, and V joints. As the input of SME-GCN, we define skeleton sequences as 𝑥 𝑠 ∈ R 𝑁 ×𝐶 ×𝑇 ×𝑉 , where 𝑁 is the batch size, and 𝑥 𝑠 𝑖 ∈ R 𝐶 ×𝑇 ×𝑉 represents the 𝑖 𝑡ℎ sequence in 𝑥 𝑠 .In self-supervised tasks, the first and most crucial step is to construct partial skeleton data, which should retain sufficient key information to preserve the original characteristics of the actions. Our masking strategy pay extra attention to the spatial-temporal properties of the skeleton. Our variance mask module aims to focus on the shift within actions. First, we subtract the coordinates of each joint from the coordinates of the central joint, which provides a standardized measure: 𝑥 𝑠 𝑖,:,:,: = 𝑥 𝑠 𝑖,:,:,: -𝑥 𝑠 𝑖,:,:,𝑚 ,where 𝑥 𝑠 𝑖,:,:,𝑚 denotes the centre joint 𝑚 of 𝑥 𝑠 𝑖 , we recommend selecting the joint closer to the human centre as the central joint (for HRNet, we choose 𝑚 = 8). Next, we define the alteration degree 𝐷 𝑖 (𝑡) to obtain a measure of change for the 𝑡 𝑡ℎ frame of the 𝑖 𝑡ℎ skeleton:EQUATION)where 𝑘 is a hyperparameter that represents the range of perception, and 𝑥 𝑠 𝑖,𝑐,𝑗,𝑣 indicates the 𝑐 𝑡ℎ channel, 𝑡 𝑡ℎ frame and 𝑣 𝑡ℎ joint of 𝑥 𝑠 𝑖 , 𝑥 𝑠 𝑖,𝑐,𝑡,𝑣 denotes the average of the data from the (𝑡 -𝑘) 𝑡ℎ to (𝑡 + 𝑘) 𝑡ℎ frames (if the index is a valid number). By calculating the variances between frames and their surrounding 𝑘 frames, we can mask out skeleton frames which have lower change. We sort the index list of the 𝑥 𝑠 𝑖,𝑡 in ascending order based on 𝐷 𝑖 (𝑡). As we define the mask rate 𝑟 𝑚 , we acquire the (𝑟 𝑚 )𝑇 𝑡ℎ index and label its 𝐷 𝑖 (𝑡) as 𝛼. The variance masked sequence x𝑠 𝑖 is represented as: Our strategy obtains strong data while preserving the dynamic characteristics of the original data. In contrast, the temporal mask strategy is primarily used to preserve the temporal characteristics of the data. Therefore, the new sequence 𝑥 𝑠 𝑖 is formed by applying the interval mask based on 𝑟 𝑚 . In summary, we have generated two samples which are preserving key characteristics from the original data. This lays a crucial foundation for improving the performance of our self-supervised learning.Encoder and loss function. As a modular approach, various skeleton encoders can be configured with SME-GCN. After conducting extensive experiments, we ultimately choose the graph convolutional part of CTR-GCN [4] and incorporate a fully connected layer as the encoder in SME-GCN. The sequences x𝑠 𝑖 and 𝑥 𝑠 𝑖 will be passed into the weight-shared encoder to generate feature vector pair, f 𝑠 𝑖 ∈ R 𝐶 𝑠 and 𝑓 𝑠 𝑖 ∈ R 𝐶 𝑠 (𝐶 𝑠 is the number of the output channel). For ease of description, we merge 𝑁 pairs of f 𝑠 𝑖 and 𝑓 𝑠 𝑖 into 𝑓 𝑚 ∈ R 2𝑁 ×𝐶 𝑠 by placing all 𝑓 𝑠 𝑖 after f 𝑠 𝑖 . With similarity measured, InfoNCE [17] is adapted as the contrast loss of our method:EQUATIONwhere 𝑠𝑖𝑚(., .) indicates the product of the 𝑙 2 normalised inputs and 𝜏 denotes a constant temperature parameter. As a robust method, our SME-GCN can be utilized as the skeleton backbone for multiple human action tasks. In addition to our main focused AQA task in this paper, we also explored the effectiveness of SME-GCN on recognition tasks in our experiments.SME-GCN for AQA. By appending a linear regress module on top of the encoder and then finetuning the entire network on the target AQA dataset, we build a skeleton-based network for the AQA task. The overall architecture of our 2M-AF is shown in Figure 1 . In the skeleton stream, the input frames are sent into HRNet [22] to estimate skeleton data. The pretrained SME-GCN is utilized as the backbone of the skeleton stream to obtain the skeleton feature 𝐹 𝑠 ∈ R 𝑁 ×𝐶 𝑠 . The RGB stream of 2M-AF. In the RGB stream, the input frames are divided into 𝑛 small clips (𝑛 is adjusted based on the length of the dataset). Then the clips are sent into I3D ConvNets for extracting features. Finally, we calculate the average of the features to realize the RGB feature 𝐹 𝑟 ∈ R 𝑁 ×𝐶 𝑟 , where 𝐶 𝑟 represents the number of the output channel.Preference Fusion Module. Our Preference Fusion Module (PFM) consists of two regression heads and a selection head to process the RGB and the skeleton features, 𝐹 𝑟 and 𝐹 𝑠 . (1) In the regression heads, 𝐹 𝑟 and 𝐹 𝑠 separately regress the predicted score, 𝑆 𝑟 ∈ R 𝑁 and 𝑆 𝑠 ∈ R 𝑁 by a fully connected layer. (2) The selection head is the core of PFM. We concatenate 𝐹 𝑟 and 𝐹 𝑠 and then send the merged feature to a fully connected layer and a Softmax layer to acquire a two-category vector 𝐹 𝑐 ∈ R 𝑁 ×2 to compare with the preference label 𝑦 𝑐 , which is formed by calculating the L1 distance between the predicted score of each stream and the ground truth score. 𝑦 𝑐 can be presented as:EQUATIONwhere 𝑦 𝑐 𝑖 is the 𝑖 𝑡ℎ sample of 𝑦 𝑐 , and 𝑆 𝑠 𝑖 , 𝑆 𝑟 𝑖 denotes the predicted score of the 𝑖 𝑡ℎ skeleton sample and RGB sample, respectively. To enable backpropagation, our loss function L combines both regression and classification losses. Motivated by MTL-AQA [20] , our regression loss L 𝑠 for the skeleton adapted L1 distance in addition to MSE:EQUATIONwhere 𝑦 𝑖 is the ground truth score, and 𝜆 1 and 𝜆 2 is represented as the weights for the two metrics. The loss function L 𝑟 for the RGB stream is the same as L 𝑠 .EQUATIONWe calculate the loss function separately for both streams and then sum them up to obtain the loss of the regression part. For the classification, based on the preference label 𝑦 𝑐 and the output vector 𝐹 𝑐 , we utilize cross-entropy loss L 𝑐 to iterate parameters:EQUATIONwhere 𝐹 𝑐 𝑖,𝑘 indicates the 𝑖 𝑡ℎ sample and the 𝑘 𝑡ℎ feature of 𝐹 𝑐 . During the training process, the complete loss function L is obtained by adding up the aforementioned losses:EQUATIONIn testing, PFM selects the scores based on the classification results as the final predicted score. As a result, PFM facilitates independent regression of the two streams and possesses the ability to select the more accurate result for each sample from outputs. Considering all the aforesaid work, our 2M-AF solution addresses the challenge of fusing multimodal networks with attribute differences. 
experiments 4.1 datasets: 
 In this section, we conducted experiments on three datasets with different characteristics and frame lengths to demonstrate the generalizability of our framework. AQA-7 [19] is currently the most widely used AQA dataset, which also has the most diverse range of action categories. In contrast, UNLV-Diving [21] and MMFS-63 [14] have longer frame sequences and consist of actions from a single sport. Additionally, MMFS-63 offers a large number of variablelength frame sequences and pre-processed, high-quality skeleton data. Consequently, experiments on the above three datasets which have different scales, frame lengths and adaptability to two modalities are conducted to showcase the strong generalization ability of our framework. To maintain consistency with existing literature, Spearman's rank correlation coefficient is employed as the measurement standard. (The detailed experimental configuration and download link of the three datasets is written in the Appendix.) AQA-7. The AQA-7 dataset comprises samples from seven actions. It contains 1189 videos with 103 frame lengths, of which 803 videos are used for training and 303 videos are used for testing. To ensure a fair comparison with other methods, we remove the trampoline category the same as the previous works and calculate the performance of the other six categories. To compare with the mainstream methods, Fisher's z-value [19] is used to measure the average performance across actions. In order to avoid potential bias towards certain categories in the results and demonstrate the performance of our method on a large-scale, multi-category dataset, we additionally trained our model using all six categories of AQA-7 data in comparing with the current leading method and the ablation study.UNLV-Diving. The UNLV-Diving dataset contains 370 video clips with various diving actions, of which 300 videos are used for training and 70 videos are used for testing. The length of each video is 151 frames. According to the dataset description, we use the execution score of actions as the evaluation label.MMFS-63. MMFS-63 is a large AQA dataset that consists of 63 classes of actions, which were collected from the World Figure Skating Championships. There are a total of 4915 samples for both skeleton and RGB data, of which 3959 samples are for training and 956 for testing.Evaluation Protocols. To maintain consistency with existing literature, we employed Spearman's rank correlation coefficient, which ranges from -1 to 1 (with higher values indicating better performance), to evaluate the accuracy of our methods in predicting the score series compared to the ground-truth data. As the metrics, Spearman's correlation coefficient is defined as follows:EQUATION)Here 𝑝 𝑖 and 𝑞 𝑖 represent the ranking of the 𝑖 𝑡ℎ predicted score and sample score series, respectively. And 𝑝 and 𝑞 denote the average of the two score series respectively. 𝑁 is the length of the series. C3D-SVR [21] 0.7800 JRG [18] 0.7630 C3D-AVG-MTL [20] 0.8383 FALCONS [15] 0.8453 CoRe [28] 0.8663 DAE-CoRe [29] 0.8477 2M-AF (ours) 0.8838 We compare our method with the state-of-the-art AQA methods on all three datasets introduced above. As results are shown in Table 1 , we achieved advantages in five out of the six single categories on AQA-7, and in the other specific single category, 'Diving', we are still close to the best result. As results are shown in Tables 1, 2 and 3, our 2M-AF achieves state-of-the-art performance with a large margin on the three datasets. The results demonstrate the excellent robustness of 2M-AF to different types, scales and frame lengths of data.Sp. Corr.0.5234 C3D-AVG-STL [20] 0.3831 CoRe [28] 0.7313 DAE-MLP [29] 0.5915 DAE-CoRe [29] 0.7548 2M-AF (ours) 0.8744 To validate the performance of each module through ablation experiments, firstly, we discuss the significance of introducing multimodality. Then, we analyze the competence of SME-GCN from the aspects of masking rate, the effectiveness of the masking strategy, and performance compared with other self-supervised methods. Lastly, we demonstrate the capability of PFM by comparing it with other fusion methods. The experiments on the AQA-7 dataset utilize all six categories (like 'All' in Table 1 ).Comparing between different modalities. To demonstrate the performance on different modalities, we discuss the comparison between the RGB modality, skeleton modality, simple modality fusion (using a fully connected layer after concatenating the features), and PFM on AQA-7, UNLV-Diving and MMFS-63 datasets. To ensure a fair comparison, we set the RGB modality to use the I3D ConvNet while the skeleton modality is uniformly implemented using CTR-GCN. As experimental results are shown in Table 4 , (1) different datasets exhibit distinct inclinations towards modalities. Both the skeleton modality and RGB modality perform well on AQA-7 because of its diverse range of action categories, while on MMFS-63, the skeleton modality exhibits a clear advantage, and on UNLV-Diving, the RGB modality performs better. This reflects that different data exhibit significant differences in performance across different modalities, highlighting the importance of multi-modality methods in the context of the AQA task. (2) for the modality fusion, we observe that the direct fusion strategy performs not only lower than PFM but also than the single modality, especially when there is a large difference between the performance of the two modalities (on MMFS-63 and UNLV-Diving). This aligns with our previous analysis, as the vast information disparity, simply concatenating features and using fully connected layers is inappropriate.Configuration of the skeleton backbone. To choose a better skeleton backbone, we compared several mainstream skeleton networks to select the most suitable skeleton backbone for our 2M-AF. For equal comparison, we added a single MLP layer on each skeleton network to regress and predict scores for all skeleton networks. The experimental results in 5 demonstrate that CTR-GCN is a more suitable skeleton backbone. [9] 0.8706 PSTL [30] 0.8699 temporal+variance 0.8848 Table 7 : Comparisons of accuracies when utilizing different contrastive strategies on the AQA-7 dataset.The masking rate configuration of SME-GCN. As a selfsupervised work designed for the AQA task, we compare the performance of SME-GCN under different configurations of 𝑟 𝑚 and only CTR-GCN backbone [4] (baseline in Table 6 ). Due to each video in AQA-7 consisting of 103 frames, we selected three sets of masking configurations 𝑟 𝑚 = 0.84, 𝑟 𝑚 = 0.69, 𝑟 𝑚 = 0.38 to retain 16, 32, and 64 frames of valid data. As experimental results are shown in Table 6 , regardless of the value of 𝑟 𝑚 , SME-GCN outperforms the regular CTR-GCN, demonstrating the effectiveness of our self-supervised strategy. Additionally, the results are even better when 𝑟 𝑚 = 0.38. We believe that this level of masking introduces sufficient variations to the data without compromising important features.Effects of our masking strategies for SME-GCN. In Table 7 , we conducted ablation experiments on different masking strategies of SME-GCN. (1) We first compared our own masking strategies. The 'random' strategy indicates masking random frames, 'temporal' denotes temporal masking and 'variance' represents variance masking. The masking rate is set as 𝑟 𝑚 = 0.38 considering the above experiment. The experimental results demonstrate that including each masking strategy improves the model's accuracy. (2) Next, we compared our SME-GCN with other skeleton-based self-supervised representation masking strategies. As the result is shown, with the same CTR-GCN backbone, all self-supervised strategies improve the performance of downstream AQA tasks. This result validates our analysis that self-supervised learning can address the limitations of skeleton models, while our method achieves superior results compared to other methods due to the emphasis on preserving crucial spatial-temporal information in the features. The effects of PFM. To validate the performance of PFM, we compared its performance with mainstream regression heads which combine RGB and skeleton modalities. All experiments were conducted on the AQA-7 dataset, using CTR-GCN backbone for the skeleton modality and I3D backbone for the RGB modality. The experimental results are shown in Table 8 . Compared to settings A, B, C and E, PFM outperforms mainstream regression heads ( [15, 31] ), which indicates our method is superior to feature fusion approaches. Additionally, we conducted additional tests by adding a regression head that concatenates features and uses an FC layer to let PFM achieve selection from three outputs (Setting D). The experimental results show that this configuration does not improve performance. We are convinced this is due to the decrease in accuracy caused by the excessive redundancy of features.The configuration of loss functions for 2M-AF. We compared several commonly used regression loss functions to determine the most suitable method for AQA tasks and our framework. Considering the three configurations in As shown in Table 10 , we compared the efficiency of our method with the State-of-the-art CoRe-based structure [28, 29] on the AQA-7 dataset. We first conducted a separate analysis of the efficiency of our skeleton network SME-GCN and found that it accounts for only about 1/5 Params and 1/10 FLOPs of the overall framework 2M-AF. This indicates that combining the skeleton modality and RGB modality will not result in an excessively large number of parameters. Additionally, we observe that our 2M-AF utilizes similar parameters but is more efficient in speed compared to the current mainstream solution, which means our 2M-AF no longer need complex regression structures but can achieve more outstanding performance. To have an intuitive understanding of the differences between our single skeleton stream, single RGB stream, and 2M-AF, we visualize the prediction results in the form of a scatter plot in Figure 2 . From the graph, we can observe the following in the prediction of AQA-7: (1) The skeleton stream has more outlier points (such as the points in the black circle) compared with other plots. This means the skeleton model performs better on the majority of samples, but it produces noticeable errors on a small portion of samples, which may cause by the poor quality of the skeleton data for a few samples.(2) In contrast, the RGB stream shows many samples in the red circle deviations from the perfect predictions, suggesting its limited regression performance for dense samples. (3) Our 2M-AF successfully addresses the above two issues, which reflects the effectiveness of our PFM in selecting more accurate predictions from both streams.To better illustrate the significant differences in the tendency of different samples towards modalities, we selected a few examples for demonstration in Figure 3 . It can be observed that: (1) for diving actions 'a' and 'b', RGB data shows a clear advantage due to the significant influence of environmental features such as splashes on the quality assessment. As well, frames falling into the water are difficult for the skeleton to estimate, which makes the skeleton stream limited. ( 2) for skating actions 'c' and 'd', the skeleton stream demonstrates a noticeable advantage. This is attributed to the fact that skating actions rely more on spatial relationships which is suited for the skeleton methods, but the RGB methods will be badly subject to environmental factors.The feature enhancement capability of SME-GCN. In order to showcase the feature enhancement capability of SME-GCN, we additionally utilized the SME-GCN encoder with an added classification head for the action recognition task. Following our backbone CTR-GCN, we utilize the fusion results from four different skeleton modalities. The results are presented in Table 11 . The experimental results demonstrate that SME-GCN achieved improvements in three branches of skeleton modalities and the final result, indicates that our self-supervised structure also plays a crucial role in feature enhancement for the classification task. We used the T-SNE method to visualize the features of the aforementioned classification task. Points with different colours represent different categories, and the results are shown in Figure 4 . It The feature enhancement capability by different mask rate on SME-GCN. In order to showcase the feature enhancement capability of SME-GCN for multiple tasks, we additionally utilized the SME-GCN encoder with an added classification head for the action recognition task. The results are presented in Table 12 . At different mask rates, SME-GCN achieves excellent performance at mask rates up to 0.25-0.5, and the performance degradation when the mask rate is higher is caused by the loss of too much information. The experimental results demonstrate that SME-GCN can achieve improvements on multiple tasks, which shows strong generalization capabilities.  
conclusion: 
 This paper proposes a framework for the AQA task named 2M-AF, which is powerful for providing richer features and facilitating the utilization of multi-modality information. The framework comprises a self-supervised graph convolutional skeleton network called SME-GCN, an RGB backbone, and the proposed PFM module.The experimental results demonstrate that our method outperforms the existing approaches on the three challenging datasets. As a simple and flexible framework, we believe that the value of 2M-AF lies not only in its significant improvement in accuracy but also in its compatibility with various components, which provides endless possibilities for research built on the foundation of 2M-AF. 
 
 Appendix: 2M-AF: A Strong Multi-Modality Framework For Human Action Quality Assessment with Self-supervised Representation Learning ACM MM, 2024, Melbourne, Australia 


Decision: We would like to thank the authors for answering the questions by the reviewers in the rebuttal. Reviewers have considered your responses. Reviewers are generally positive towards this paper. Also, some concerns remain: (1) The novelty is limited. (2) Please check the statements and ensure they are precise. (3) Please ensure the comparisons are fair and carefully proofread the paper. This paper is recommended for accept, where authors should address (2) and (3) above in the final version.

Reviews: review: paper_summary: paper_summary: This paper presents a multi-modality framework for action quality assessment. The authors proposed to determine the action quality by integrating skeleton and RGB feature frames, and the skeleton features are enhanced in a self-supervised manner.
strength_weakness: strength_weakness: The paper is well written and organized. Experiments are sufficient and comprehensive to illustrate the effectiveness of the proposed framework.
limitations: limitations: 1. The novelty of this paper is limited. Self-supervised learning for skeleton data has been studied and applied in LongTGAN [1] and MS^2L [2].

[1] Unsupervised Representation Learning With Long-Term Dynamics for Skeleton Based Action Recognition. AAAI 2018.

[2] MS^2L: Multi-task Self-supervised Learning for Skeleton Based Action Recognition", ACM MM 2020. 

The authors are suggested to elaborate the difference with the previous works in self-supervised learning.
The multi-modality fusion is not new in action analytics [3][4] etc.

[3] Temporal Segment Networks: Towards Good Practices for Deep Action Recognition. ECCV 2016.

[4] Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection. TCSVT 2020.

2. In Table 1,2,3, the authors are suggested to indicate the modalities each method employs for a fair comparison.

3. In Table 12, it’s interesting to see action recognition accuracy gets improved as mask rate increases when mask rate < 0.5 The authors are suggested to give more explaination on it. 

4. Minor: In Eq. (5), y_i is not explained until Eq.(6).
paper_summary: paper_summary: The paper introduces a novel dual-modality framework, 2M-AF, which enhances the performance of human action quality assessment by integrating a skeleton stream with the traditional RGB-based approach. This work uses a Preference Fusion Module (PFM) to adaptively select results from both modalities, achieving state-of-the-art results on mainstream datasets.
strength_weakness: strength_weakness: 1. The introduction of a skeleton stream alongside the traditional RGB stream is a significant enhancement, particularly given the self-supervised learning strategy employed to enable effective feature extraction from the skeleton data. This dual-modality approach effectively addresses the shortcomings of RGB-only systems in some environmentally irrelevant scenarios.
2. The method's superior performance on benchmark datasets confirms its efficacy and the practical value of integrating multiple modalities for action quality assessment.
limitations: limitations: ### Major weaknesses
1. **Incomplete Review of Related Work:** 
The paper discusses RGB-based action quality assessment methods but lacks a comprehensive review of existing skeleton-based approaches, which are also relevant to this research. 
For instance, the study "Skeleton Based Action Quality Assessment of Figure Skating Videos[1]" should be discussed to highlight how the proposed method differs from or improves upon these approaches.
Including a broader range of related works would provide a more thorough context for your contributions and help clarify the novelty of your approach.

2. **About PFM：**
Table 8 lacks references to the methods BPAN and FALCONS, nor are they introduced in the PFM effectiveness subsection. Additionally, PFM essentially represents a hard selection for two-modality streams. How would the effect change if we opt for a soft selection by using the predicted scores from PFM for weighted summation of the two streams?

### Minor weakness
There is a typo in line 576: "results in 5" -> "results in Table 5"


[1] Li, Hui-Ying, et al. "Skeleton based action quality assessment of figure skating videos." 2021 11th International Conference on Information Technology in Medicine and Education (ITME). IEEE, 2021.
paper_summary: paper_summary: This paper designs a competitive skeleton-based network and further build a multi-modality, simple and efficient AQA framework. It also proposes a Self-supervised Mask Encoder Graph Convolution Network (SME-GCN) to achieve representation learning for skeleton data.
strength_weakness: strength_weakness: Overall, the paper is well-organized, and the technical contributions appear to be substantial.
limitations: limitations: 1.It is recommended that the authors consider making the source code publicly available.
2.In the contributions, the authors state, "To the best of our knowledge, we are the first to apply self-supervised representation learning to the AQA task." However, in the related work, it is mentioned that "some methods have started to explore the application of self-supervised learning in skeleton-based action recognition." Therefore, I believe the authors' statement is not precise.
3.The paper compares different models for both performance and efficiency. Why were two different models chosen for comparison?
4.Ablation experiments aim to investigate the impact of adding or removing specific modules from the model on overall performance. However, in Section 4.3 of this paper, the ablation experiments discuss "The masking rate configuration of SME-GCN" and "The configuration of loss functions for 2M-AF."
5.There are some grammatical mistakes. For example, "Then, self-supervised learning rapidly progressed and achieve remarkable performance."