Towards Video-based Activated Muscle Group Estimation in the Wild

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 21:12:36

Abstract: In this paper, we tackle the new task of video-based Activated Muscle Group Estimation (AMGE) aiming at identifying active muscle regions during physical activity in the wild. To this intent, we provide the MuscleMap dataset featuring >15K video clips with 135 different activities and 20 labeled muscle groups. This dataset opens the vistas to multiple video-based applications in sports and rehabilitation medicine under flexible environment constraints. The proposed MuscleMap dataset is constructed with YouTube videos, specifically targeting High-Intensity Interval Training (HIIT) physical exercise in the wild. To make the AMGE model applicable in real-life situations, it is crucial to ensure that the model can generalize well to numerous types of physical activities not present during training and involving new combinations of activated muscles. To achieve this, our benchmark also covers an evaluation setting where the model is exposed to activity types excluded from the training set. Our experiments reveal that the generalizability of existing architectures adapted for the AMGE task remains a challenge. Therefore, we also propose a new approach, TransM3E, which employs a multi-modality feature fusion mechanism between both the video transformer model and the skeleton-based graph convolution model with novel cross-modal knowledge distillation executed on multi-classification tokens. The proposed method surpasses all popular video classification models when dealing with both, previously seen and new types of physical activities. The contributed dataset and code will be publicly available.

Main paper: 
 introduction: 
 Human activity understanding is important as it enables the development of applications and systems that can enhance healthcare, improve security, and optimize various aspects of daily life by automatically identifying and understanding human actions and behaviors [1, 30, 39, 68, 71, 89] . Knowing which skeletal muscles of the human body are activated benefits human activity understanding, and sport and rehabilitation medicine from multiple perspectives and prevents inappropriate muscle usage which may cause physical injuries [20] . In health care, patients need to know how to conduct the exercise correctly to recover from surgery [35] or specific diseases [3] , e.g., COVID-19 [58] . Knowledge about muscle activations allows for user-centric fitness applications providing insights for everyday users or professional athletes who need specially adapted training. The majority of existing work on Activated Muscle Group Estimation (AMGE) is based on wearable devices with electrode sensors [15] . Yet, many wearable devices are inconvenient and heavy [41] , even harmful to health [4] , and have limited usage time due to the battery [64] . A big strength of wearable devices is the high accuracy achieved through direct signal measurement from skin or muscle tissue. However, such exact bio-electrical changes are not required in a large number of medical recovery programs, and knowing the binary activation status of the muscle as shown in Figure 1 is sufficient in many situations [47, 61, 78] . In contrast to wearable devices, most people have a video camera available at hand on their phone or laptop. Applying video-based AMGE on in-thewild data collected by using smartphones or other widely available smart devices would allow for the application of such programs even without access to specialized hardware. Thereby, end-to-end videobased AMGE approaches are expected to be developed to prevent overburdens caused by wearable devices from both physical and psychological points of view. Can modern deep learning algorithms relate fine-grained physical movements to individual muscles? To answer this question, we tackle the barely researched task of videobased active muscle group estimation under an in-the-wild setting, which estimates muscle contraction during physical activities from video recordings without a restricted environment and background constraints.Current research in video-based AMGE is limited by small-scale datasets and constrained data collection settings [13] , where the data is often annotated with sensor signals and confined to restricted environments, covering only a limited range of actions. However, with the expansion of deep learning model capacities, there is a pressing need for larger datasets encompassing a wider variety of environments and activities. This expansion is vital for advancing the field of video-based AMGE within the research community.In this work, we collect the first large-scale in-the-wild AMGE dataset from YouTube without environment constraints and give binary activation for different muscle regions by inquiring about sports field researchers. We created the MuscleMap dataset -a videobased dataset with 135 different exercises collected from YouTube considering in-the-wild videos. Each exercise type is annotated with one or multiple out of 20 different muscle group activations, as described in Table 1 , which opens the door for video-based activated muscle group estimation in the wild task to the community. We annotate the dataset in a multi-label manner since human body movement is produced by the coordinated operation of diverse muscle regions. To acquire such annotations, we ask two senior researchers in the biomedical and sports research field to give the annotations.We select various off-the-shelf Convolutional Neural Networks (CNNs) [8, 23] , Graph Convolutional Networks (GCNs) [11, 36, 81] , and transformer-based architectures [21, 40, 45] from human activity recognition field together with statistic methods as baselines. Our proposed MuscleMap benchmark describes a multi-label classification problem where each sample might be annotated with one or up to twenty labels. However, we find that it is challenging for all these models when they deal with new activity types containing new activated muscle combinations at test time considering the AMGE generalizability. Skeleton-based models are observed to show good performance on the new activity types while working not well on known activity types. The video-based models show good performance on the known activity types while delivering limited performance on the new activity types. An approach that can work well on both known and new activity types is thereby expected.To tackle the aforementioned issue, we propose TRANSM 3 E, a cross-modality knowledge distillation and fusion architecture that combines RGB and skeleton data via a new classification tokensbased knowledge distillation and fusion mechanism. To achieve better extraction of underlying cues for AMGE, we propose and equip TRANSM 3 E with three essential novel components, i.e., Multi-Classification Tokens (MCT), Multi-Classification Tokens Knowledge Distillation (MCTKD), and Multi-Classification Tokens Fusion (MCTF), atop the most competitive performing architecture MViTv2 [40] as the backbone. As it is fundamental to mine and predict the activities at the global level for AMGE, the proposed TRANSM 3 E, appearing as a transformer-based approach, is endowed with the capacity for long-term reasoning of visual transformers [76] . Since AMGE is a multi-label classification task, MCT is introduced, in view that using more classification tokens is expected to introduce more benefits toward finding informative cues. MCT also builds up the base for cross-modality MCT-level knowledge distillation.Knowledge distillation [29] is leveraged for cross-modality knowledge transfer after the feature map reduction of the transformer block to enable a more informative latent space learning for different modalities. Transferring cross-modality knowledge during training significantly benefits the model in finding out cross-modality informative cues for the AMGE task. However, we find that it is difficult to achieve the knowledge distillation between two models with obvious architecture differences, e.g., graph convolutional networks (GCNs) and video transformers, considering the alignment of the feature maps coming from different backbones and modalities to achieve the appropriate and effective knowledge distillation. Aside from the architectural differences, we examine that using late fusion to fuse the skeleton-based model and video-based model can not achieve a satisfactory performance due to the lack of alignment of the two different feature domains.We thereby propose a cross-modality MCT-level knowledge distillation scheme considering knowledge distillation on the intermediate and final layers by specifically designing a knowledge distillation MCT for the model of each modality. Alongside the MCT used solely for the classification, we leverage another MCT to execute the knowledge distillation for each modality. The cross-modality knowledge distillation is then executed only between the knowledge distillation MCTs from the two modalities, whereas existing works mostly use knowledge distillation calculated from the full embeddings or single knowledge distillation token at the final layer and use a larger teacher [22, 29, 42, 48, 75] . While the mentioned MC-TKD mechanism integrates cross-modal knowledge into our main network with additional MCT for the knowledge distillation, another contribution, MCTF, merges the MCT of the distilled knowledge and the MCT for classification to achieve a final prediction towards the active muscle regions during human body motion for each modality. By combining these three components, TRANSM 3 E achieves stateof-the-art performances with superior generalizability compared to the tested baselines.In summary, our contributions are listed as follows:• We open the vistas of video-based Activated Muscle Group Estimation in the wild task with the aim of lowering the threshold of entry to muscle-activation-based health care and sports applications.  
related work: 
 Activate Muscle Group Estimation (AMGE) analysis is predominantly performed using electromyographic (EMG) data [5, 74] either with intramuscular (iEMG) or surface EMG sensors (sEMG). These methods use EMG data as input and detect activated muscle groups to achieve an understanding of the human body movement and the action, while we intend to infer muscle activations from body movements, therefore describing the opposite task. Chiquier et al. [13] propose a video-based AMGE dataset by using the signal of the wearable devices as the annotation. Yet, the data collection setting and the environment are restricted. The scale of the introduced dataset is relatively small and it encompasses limited action types. In our work, we collect a large-scale dataset based on HIIT exercises on YouTube while delivering binary annotation for each muscle region. We reformulate it into a multi-label classification task, namely AMGE in the wild. The annotations are first derived from online resources and then checked and corrected by researchers in sports fields.Activity Recognition is a dominating field within visual human motion analysis [1, 10, 28, 30, 39, 68, 72, 85, 89] which was propelled by the advent of Convolutional Neural Networks (CNNs) with 2D-CNNs [25] in combination with recurrent neural networks (RNNS) [17] or different variations of 3D-CNNs [8, 23, 54] . More recently, transformer-based methods advanced over 3D-CNNs, especially with advanced pre-training methods and large datasets [40, 44, 45] . Action Quality Assessment (AQA) [51, 73] and Visual Calory Estimation (VCE) [53] relate to our work since these methods likewise shift the question of research from what? to how? with the aim of detailed analysis of human motion. Multimodal data is a common strategy, e.g., by combining RGB video with audio [2, 52, 57] , poses [62] , optical flow [57] , or temporal difference images [50] . Skeleton data is also commonly used as a modality for activity recognition on their own. Yan et al. [81] and follow-up research [11, 36, 66, 67 ] make use of GCNs, while competitive approaches leverage CNNs with special pre-processing methods [14, 19] . Knowledge distillation (KD) [29] became a common technique to reduce the size of a neural network while maintaining performance.In review [27] , methods can be categorized to focus on knowledge distillation based on final network outputs (response-based) [31, 87] , based on intermediate features (feature-based) [82, 86] , or based on knowledge about the relations of data samples or features (relationbased) [9] . Recently, adaptations of distillation for transformer architectures gained attraction [37, 42] . Fusion strategies can be grouped into feature-fusion [56] and score fusion [33] .Multi-label classification methods allow for assigning more than a single class to a data sample. Common strategies include per-class binary classifiers with adapted loss functions to counter the imbalance problem [59] , methods that make use of spatial knowledge [83, 84] , methods that make use of knowledge about label relations [12, 70] , or methods based on word embeddings [43, 80] .Datasets which combine visual data of the human body with muscle activation information are sparse and mainly limited to specific subregions of the human body, e.g., for hand gesture recognition [26] .In contrast, a large variety of full-body human activity recognition datasets were collected in recent years, which are labelled with high-level human activities [34, 55] , fine grained human action segments [38, 88] , or action quality annotations [65] . We leverage such datasets by extending them with muscle group activation labels.  
activated muscle group annotation: 
 We cluster skeletal muscles of the human body into 20 major muscle groups with binary activation as shown in the checkboxes in Figure 1 .To ensure the quality of the annotation, we ask 2 researchers from the biomedical and sports fields to give the annotation for each activity by watching the video from the dataset. If the two biomedical and sports researchers fully agree with the AMGE annotation towards one activity, this activity is included in our dataset. Both of the two annotators are senior researchers in the biomedical and sports fields.  
architecture 4.1 preliminaries of mvit: 
 TRANSM 3 E is based on the improved multi-scale visual transformer (MViTv2) [40] , which is based on MViTv1 [21] . The model architecture of TRANSM 3 E is shown in Figure 3 . Compared with ViT [18] , MViTv1 increases the channel resolution progressively and reduces the resolution on the spatiotemporal plane simultaneously, which realizes pooling operations both on Keys (K) and Queries (Q). The basic idea of MViTv1 is the construction of different low-and high-level visual modeling stages [21] . Multi-scale pooling attention is one of the major components of MViTv2 compared with ViT. MViTv2 uses decomposed relative position embeddings and residual pooling connections to integrate the principle of shift-invariance into the model and reduce computational complexity, while the downscaling in MViTv1 is achieved by large strides on the Keys (K) and Values (V).MCTs are used to harvest more informative components to achieve good generalizability for AMGE and to construct sender and receiver for cross-modality knowledge distillation in our work as shown in Figure 3 . In our MCT setting, we directly use the final layer output of MCT and aggregate the MCT along the token dimension together with SoftMax to achieve multi-label classification.Assuming the classification tokens of MCT to be referred to by {cls 𝑗 | 𝑗 ∈ [1, . . . , 𝐶]} and the flattened patch embeddings to be referred to as {p 𝑖 | 𝑖 ∈ [1, . . . , 𝑁 𝑃𝑎𝑡𝑐ℎ𝑒𝑠 ]} for the given input video, where 𝑁 𝑃𝑎𝑡𝑐ℎ𝑒𝑠 is the length of the patch sequence, the input of the first MViTv2 block is cls 1 , . . . , cls 𝐶 , p 1 , . . . , p 𝑁 𝑃𝑎𝑡𝑐ℎ𝑒𝑠 . The final prediction y is computed through,EQUATIONwhere P 𝛼 indicates a fully connected (FC) layer projecting the merged MCT to a single vector with the number of muscle regions as dimensionality. We make use of the same MCT settings for both the video-based backbone and the skeleton-based backbone according to Figure 4 . After the first GCN block, the MCT for knowledge distillation and the MCT for classification are added to the model. We first flatten the spatial temporal nodes from the graph structure preserved by the GCN block. We use z * 𝐺𝐶𝑁 to denote the nodes of the constructed graph structure, cls * 𝑚 to denote the MCT for classification, and cls * 𝑟 to denote the MCT for knowledge distillation regarding skeleton branch. We then concatenate all of these components along the node dimension and execute feature projection by using linear (2) Then we execute an internal knowledge merge from the nodes to the MCT for the classification, as follows,EQUATIONwhere P 𝑠 denotes a FC layer. Finally, the node features, MCT for classification, and MCT for the knowledge distillation will be transferred to the next GCN block and the same procedure will be executed.Multi-Classification Tokens Knowledge Distillation (MCTKD) is one of our main contributions. To the best of our knowledge, we are the first to introduce this technique which can enable knowledge distillation on the multi-classification tokens between two backbones with obvious structure differences. Through our observation in this work, we find that directly merging the feature from the skeletonbased model and video-based model can not achieve a satisfactory performance due to the huge structure and modality difference. To achieve appropriate feature fusion between two architectures on different modalities with obvious differences, we need a new solution focusing on this issue. Knowledge distillation, which has the capability to accomplish the feature space alignment, is firstly explored in our work on the MCT perspective to assist the cross-modality feature fusion for the AMGE in-the-wild task.In the past, transformer-based knowledge distillation mainly focused on using intermediate full patch embeddings [48] or final classification token [75] , while we propose knowledge distillation on the proposed MCT for both intermediate and final layers by using additional MCT for the knowledge distillation.The underlying benefit of MCTKD is that the token number of the MCT is fixed, while knowledge distillation on the patch embeddings [22] may encounter the alignment issue when facing different modalities with different token sizes. Instead of directly distilling knowledge from the MCT of an auxiliary modality towards the MCT of a major modality, knowledge distillation MCT is introduced to serve as a knowledge receiver. This approach avoids disruption on the MCT for classification for the major modality, i.e., RGB video modality. The knowledge distillation MCT of the major modality branch is denoted as cls 𝑟 = {cls 𝑟,1 , cls 𝑟,2 , ..., cls 𝑟,𝐶 } and the knowledge distillation MCT from the branch of auxiliary modality is indicated by cls 𝑠 = {cls 𝑠,1 , cls 𝑠,2 , ..., cls 𝑠,𝐶 }, MCTKD is achieved by applying KL-Divergence (KL-Div) loss after each feature map reduction block of MViTv2 on cls 𝑟 and cls 𝑠 :EQUATIONwhere 𝑁 𝐵 and 𝐿 𝑀𝐶𝑇 𝐾𝐷,𝑎𝑙𝑙 refer to the block number and the sum of MCTKD losses. 𝐿 𝑀𝐶𝑇 𝐾𝐷,𝑎𝑙𝑙 is combined equally with the binary cross entropy loss (𝐿 𝐵𝐶𝐸 ).Multi-Classification Tokens Fusion (MCTF) is designed to fuse MCT for knowledge distillation and the MCT for classification as in (EQUATIONAfter obtaining the Q m/r , K m/r , and, V m/r from the MCT for classification and the MCT for the knowledge distillation, a mixed attention mechanism is calculated as follows,EQUATIONwhere 𝐴𝑡𝑡 denotes the attention operation 𝐴𝑡𝑡 (Q m/r , K m/r , V m/r ) = 𝑆𝑜 𝑓 𝑡𝑀𝑎𝑥 (Q m/r @K m/r ) * V m/EQUATIONThen the attention is finalized as,EQUATIONThe fused attention is thereby calculated through,EQUATIONwhere P 𝑓 denotes an FC layer. The whole procedure is indicated by,EQUATIONwhere 𝐿𝑁 demonstrates the layer normalization and 𝐶𝐿𝑆 𝑓 is the CLS-Fusion. Assuming we use cls 𝑎 to denote the average of MCT for classification and the MCT for knowledge distillation by cls 𝑎 = (cls 𝑚 + cls 𝑟 )/2, the final classification tokens are harvested by,EQUATIONwhere M 𝜃 denotes a Multi-Layer Perception (MLP) based projection and DP denoted dropout operation. MCTKD and MCTF are added after 𝑁 𝑀𝐶𝑇 epochs of training of TRANSM 3 E with only MCT, for both of the leveraged modalities and models. During the test phase, we make use of the average of the prediction results from the two branches as the final prediction.All the video models are pre-trained on ImageNet1K [16] using PyTorch 1.8.0 with four V100 GPUs. To reproduce TRANSM 3 E, we first train MViTv2-S with only MCT for classification on RGB modality and HD-GCN with only MCT for classification on skeleton modality for 80 epochs and then train TRANSM 3 E with all components for another 80 epochs. We use AdamW [46] with learning rate of 1𝑒 -4 . The input video for train, test, and val is center cropped and rescaled as 224×224 with color jitter parameter as 0.4.The results of different architectures on our benchmark are provided in Table 2 . First, the approaches include Random, in which the muscle activation is predicted randomly, and All Ones, in which all the samples are predicted as using all the muscle regions. These two simple approaches are used to serve as statistic baselines. Random and All Ones show overall low performances with <30% mAP on all the evaluations. These statistical approaches are leveraged to make comparisons between deep-learning-based approaches to verify whether the model predicts muscle activation randomly or not.The skeleton-based approach, e.g., HD-GCN [36] , ST-GCN [81] , and CTR-GCN [11] , obviously outperform the statistic approaches and deliver promising performances when dealing with unseen activity types. Video-based approaches surpass statistic and skeleton baselines in terms of the AMGE of the known activities, where transformer-based approaches, e.g., MViTv2 S/B [40] and VideoSwin S/B [45] , and CNN-based approaches, e.g., C2D [24] , I3D [8] , Slow [23] , SlowFast [23] , are leveraged. MViTv2-S shows good performance compared with the other methods due to the capability for reasoning long-term information and the multi-scale pooling setting which can extract informative cues from different abstract perspectives, especially with 79.6%, 79.7% for mean val and mean test on the MuscleMap dataset. However, we find that skeletonbased approaches work well on the new activities while they can not deliver satisfactory results on the known activities. On the other hand, video-based approaches work well on the known activities while they can not provide promising AMGE results on the new activities. A good AMGE model is expected to work well in both of the scenarios.To simultaneously achieve good performances for both the new activities and the known activities, we would like to grasp the advantages both of the skeleton-based approaches and the videobased approaches. We proposed TRANSM 3 E, which achieves feature fusion and knowledge distillation by using multi-classification tokens (MCT) on both the video-based model and the skeletonbased model, by using the most outperforming backbones from the two different modalities, i.e., MViTv2-S and HD-GCN. This new proposed method incorporates knowledge distillation from the multi-classification token level and feature fusion from the multiclassification token level to harvest more underlying attributes of the body motion which can benefit the activated muscle group estimation task in the wild task. TRANSM 3 E surpasses all the others by large margins. TRANSM 3 E is a transformer-based approach due to the capability for long-term reasoning of visual transformers [76] since the AMGE should consider the activities at the global level, which requires long-term information reasoning. TRANSM 3 E has 64.1%, 97.8%, 64.2%, and 81.0% mAP considering new val, known val, new test, and known test on our benchmark, while the generalizability to new activities is mostly highlighted. TRANSM 3 E outperforms MViTv2-S by 1.4% and 1.3% on the mean val and mean test, which especially works well for new val and new test as TRANSM 3 E surpasses MViTv2-S by 2.7% and 2.8%. Our method achieves significant improvements in AMGE through three main strategies: enhanced attribute reasoning using MCT, effective knowledge exchange between video and skeleton models, and integration of distillation and classification MCTs for better information consolidation. Despite its strengths, the performance gap in AMGE between known and new activities highlights areas for improvement, particularly in reducing misclassifications and biases, with more insights to be shared in a forthcoming ablation study analysis.Module ablation. The ablation study of MCT, MCTKD, and MCTF, is shown in Using MCTKD in sparse locations-specifically after each feature map size reduction-yields the best results on the MuscleMap benchmark. This method enhances the aggregation of AMGE cues across modalities after pooling, facilitating more effective knowledge distillation. SP-MCTKD surpasses SP-KD by utilizing multiclassification tokens to expand the prediction space for attributes during training, thereby capturing and transferring more crucial AMGE cues from various modalities. Consequently, SP-MCTKD, which excels in performance, is chosen as the foundation for our model, facilitating knowledge exchange between skeleton and video models. SP-MCTKD achieves the best performance and is selected. 5 , where our approach is compared with existing fusion approaches, e.g., Sum, Multiplication, SelfAttention, and CrossAttention. MCTF shows the best performance with 81.0% and 81.0% on mean val and mean test. Compared with CrossAttention, our MCTF achieves performance improvement by 1.7% and 1.7% in terms of the mean val and mean test. The superiority of MCTF compared to other approaches, especially on generalizability, depends on using attention from a more diverse perspective, which benefits the capability of integration considering different focus formats.Table 6 presents the comparison between TRANSM 3 E and existing multi-modality fusion approaches, i.e., LateFuionSum, LateFusion-Concat, and LateFusionMul. We compare our proposed method to these conventional multi-modal fusion approaches to illustrate that the performance improvement of our approach is not solely delivered by using the feature fusion between the skeleton modality and the RGB video modality. Compared with the best performing baseline LateFusionConcat, our approach achieves a performance improvement by 14.3%, 3.3%, 8.8%, 14.5%, 3.0%, and 8.7% in terms of known val, new val, mean val, known test, new test, and mean test.Qualitative results are shown in Figure 5 , the label and GradCam [63] visualizations of MViTv2-S and TRANSM 3 E are given from left to right. The true/missed/false prediction is marked as green checkmark/purple crossmark/red crossmark. Overall, our approach has more accurate predictions and fewer false and missed predictions for all the samples considering known activities, i.e., 1 and 2 in Figure 5 , and new activities, e.g., 3 and 4 , where 1 and 2 are correctly predicted by our model. TRANSM 3 E concentrates mostly on the accurate body regions, e.g., in sample 3 TRANSM 3 E focuses on the leg and abdominis related region, while the focus of the MViTv2-S is distracted, which results in more false predictions of MViTv2-S. Due to the integration of the learned knowledge from both the video and skeleton modalities, our model can achieve a better focus. 
conclusion: 
 In this paper, we open the vistas of video-based activated muscle group estimation in the wild. We contribute the first large-scale videobased activated muscle group estimation dataset considering in-thewild video and build up MuscleMap benchmark for the AMGE by using statistic baselines and existing video-based approaches including both video-based and skeleton-based methods. We take additional consideration regarding the AMGE generalizability. We propose TRANSM 3 E with multi-classification token distillation and fusion in a cross-modality manner to enhance the generalization to new activity types. TRANSM 3 E sets the state-of-the-art on the proposed MuscleMap benchmark and it delivers promising generalizability towards unseen activities in terms of the AMGE in-the-wild task. In the future, the utilization of large language models (LLMs) for the purpose of video-based active muscle group estimation is anticipated to enhance the extensibility of the approach with respect to estimating muscle groups during novel physical activities. 
 
 Appendix:  


Decision: The paper tackles the novel problem of activated muscle group estimation from in-the-wild exercise videos of High Intensity Interval Training. The primary contributions from the work are a new multi-modal dataset and a new approach for the challenging task.  	

The paper received reviews which were positive for the most part - 4 weak accept and 3 borderline accept. The rebuttal from the authors convinced two of the reviewers to revise their scores upwards. There were some concerns about the writing quality and to some extent, the methodology and evaluation.  

I agree with the assessment of the reviewers. For a first of its kind task, the paper covers reasonable ground and sets the stage for further work on an important problem related to human activity understanding. 

**I recommend acceptance of the paper**. The authors should address the concerns raised by the reviewers and improve the quality of final draft. The authors should also share necessary links for downloading the dataset and the codebase of the paper.

Reviews: review: paper_summary: paper_summary: This study tackles the challenge of Activity Muscle Group Estimation (AMGE), focusing on the identification of active muscle groups in the wild. The authors have created the MuscleMap dataset, an extensive resource comprising more than 15,000 video clips with 135 different activities and 20 muscle groups. This work introduces a method, TRANSM3E, which combines the strengths of video transformer model with skeleton-based graph convolutional model. The experimental results presented in the paper illustrate that TRANSM3E surpasses current video classification models in both its recognition of known activity types and its ability to generalize to new activities. It is commendable that the authors will release the source code and the MuscleMap dataset, which will facilitate further advancements in the field.
strength_weakness: strength_weakness: 1.	The study introduces a task of AMGE in the wild, which presents prospects for applications in sports and the field of rehabilitation medicine.
2.	The construction of the MuscleMap dataset by the authors is commendable. Its large scale and diverse categories, coupled with the decision to make it publicly accessible, are likely to foster advancements in this area of research.
3.	The introduction of the TRANSM3E method is a contribution. The integration of transformer architectures and Multi-Class Tokens (MCT) is impressive, and the method's generalizability is evidenced by its state-of-the-art performance when benchmarked against leading video classification models.
limitations: limitations: 1.	The dataset is currently limited to HIIT exercises, which somewhat narrows its scope for general use. To enhance its versatility, it would be advantageous to expand the dataset to include a broader range of activities, such as aerobic exercises and strength training.
2.	Section 3 lacks a detailed description of the methodology used to construct the skeleton data within the MuscleMap dataset. Given the effort and interest in this aspect of data annotation, the paper would benefit from a more comprehensive description.
3.	The paper could be strengthened by a more thorough analysis of cases where the model's predictions have failed. It would be beneficial to examine instances of high misclassification rates for certain labels and to propose solutions for addressing these challenges.
4.	The requirement for skeleton data as input for the TRANSM3E model may present practical challenges, as skeleton data may not be readily available in all video datasets. The computational overhead associated with extracting this data could impact the model's applicability in real-world scenarios.
5.	The length of Section 4 is disproportionate to the rest of the paper. It is recommended that the experimental section could be a separate section, titled EVALUATION (Section 5).
paper_summary: paper_summary: The paper introduces a  video-based solution for identifying active muscle regions during physical activity. Unless previous works, the proposed solution is designed for practical, non-controlled "in-the-wild" settings. The paper starts by first developping a rich datasets by exploting excercise YouTube videos and labelling them with the help of two researchers in the biomedical and sports research field. To address the  multi-label classification problem, the paper introduces TRANSM3E a cross-modality knowledge distillation and fusion architecture. TRANSM3E is thoroughly evaluated using the dataset collected from YouTube and is shown to outperform the state of the art, in particular in terms of generability.
strength_weakness: strength_weakness: - The dateset is very interesting and could provide very helpful to the commmunity.
- The idea of video-based physical activity recognition in non-controlled settings is very interesting and can have many practical applications.
- TRANSM3E is novel and shown to outperform the state of the art, especially in terms of generability, which is very important in practice.
limitations: limitations: - The writing and the structure need improvement to help with clarity. Section 4, for example, is called Architecture and yet it includes the performance evaluation as well. The writing is at times verbose and repetitive. For example, TRANSM3E is explained multiple times, including in the performance evaluation. A more focused and "lean" writing would help the reader better understand the contributions and the results.
paper_summary: paper_summary: The paper explores a new task in the field of video-based Activated Muscle Group Estimation (AMGE). This task aims to identify active muscle regions during physical activities in various environments. The researchers introduce the MuscleMap dataset, collected from YouTube, specifically target High-Intensity Interval Training exercises, emphasizing real-world applications in sports and rehabilitation without stringent environmental constraints.

To address the challenges of this task, the researchers developed a new model called TRANSM3E. This model employs a feature fusion mechanism that combines video transformer models and skeleton-based graph convolution models. It also incorporates a novel approach to cross-modal knowledge distillation using multi-classification tokens. This innovative method allows the model to generalize effectively to new types of physical activities that were not included in the training data.

A significant contribution of this paper is the creation of the MuscleMap dataset, which provides a comprehensive and annotated video resource for AMGE. This dataset enables the development and benchmarking of AMGE models. The researchers conducted extensive experiments with various off-the-shelf models, including Convolutionsl Neural Networks, Graph Neural Networks, and transformer-based architectures, to evaluate their effectiveness in handling the AMGE task. These experiments revealed that generalizability remains a challenge for existing architectures adapted for AMGE. However, the proposed model demonstrated superior performance compared to popular video classification models, effectively handling both known and new activity types.
strength_weakness: strength_weakness: * Paper is well-structured on the higher level
* Paper introduces the novel task of **video-based** Activated Muscle Group Estimation (AMGE).
* Paper presents a novel dataset and benchmark  
* The relevance for fitness apps or rehabilitation is described and it presents a lower entry barrier and less expensive approach compared to existing approaches
* Provides thorough evaluation using the newly created MuscleMap dataset, comparing the performance of TRANSM3E against various off-the-shelf models.
* Dataset includes a high number of clips and metadata
* Public available code and dataset
limitations: limitations: * The introduction contains much modeling work (i.e. 138-141; 141-143; 154-161, 202-207) 
* The use of YouTube videos raises privacy and rights concerns, which are not addressed in the paper.
* No comparison with non-video-based methods as the mentioned wearable devices with electrode sensors, or real world/user study
* Title: based on the evaluation, is it a Towards paper?
paper_summary: paper_summary: This paper proposes a MuscleMap dataset for video-based Activated Muscle Group Estimation in the wild. It collects 135 different exercises from YouTube, each type of exercise is annotated with one or more of 20 different muscle group activations. There are always 15004 video sequences. This paper also proposes the $TRANSM^3E$ method to solve the Activated Muscle Group Estimation task of multi-label classification. It is a cross-modality knowledge distillation and fusion classification architecture, which includes three basic new components: Multi-Classification Tokens (MCT) (for multi-label tasks), Multi-Classification Tokens Knowledge Distillation (MCTKD), and Multi-Classification Tokens Fusion (MCTF). Firstly, MCT is used to process video-based models and backbone-based models. Then, MCTKD is used to fuse the multi-modality features of the two models, and an additional MCT is added to perform knowledge distillation on both the intermediate and final layers as a knowledge receiver. Finally, MCTF fuses the multi-classification labels used for knowledge distillation with those used for classification.
strength_weakness: strength_weakness: 1. This paper proposes a new dataset called MuscleMap for the Activated Muscle Group Estimation task, which opens the prospect of video-based Activated Muscle Group Estimation and reduces the entry threshold for muscle activation-based healthcare and exercise applications.
2. The Multi-Classification Tokens Knowledge Distillation (MCTKD) proposed in this paper can fuse features of two backbone models with significant structural differences on different modalities, achieving feature space alignment.
limitations: limitations: 1. The keywords are too large, broad, and not prominent enough.
2. The proposed dataset is all videos of professionals. Can you add some videos of ordinary people to verify the universality of the proposed dataset?
3. Inconsistent capitalization and abbreviations of paper titles in the references.
paper_summary: paper_summary: This paper introduces a video-based activated muscle group estimation (AMGE) task, which aims to identify active muscle regions during physical activities in natural environments. To this end, the authors provide a dataset called MuscleMap, which includes over 15,000 video clips, covering 135 different activities and 20 labeled muscle groups. For practical application of the AMGE model, it is crucial to ensure that the model generalizes well to new types of activities and new combinations of activated muscles that were not present during training. The authors also propose a new method called TRANSM3E, which combines multimodal feature fusion between a video transformer model and a skeleton-based graph convolution model, along with cross-modal knowledge distillation, surpassing all popular video classification models.
strength_weakness: strength_weakness: 1.The MuscleMap dataset is large-scale and unconstrained by the environment, which helps advance related research.
2.The proposed TRANSM3E method combines a video transformer and a skeleton graph convolution network, and uses multi-class tokens for cross-modal knowledge distillation, showing excellent performance and generalization ability.
3. This method and dataset have significant practical applications in fields such as sports and rehabilitation medicine.
limitations: limitations: 1. The TRANSM3E model is quite complex, involving various techniques and components, which might increase implementation and deployment difficulties.
2. Despite the large dataset, all videos are from YouTube, which may have limitations in video quality and angles, affecting the model's generalization ability.
3. The dataset labeling relies on expert subjective judgment, which may lead to annotation errors and consistency issues. How can this bias be addressed?
4. Only one dataset, which is self-constructed, was used, making it hard to validate the algorithm's universality and robustness.
paper_summary: paper_summary: **summary**
this paper proposes a new task of muscle group activation, where for each activity, multi-labels of activated muscle groups are provided. a new method seems to perform reasonably well as compared to existing baselines like slowfast.
strength_weakness: strength_weakness: **strength**
authors propose:
- a new task of video-based activated-muscle group estimation which is a multi-label classification problem 
- a new dataset/benchmark called MuscleMap, which consists of more muscle groups/activities 
- Multi-Classification Tokens (MCT): classs-specific tokens are injected into the net, in contrast to vit where only one class token does into the net. this appears novel.
limitations: limitations: **weakness**

the weakness i mentioned below are some minor clarifications which authors can make, it does not impact the strong technical aspects presented in the paper. 

**technical clarifications:**

- line 343: "For MuscleMap, 20 of 135 activities are leveraged to formulate the new val/test set": why are only 20 activities being considered in val/test. what about other 115? it would have made sense if these 20 were not seen during training, and zero-shot was being evaluated. however, it seems that this type of analysis was not in the scope of the paper? 
however, i only see a softmax with fixed neurons being trained, not a contrastive thing as done in clip. 

- line 505: "we find that directly merging the feature from the skeletonbased model and video-based model can not achieve a satisfactory
performance due to the huge structure and modality difference": is skeleton modality input as a sequence of coordinates, or black white skeleton map? this seems suprising, considering that there are other works like omni-bind which integrate multi-modality. is it possible to keep modality-specific encoder, but a SHARED backbone to process these two disparate modalities together? in that case, one might not need to process them with gcn layers. 

- line 634- 642: it seems that the architecture is trained in 3 phases? any possible comments on making it  a joint -training instead? 

- balanced dataset: it seems neck/head muscle group is lower in number... perhaps add some videos of people doing inverted headstand or other yoga pose involving head? for thigs, it could be a yogic pose like virbhadrasana?


**qualitative questions:**

- fig 5: shows gradcam on mvitv2 vs authors method. i see a blue blob at the correct place. but it is too broad, and not specifically focused on individual muscle group, 
do the authors have any insights why this might be case?  perhaps some other form of qualitative analysis might be helpful. 

**additional experiments:**

- an anlysis on open-world ability would be interesting: it might be tough given the rebuttal phase, but it definitely would be interesting, even if not, perhaps the authors could at least release an open-world TEST split so that the community could build on this work in the future. 

- an analysis on "which" muscle group is toughest to classify? i think that muscles like neck etc which occupy less area might be tougher to do, but it would be interesting to see that being qualitatively analyzed, ... 

- additional ablation: can the authors try one ablation with increasing some "filler" tokens, i.e. the parameters which dont contribute to the final softmax, but just used to guide the network to absorb some errors. does increasing such fillers results in better perfomance? [1]

**minor suggestions which dont impact the review**
 - line 223/912: please rephrase "the vistas of", sounds non-scientific, perhaps, "propose a new task of".

[1] hinton et al, how to representa part-whole hierarchies in connectionist-nets. 

**rating**
weak accept. 

**justification of rating**

this is a good work and i commend author's efforts in this work. It deserves acceptance PROVIDED that some of the clarifications, and additional experiments might be performed.... depending on how the rebuttal goes, i might **lower/retain my rating** later...

- thx, 
your sincere reviewer :-)
paper_summary: paper_summary: The authors introduce a new approach  $TRANSM^3E$ , utilizing a multi-modality feature fusion mechanism that combines Vision Transformer and GCN to process both RGB and Skeleton inputs. Additionally, the approach implements cross-modality knowledge distillation on multi-classification tokens.
strength_weakness: strength_weakness: - Traditional AMGE relies on sensors to obtain electromyogram signals. For video-based AMGE tasks, due to occlusions such as clothing or body parts, estimating activated muscle groups using intuitive human body motions is effective.

- The ablation studies in the paper are comprehensive.
limitations: limitations: - In Lines 136-137, the entire dataset annotation relies solely on two professional researchers, which raises concerns about potential cognitive biases. Additionally, there are unreasonable annotations in the GroundTruth part of Figure 5 (for example, is there no involvement of the chest muscle group in the shoulder movement in Sample 1?). Furthermore, the physical concept of "activation" in the task requires further clarification.

- Since the proposed MuscleMap dataset includes not only bodyweight exercises but also interactions with fitness equipment, the muscle activations in such interactive actions tend to differ from those in bodyweight resistance exercises, which is not discussed in the paper.

- The paper does not mention how the RGB Diff and optical flow modalities in the MuscleMap dataset are utilized, nor does it explain the acquisition method for the skeleton annotations.

- Input-output clarity: Does the proposed $TRANSM^3E$ output the recognition of action types and activated muscle groups for each input frame of RGB and skeleton?

- The backbone network MViTv2 and the compared work VideoSwin are both from before 2022, and the paper does not adequately discuss more recent work from the past year.

- In Section 4.6, the paper fails to effectively explain why skeleton-based methods and video-based methods cannot achieve good performance on both known actions and new emerging actions.

- Other: The paper title does not match the title on the OpenReview submission page.