Title: Learning Skill-Attributes for Transferable Assessment in Video
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video understanding

Skill assessment
Abstract: Skill assessment from video entails rating the quality of a person’s physical performance and explaining what could be done better. Today’s models specialize for an individual sport, and suffer from the high cost and scarcity of expert-level supervision across the long tail of sports. Towards closing that gap, we explore transferable video representations for skill assessment. Our CrossTrainer approach discovers skill-attributes—such as balance, control, and hand positioning—whose meaning transcends the boundaries of any given sport, then trains a multimodal language model to generate actionable feedback for a novel video, e.g., “lift hands more to generate more power” as well as its proficiency level, e.g., early expert. We validate the new model on multiple datasets for both cross-sport (transfer) and intra-sport (in-domain) settings, where it achieves gains up to 60% relative to the state of the art. By abstracting out the shared behaviors indicative of human skill, the proposed video representation generalizes substantially better than an array of existing techniques, enriching today’s multimodal large language models.


Peer reviews:
comments: SUMMARY:
The paper proposes CROSSTRAINER, a method for video-based skill assessment that learns transferable skill-attributes (e.g., control, balance) shared across sports. It uses a multimodal LLM to predict incorrect attributes, generate actionable feedback, and estimate proficiency

STRENGTHS AND WEAKNESSES:
Strengths:
1. Demonstrates consistent improvements over SOTA baselines across skill-attribute prediction, feedback generation, and proficiency estimation.
2. Handles unseen skills and sports with minimal performance drop, showing strong generalization across tasks and domains.
3. Decomposing feedback into generalizable “skill-attributes” makes the model more interpretable and adaptable.

Weakness:
1. While skill-attributes reduce label cost, they still rely on curated expert feedback for supervision, which may not scale well.
2. Current formulation lacks mechanisms for reasoning about external factors (e.g., terrain, opponent behavior) that may influence skill.
3. Focuses only on individual physical tasks; unclear if the approach extends to multi-agent, strategic, or cognitive skills.
4. Heavy reliance on LLM prompting and proprietary components (e.g., LLaMA-3) without detailed insight into prompt design or tuning.

QUESTIONS:
How do you ensure the extracted skill-attributes are accurate and consistent? Are they verified or clustered?
Do you expect CROSSTRAINER to work for non-physical tasks (e.g., surgery, industrial skills)? Any thoughts on adapting it?

LIMITATIONS:
The method still depends on expert commentary for supervision. Addressing how this could be scaled or replaced with weak/noisy supervision would be useful.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of video-based skill assessment, aiming to overcome a major limitation of existing methods: their poor transferability across different sports or activities. The authors propose an approach centered on the idea of learning a set of universal, sport-agnostic skill-attributes like balance, control and body positioning. The core contribution is a two-stage training paradigm. First, they cleverly bootstrap these skill-attributes by using a LLM to parse existing expert commentaries from datasets, thus creating weakly-supervised labels. Second, they pre-train a multimodal model to predict these attributes from video, and then finetune it to generate actionable feedback and estimate a performer's proficiency level. The experiments conducted on datasets like Ego-Exo4D and QEVD, demonstrate that this approach  improves in-domain performance and shows significantly better generalization to novel sports and drills in various zero-shot settings compared to state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper addresses the problem of the lack of transferability in action quality assessment. The core idea of identifying and using intermediate skill-attributes as a transferable representation is insightful. It provides a structured way to reason about why a performance is good or bad, which aligns well with the principles of AQA and effective coaching.\n\n2. The setup for testing transferability with its multiple zero-shot configurations (ZS-1, ZS-2, ZS-3), is rigorous and provides strong evidence for their main claim. The results show their method outperforming strong baselines, especially in the challenging zero-shot scenarios where it degrades much more gracefully.\n\nWeaknesses:\n\n1. The quality of the bootstrapped skill-attributes is fundamental to the entire pipeline, yet this process relies heavily on an LLM prompted to extract concepts from text. The robustness of this extraction process to different LLMs, prompts, or noisy/ambiguous commentary is not deeply analyzed. \n\n2. The overall framework of generating soft labels using LLM then fintune LoRA and the underlying model architecture like a frozen video encoder, a trainable mapping network, and a frozen LLM is a relatively standard pattern in recent multimodal research and lacks novelty. \n\n3. While the experimental evaluation is extensive, the central claim about the necessity of the skill-attribute pre-training stage could be further substantiated with more direct ablation studies, as detailed in the questions below.\n\nQUESTIONS:\n1.  The paper compellingly argues that the two-stage, attribute-centric approach is superior to other SOTA models. To more directly isolate and quantify the benefit of the proposed pre-training stage, have you considered running an ablation study where the exact same  architecture is trained end-to-end? Specifically, a baseline trained to go directly from video to actionable feedback, without the intermediate skill-attribute discovery and pre-training stage. This would provide a very clean and powerful demonstration of the value added by your proposed pre-training methodology.\n\n2. Regarding the crucial skill-attribute discovery stage (Sec 3.3): Could you provide more analysis on the quality and stability of the LLM-generated attributes? How consistent are the extracted attributes if a different LLM or slightly different prompts are used? \n\n3. The paper mentions creating open-vocabulary skill-attributes. Is it a fully open vocabulary discovered on the fly, or is there a process to consolidate or cluster similar LLM outputs into a more canonical set of attributes?\n\n4. Have you conducted any human studies to assess whether the generated feedback is perceived as actionable and correct, especially for novel sports?\n\n5. In the ablation-like discussion, you mention that adding explicit body pose features didn't provide a significant accuracy boost relative to the computational cost. Could you quantify this trade-off?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses video-based skill assessment—including incorrect skill-attribute detection, actionable feedback generation, and proficiency estimation—and introduces the CROSSTRAINER framework. The core idea is to incorporate an intermediate representation, namely skill attributes, into the multimodal LLM (MLLM) pretraining and prediction process. Results show that CROSSTRAINER outperforms existing methods across various tasks, datasets (Ego-Exo4D, QEVD, and self-collected YouTube videos), and settings, particularly in diverse zero-shot scenarios such as testing on unseen skills from previously unseen sports during training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper establishes a strong connection between the proposed method and cognitive science (e.g., the transferability of skills across different sports), enhancing its overall coherence and plausibility (Ln 55-62).\n\n2. The experimental results presented in Tab. 1, Fig. 4, and Fig. 5 are promising, highlighting the superiority of CROSSTRAINER over many state-of-the-art approaches, including ExpertAF [1], in both in-domain and zero-shot setups.\n\n3. A few failure cases are presented in Fig. 5 and discussed in the supplementary material, which is reasonable and highly helpful for understanding the method.\n\nMajor Weaknesses:\n\n1. Lack of essential training details for zero-shot setups: According to Section 3.3, the proposed method derives skill attributes from 32k Ego-Exo4D and QEVD commentaries. In zero-shot setups, were the skill attributes extracted from the “test” sports or skills excluded from all stages of training, including pretraining? Clarifying this is crucial to ensure that no information from the test sports or skills leaks into the training process.\n\n2. Potential unfair comparison: For actionable feedback generation, the proposed method utilizes video features, predicted skill attributes, and a prompt (Lines 204–206) as input. This prompt appears to include a placeholder for the activity name (e.g., &lt;activity&gt;). Does this imply that the actual activity name (e.g., dribbling) is inserted during both training and testing? If so, do the competing methods listed in Tab. 1 employ a similar mechanism for generating actionable feedback? Otherwise, the proposed method may have an advantage, as it is already provided with contextual information (e.g., the sport or skill) and only needs to generate feedback based on the attributes within this known context.\n\n3. Limited number of tested sports and skills: Although the proposed method aims to transfer skill knowledge across different sports and skills, the range of evaluated activities is quite narrow, comprising only 3 sports and 5 skills in Ego-Exo4D, 23 workout moves in QEVD, and 3 sports in the collected YouTube dataset. This limited scope makes it difficult to assess whether the observed performance gains are genuinely attributable to the method’s capabilities or simply to the specific choice of tasks. In fact, it is easy to imagine scenarios where skill transfer between sports may not be effective. For example, leaning forward may be beneficial in skiing but detrimental in basketball defense. Negative transfer [2] is a well-known challenge in machine learning, further underscoring the need for the proposed method to be evaluated across a broader and more diverse set of sports and skills to validate its generalizability.\n\n4. The paper does not provide justification for the selection of the three sports, frisbee, water polo, and a juggling drill in soccer, for collecting the YouTube videos. Clarifying the rationale behind choosing these specific activities is important to assess whether the proposed method can truly generalize to a broader range of sports and skills.\n\nMinor Weaknesses:\n\n5. The paper lacks ablation studies on skill attributes, particularly regarding how their diversity or quality impacts final performance. This analysis is important to support the claim in Ln 155–157 that the proposed method can effectively leverage commentaries collected from online video-sharing platforms. Without such an evaluation, it remains unclear how robust or sensitive the method is to variations in the source and quality of skill attributes.\n\n[1] Ashutosh, Kumar, et al. \"ExpertAF: Expert actionable feedback from video.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Zhang, Wen, et al. \"A survey on negative transfer.\" IEEE/CAA Journal of Automatica Sinica 10.2 (2022): 305-329.\n\nQUESTIONS:\n1. Please refer to Weakness 1 and provide additional training details for the zero-shot setups. Specifically, clarify whether any skill attributes derived from the test sports or skills were used during training, including the pretraining stage, to rule out potential information leakage.\n\n2. Please address Weakness 2 by clarifying whether the activity name (e.g., dribbling) is provided in the prompt during both training and testing. Additionally, indicate whether competing methods listed in Tab. 1 use a similar strategy, to ensure a fair comparison in actionable feedback generation.\n\n3. In response to Weaknesses 3 and 4, please explain why testing on a limited number of sports and skills is still considered meaningful. Furthermore, clarify the rationale behind selecting the three specific sports for YouTube video collection.\n\n4. Please address Weakness 5 by including additional ablation studies on skill attributes, if possible. In particular, analyze how variations in their diversity or quality influence overall performance.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The article suggests to use cross training approach as multiple sports have common skill set requirements. The model trained performs better as it has already learned the features which are common in the sport. Multiple set of experiments are conducted: fully supervised, zero shot, similar sport zero shot and unfamiliar sport zero shot. The results show that the approach works well and results are presented in quantitative as well as qualitative forms. The results demonstrate that approach is effective and it can help in real world scenarios where datasets for all different sports are not available.

STRENGTHS AND WEAKNESSES:
strength
- paper is well written
- thorough experiments are conducted under different set of conditions to demonstrate the effectiveness of the approach
- the approach is simple and is represented well by mathematics in the article. 
- the approach has many implementations as it is a concern in the literature to create individual datasets for each sport.

weaknesses
- the idea is not novel and has been used in many other fields in different forms such as computer vision and medical imaging
- the approach is pretty much similar to fine-tuning of the existing models. It is slightly different as it is learning from the other datasets.

QUESTIONS:
Please check comments above

LIMITATIONS:
Please check weaknesses mentioned above.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-05