Learning Skill-Attributes for Transferable Assessment in Video
Authors: Kumar Ashutosh
Kristen Grauman
Venue: NeurIPS 2025 poster ()
Keywords: Video understanding
Skill assessment
Primary area: applications
TL;DR: We propose a method to transfer video-based skill assessment knowledge across various sports domains.

Abstract: Skill assessment from video entails rating the quality of a person’s physical performance and explaining what could be done better. Today’s models specialize for an individual sport, and suffer from the high cost and scarcity of expert-level supervision across the long tail of sports. Towards closing that gap, we explore transferable video representations for skill assessment. Our CrossTrainer approach discovers skill-attributes—such as balance, control, and hand positioning—whose meaning transcends the boundaries of any given sport, then trains a multimodal language model to generate actionable feedback for a novel video, e.g., “lift hands more to generate more power” as well as its proficiency level, e.g., early expert. We validate the new model on multiple datasets for both cross-sport (transfer) and intra-sport (in-domain) settings, where it achieves gains up to 60% relative to the state of the art. By abstracting out the shared behaviors indicative of human skill, the proposed video representation generalizes substantially better than an array of existing techniques, enriching today’s multimodal large language models.

Learning Skill-Attributes for Transferable Assessment in Video
Kumar Ashutosh Univeristy of Texas at Austin
Kristen Grauman University of Texas at Austin
Abstract
Skill assessment from video entails rating the quality of person's 's physical perfor- mance and explaining what could be done better. Today's 's models specialize for an individual sport, and suffer from the high cost and scarcity of expert-level supervi- sion across the long tail of sports. Towards closing that gap, we explore transferable video representations for skill assessment. Our CROSSTRA TRANER approach dis- covers biill-atributessssiie meaning transcends the boundaries of any given sport, then trains a multimodal language model to generate actionable feedback for a novel video, c.g., "lift hands more to generate more power" " as well as its proficiency level, e.g., early expert. We validate the new model on multiple datasets for both cross-sport (transfer) and intra-sport (in-domain) settings, where it achieves gains up to 60% relative the state of the art. By abstracting out the shared behaviors indicative of human skill, the proposed video representation generalizes substantially better than an array of existing techniques, enriching today's 's multimodal large language models. Project page: https://vision.cc cs. www.uut..aeeuuuee wwtpsssop/ppoooo//Coooooo CttprrojeeessCrssTaaaieeeeee
as balance, control, and hand positioning- whose
1 Introduction
The basis of assessing skilled physical activities, particularly sports, is largely visual. Precisely how a tennis player grasps and swings their racquet; how a basketball player releases the ball to shoot free throw; how rock climber stretches and pulls to traverse the boulder-ae suulderr-uuhh visual details are discernible to the expert eye and essential for providing meaningful coaching. Advances in multimodal video understanding could, therefore, transform Al-assisted coaching and skill assessment. For example, future Al agents could provide personalized feedback to users based on videos captured on their phone smartglasses, greatly expanding the accessibility of 1-1 coaching. Similarly, AI could analyze how multiple players' skills would complement each other when building a team, or even detect patterns in injuries as function of execution style.
Towards achieving the above, the core vision and machine learning task is as follows: given a video of an athlete's 's performance, estimate their skill level [12, 28, 37, 39, 67, 70, 71, 73] and indicate what could be improved [9, 16, 68].
This task raises important unsolved challenges. First of all. assessing skill requires the-ggriied understanding of all physical aspects. Whereas traditional action understanding emphasizes high-level semantics [17, 18, 27, 55, 60, 87, 88, and thus seeks invariance to execution differences- here the subtle differences are exactly what matters. For example, a novice and expert shooting a soccer ball into the net might accomplish the same overall action, but details about their approach, footwork. and trajectory the ball are essential to analyze skill. Secondly, compounding the challenge, supervision is costly and difficult scale across the long tail of sports. An estimated 8,000 unique sports exist today across the world, but only a small subset is widely recorded and distributed often driven by geographic and economic factors. Similarly, expert supervision is expensive and cannot be scaled up easily using traditional annotator pools. Making it worse, due to the common assumption that the axes of evaluation vary so wildly between sports, all prior work trains and tests on in-domain data, i.e., the same drill, exercise, or sport [9, 12, 37, 70, 70-72, 85, 98].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Learning incorrect skill-attributs
Basketball
Badminton
Cricket
Foot positioning
Balance
Tennis
Volleyball
Follow-through
Control
Squash
Golf
Soccer
Table
Rugby
Inference
Zero-shot skill assessment on novel uncommon sport
Improved skill assessment
Proficiency: Late Expert Skill-attribute:: Bending of knees Actionable feedback: You should bend your knees more before the shot.
Proficiency: Novice
Skill-attributs: Head position Actionable feedback: Keep your head down when swinging for control.
Basketball
Shinty
idea. Given a short video an athletic skill, what could better? Given video demonstrations from multiple sports, we learn kklll-atrrbutes that are incorrectly demonstrated, e.g., wrong foot positioning in badminton (top). These hill--atributes are common across various sports, and transfer to novel uncommon sports, e.g., shinty (hottom). Our method improves both the in-domain and zero-shot settings. Sports chosen for illustration; Sec. 4 for dataset details.
Figure Overview
propose CROSSTR TRINER, a new approach to video-based skill analysis that accounts for these challenges. Our key insight is leam universal skill hill--attibbte representation that transcends the boundaries of sports and hence allows sharing and transfer between them. A skill- still-attribute is describable fine- iine-grained concept about the physical performance that can take different visual forms in different sports, e.g., ball control, foot positioning, coordination, timing, or balance. By automatically learning these shared properties from ladoo-language commentaries, and then surfacing them during training, we aim to amplify the value of the limited training data available for any one sport.
Building on the skill-attributes, we then train a generative multimodal language model to predict i) which skill-attributes are incorrectly demonstrated in a given video, ii) what the proficiency level is, and iii) free-form commentary about which adjustments would improve the performance. Essentially factor skill assessment into its generalizable cross-sport component (e.g.. lacks control) and its focused sports- sports-specific component (e.g.. "you should increase the spacing between your legs for better control of the soccer ball"). Our formulation aims unlock the translation of skill assessment from widespread, highly marketed sports to uncommon, low-resource sports, such as kabaddi, frishee, waterpolo, shinty, kho-kho, or bandy. See Fig. 1.
Though represents a departure from today's models [9, 12, 37, 70, 70-72, 85, 98], cognitive science supports both transferabilitt across sports having similar skill sets and shared terminology for assessment [19, 65, 77, 82]. For example, basketball players make better decisions playing soccer compared to tennis players, due the former two sports' shared underlying properties [77]; similarly,

---

contact-sport athletes exhibit better zero-shot understanding of new contact sport compared to non-contact sport athletes [19]; and transferable skills arise from similar affordances [43, 81] or environmental properties [82]. Our work is the first to translate these intuitive findings from cognitive science into a working video understanding model.
We validate our ideas on three diverse datasets: Ego-Exo4D [37], which contains soccer, basketball, and rock climbing; QEVD [68], which contains fitness exercises; and in-the-wild YouTube videos of people tutoring physical activities. We show that training an intermediate skill representation yields both superior in in-domain performance (for familiar sports) as well as better zero-shot performance (where the particular sport or drill is never seen by the model during training). CROSSTR. TRASEE out- performs all competitive baselines on all axes of assessment aastsmment-cttionaale feedback, aatribasti and proficiency estimation -iith relative gains up to 60%. Furthermore, compared to any of the baselines, PROSSTRAINEE shows much more graceful degradation when transferring to a novel sport. Overall our approach is stepping stone for learning a unified skill-centric video representation for assessment and coaching in the presence of real- world data and annotation constraints.
2 Related Work
Learning representations from videos. Substantial research explores new ways to learn video representations [6, 48, 51, 74, 92], targeting video understanding tasks of action recognition [36, 45, 49, 89], action anticipation [1, 33, 35, 57], procedural understanding [13, 14, 20, 62, 101], and temporal step localization [3. 58, 97, 99, 100]. Instructional videos offer a valuable window into skilled human activity [59, 84, 103]. and recent work explores how navigate between related how-to's [7, 8, 102] and identify their differences [61]. Whereas prior work explores activity-centric representations emphasizing the semantics of what is being done, our problem requires capturing how it is being done, which we show is essential for skill assessment.
Video-based sports analytics. Sports analytics and assessment raises number of interesting challenges for computer vision [12. 37, 70, 71, 73, 85, 98]. Multiple ongoing workshops and chal- lenges [22, 34, 64] offer tasks including ball spotting, foul recognition, and game state reconstruction. Prior work on skill assessment either assigns a score (or equivalence class label) to a video demon- stration [12, 37, 67, 70, 71, 73], optimizes group contrastive score distribution [85, 98], chooses the better of two demonstrations [12. 29]. Broadening the scope of skill assessment beyond scoring videos, ExpertAF [9] aims provide actionable feedback in the form of natural language commen- tary, relevant video retrievals, and generated pose corrections. As discussed above, all the above existing work in video-based skill assessment is limited in-domain testing, whereas we explore transfer across sports, enabled by the proposed killlattributes, In addition, orthogonal to the transfer contribution, our results across three datasets representing 6 distinct sports and fitness activities and 30 distinct drills raises the bar in breadth of validation compared to any prior skill assessment work.
Zero-shot generalization and attributes. Testing on novel classes and scenarios is crucial for real open-world settings. In one line of work, zero-shot generalization is enabled by shared multimodal representations learned from noisy vision-language data, benefiting image classification [75, 90, 93], action recognition [10, 95, 96]. video to text and text to video retrieval [6. 80, 87, 88, 91, 92], or image segmentation [15, 26, 40]. In another line of work, attributes are intermediate variables [41, 42, 54, 66, 83, 94] that can express a new category even without training images (e.g., zebras are black and white and striped). Though they share our motivation for shared representations, all of the existing methods focus on semantics (what) as opposed to dynamic execution (how), and none are directly applicable to skill assessment from video. Ours is the first work explore attributes for fine-grained activities in video and the first to demonstrate the relevance for skill feedback.
3 Method
We introduce the problem statement in See. 3.1, followed by an overview of key datasets (Sec. 3.2), our idea to extract skill-attributes (Sec. 3.3), the full model (Sec. 3,4), and training (Sec. 3.5).
3

---

ballsplan Salleppacemme placement
speedweigt
opsacioning
www.eeeetttwwerreee power
contro
distribution
ba Llance
thand Ter Exoning
hand
heighcleg
tioning
RUgh kowes
lift
foot WO r Karmn It_ Post Lioning
Speed
Felge
contro
depth
tinins
foot
Dos
tioning
LEE TRI
posi
Dositionins
Kick
ne ghtt
accu
racy
ang
peed
squat
TUR
depth
head pos positioninge
hand
tiniing
g
foot
pos iositioning
retss
Lon
ball l Ii control
mov
mostiating
the the turn
bbaal lacee
body
pos
it ioniining
body
Sore
en
emggeeent
positions malre
wioh det
spacing ing
timing
movement
1ow
Lw
dn through
t lott placement
hhip bosittoning
pos t ooonn censastopcy
e bow post resttion
tiningg
seesch
form
Figure 2: Discovered skill-attribtees from Ego-Exo4D [37] (left) and QEVD [68] (right). We see phrases
reflecting generalizable physical concepts like control, hand/body positioning, and movement.
3.1 Problem definition
At inference time, given a short video clip V Die from the test set Die, want to assess its quality, even if the exact same skill/drill was never seen in training-and even (more extreme) if the sport in V was not seen during training. The desired assessment output covers three aspects: report the skill-attributes performed incorrectly, generate actionable feedhack that can help the learner improve, and finally, estimate a proficiency score. Consistent with the transfer observed in cognitive science studies [43, 81, 82], we focus on physical skills executed by an individual and hence assume the video contains one person of interest; multi-player team interactions are also interesting but less amenable to transfer and outside the scope of this work.
To handle this task of assessing both in-domain and novel data, we employ two-stage training process. In the pretraining stage, we train the model to generate the incorrectly demonstrated skill- attributes, i.e., we learn a function Fa to predict the skill-atribbtes $ = {81, $2,... that the person in the video should improve on:
(I) where Der denotes the training set, e.g., if person in the video is dribbling a soccer ball, S can be control and leg positioning, supposing the person is incorrectly executing those two. See Fig. 1, left.
Fo(V D(r)= S,
In the second stage, we finetune the model for the remaining two aspects of assessment. Firstly, we generate feedback conditioned on the inferred skill-attbbute set S,
FV, $1 D++)= =T
(2)
where T' is textual actionable feedback statement for improvement, e.g., "the player is too straight and should bend more for a better control". While the skill-attributes are words or short phrases indicating suboptimal dimensions in general terms, the feedback consists of sentence(s) elaborating on what to fix in the context of the observed sport. Finally, we generate the proficiency level of the person in V, again conditioned on S:
(V, D..)= P.
(3)
where P is the proficiency class label, e.g., novice, intermediate, early expert or late expert.
All aspects of assessment are evaluated in both the in-domain and zero-shot settings, to explore both the absolute performance of our proposed approach with respect to the literature (in-domain) as well as its ability to transfer to novel sports and scenarios (zero-shot). In the experiments we tackle multiple datasets of various sports and fitness activities, introduced next and detailed more in Sec. 3.5.
3.2 Skilled physical activity datasets
Before introducing our model, we next overview the key existing datasets leveraged in this study, since being familiar with their contents will help visualize our overall learning paradigm. Ego-Exo4D [37] contains 2,593 videos, totaling 239 hours with 289 total participants playing 3 sports ssccer, rock climbing, and basketball. Qualcomm Exercise Videos Dataset (QEVD) [68] has 223 long home fitness exercise videos, totaling 13 hours, where 28 total participants perform 23 structured workout

---

Test drili: Mid-range shooting
Incorrect skill-attribute: Jump
Proficiency: Late experi
Actionable feedback: Bend
Jm
knees more to jump generate power
Proficiency estimation
Fu Generating skill attribules
Fi Generaatng actionable feedback
fp
All sports except shooting
C
C
basketbal except shooting
2S-3: All sports except basketball
Figure 3: Method overview and evaluation settings. (Left) We encode videos into tokens that be fed to multimodal LLM L. with a mapper fm that trains for atili-attributes We use these visual tokens generate skill-attributes (bottom left). Next, this pretraining is used to generate actionable feedback (bottom middle) and proficiency score (bottom right). (Right) Example of the various training settings for in-domain and zero-shot.
moves (like jumping jacks, planks, squats, etc.). To stretch model in-the-wild zero-shot settings, we also show qualitative results using YouTube sports videos collected by us ur-fbieeee, water polo and soccer (juggling).
Both Egg-Exo4D and QEVD provide actionable feedback commentary for the videos, which consists of timestamped positive and negative critiques, ideas to correct form, pacing, or other aspects of execution. Each sport is critiqued by expert coaches or players from that same sport. Given a video V. the expert pauses the video at multiple timepoints i and provides verbal feedback, e.g.. "er. he's 's showing good control but lacks speed, which is critical to an effective dribble..." Ego-Exo4D additionally has proficiency labels corresponding to four distinct skill levels, from novice to late expert. See Figure and Supp. for examples.
3.3 Stage I: Discovering skill-attributes for pretraining supervision
To supervise the training of Fas obtain the skill-attributes set S for every video demonstration by sourcing from Ego-Exo4D and QEVD commentaries-toonn strings. We use these commentaries should improve on, as detailed next. While the Ego-Exo4D and QEVD expert commentary represent existing video-language datasets nicely suited for our purposes, such commentary could potentially be sourced "in the wild" from current video sharing platforms (e.g., Reddit, TikToo,, etc.) where people provide similar verbal assessments for videos shared on social media. See Supp.
around 34k unique feedback signal to extract the skill-attributes the learner in the video
For each training sample, we prompt large language model (LLM) [2] to extract skill-attributes that suboptimally demonstrated according to its expert commentary text I', We provide examples to help define the intent of skill-attributes to the LLM. See Supp. for full prompt. This process yields S {s1, $2, ... from the expert commentary T at time t. Note that |.S| can be of any length. Fig. shows word clouds the discovered skill-atrributes for the two video datasets. We observe that many salient phrases transcend sport boundaries, like body positioning, bulance, control, and movement.
Finally, we sample a video chunk around the time I, i.e.. [L [L- p1, + μ2] and associate it with the skill-atrribute set S.! By using an LIM combined with the raw commentary data, we obtain diverse, opca-locabulary skill- attriutti for every training sample. We draw on these resulting LLM-inferred annotations when training the multimodal LLM (Sec. 3.4).
3.4 Stage II: Skill assessment from video
Next we present our model design for learning F... Taking inspiration from the success of multimodal LLMs [44, 46, 53, 56], convert query video V into visual tokens. Next, we input the visual
'The commentary text has only a timepoint not temporal extent; this window extraction is consistent with how the temporal ambiguity handled in prior work 51, 76].
5

---

tokens to a multimodal LLM, along with a prompt to predict the skill-attributes S. The output is parsed to create the prediction S. Fig. 3 shows the overview, and we describe each step next.
Encoding video demonstrations. We first use standard video encoder fo (we use VLP [74] and CLIP [75], though others are possible) to extract video features from the video demonstration, ='= (VV,, see Fig. 3 (top left). Our model supports either single-view multi-view inputs. When available. multi-view observations of a skilled physical activity can give valuable detail; for example, an egocentric view of the basketball being shot by the hands along with one or more exocentric viewpoints of the player's 's full body pose would provide complementary information. When using multiple views, v' is simply the concatenation of features from individual views. The video spans ji + Hl seconds, as discussed in Sec. 3.3, and one feature is extracted per second. Therefore, v' is a vector of dimension (pi + (12) X N where N is the output feature dimension.
Note that our representation is video-frame based, meaning the person's 's body pose is visible in the RGB but not explicitly extracted. While one could alternatively supplement the input with explicit body poses, our early experiments showed that the accuracy advantage is minimal compared to the high compute needed to get state-of- of-the-ar art poses [79], i.e., 34 seconds for 10 second clip.
Multimodal LLM for skill-attribute generation. After encoding the videos, we input it to multimodal LLM (M-LLM), denoted as L. An LLM is suitable choice since the output attributes are expressed in text format. We prompt the model as follows: "<video> Here is a video of a person doing <sport name>. Highlight up <k> key concept areas where the person can improve:
Pretraining:
We replace the '<video>' tag with video encodings obtained above.
The output of the M-LLM contains 'k' skill-attributes, distinct phrases that represent the dimensions of suboptimal execution in the provided video demonstration. These outputs are parsed to obtain the skill-atributes set S. see Fig. 3 (bottom left). To this end, we employ a trainable mapper fm that converts the feature vector v' to representation compatible with the LLM, v = fm (v'). The idea is that fo is a large pretrained model and kept frozen, while fm is trainable to convert the visual features to multimodal representation. The supervising signal train this M-LLM is the skill-attribute set S obtained in Sec. 3.3. We use standard log- ligllikelihood loss [30, 53, 63, 86). We emphasize that this pretraining enables the model to generate skill-attributes, as opposed to retrieving from a closed set. We show the superiority of this approach over retrieval in the experiments.
This pretraining stage results in aligning the video features towards skill-attributes, thus making it suitable for the other axes of assessment cocessment-actiinaalee actionable feedback and proficiency estimation. We now use the pretraining weights for completing the assessment task suite, as described next.
Skill-attributes for actionable feedback. For generating expert actionable feedback, we provide the output skill-attributes S, along with video V, as input to the model L and output the actionable feedback. See Fig. 3 (bottom middle). Next, we prompt the model L with a prompt as follows: "<video> Here is a video of a person doing <sport name>. Here are some possible axes that need improvement, as rated by an Al coach (may contain mistakes): <S> Give feedback on the execution that will help the person improve." " As above, <video> is replaced by v. Importantly, this prompt provides both V and S an additional guiding signal for the actionable feedback generation process. The generated actionable feedback T contains ideas for improvement that are personalized to the video specifics, e.g., "you need to bend more while dribbling maintain control" as opposed to simply naming the skill dimension to improve, e.g., control. While skill-atributes can be the same for two sports, the resulting actionable feedback will be geared towards the specific sport. This factoring helps make our model transferable, while also enabling sport-specific actionable feedback.
Skill-attributes for proficiency estimation. Lastly, we deploy skill-attributes for proficiency esti- mation. As we want to discern how the skill-attributes representation compares to standard video features without such training, we employ linear probe setting where we have a linear layer fp that takes in frozen v and outputs a class representing the proficiency P, see Fig. (bottom right). The proficiency is label of expertise: novice, intermediate, expert, or late expert. Only fp is trainable.
3.5 Training and inference settings and implementation details
Train/test splits. Our approach aims to improve both traditional in-domain and zero-shot skill assessment. The datasets organize video clips by their superclass sport and their subclass skill. A skill is drill or specific exercise, and each sport can have multiple skills. Ego-Exo4D has 3

---

Table Quantitative results. Skill-attribute generation results (loU@0.7) for Ego-Exo4D [37] and QEVD [68]
(top left). Actionable feedback generation results Ego-Exo4D [37] (top right) and QEVD [68] (bottom
left). Metrics used to match SOTA on the corresponding dataset, B B@4=BLEU @4, M-Meteor, R-L-ROUGE-L,
B-BERT score. Proficiency estimation for individual sports (bottom right). Standard errors reported in text.
Skill-attribute generation
Actionable feedback Ego-Exo4D [37]
Method
IoU@0.7
Ego- Exo4D [37]
QEVD [68]
Method
B@4
M
R-L
Intern Video2-NN [88]
42.1
46.9
49.3
IntemVidco2-NN, [88]
14.0
23.8
Intern Video2-FT [88]
42.9
47.6
50.0
Intern Vndco2ido22FF [88]
15.0
24.5
VidcoChat2 [47]
27.8
44.3
41.9
VidcoChat2 [47]
9.3
16.9
LLaVA [53]
28.5
44.1
44,2
LLaVA [53]
9.7
17.3
LLaVA-FT [53]
43.5
48.5
51.5
LLaVA-FT [S3]
14.6
26.9
LLaVA-FT w/ pose [53]
43.6
48.5
51.7
Atream-VLM [68]
28.0
PoseScript/Fix [23, 24]
24.1
46.3
ExpertAF [9]
15.0
28.1
ExpertAF [9, 63)
44.9
49.6
54.6
Attribute-ettrivall
Retrieval
19.7
32.4
CROSSTRAINER
45.6
51.7
57.8
CROSSTRAINER
25.7
37.6
w/o two-stage
43.8
48.8
52.3
Actionable feedback on QEVD [68]
Method
R-L
Proficiency estimation Ego-Exo4D [37]
Socratie-LLaMA-2271
9.4
7.1
Method
B.ball
Soccer
Rock CI.
Tideo-ChatGPPe [56]
10.8
9.3
86.3
LLaMA-VID [50]
10.6
9.0
86.0
EgoVLPv2 [74]
48.0
62.5
34.0
Stream VLM [68]
12.7
11.2
86.3
CROSSTRAINES TRAINER
53.1
68.8
37.1
CROSSTRAINER
17.6
18.1
87.8
w/o two-stage
12.1
10.8
86.0
superclasses and 5 subclasses: soccer has skills dribbling and penalty kick; basketball has skills Mikan layup, reverse layup, jump shot; rock climbing (fitness) and 23 subclass skills (jumping jacks, squats, etc.). To explore models' generalization ability, we perform controlled experiments with the following train/test settings, in decreasing volume of available training data (see Fig. (right)):
sorted into skills. QEVD has superclass
Fully supervised (FS): Train on all sports and skills, and test on held-out set of videos. This represents in-domain testing.
All sport zero-shot (ZS-1): Train on all sports and skills, except the target skill. This means other skills from the same sport seen during training.
Familiar sport zero-shot (ZS-2): Train on all skills from the same sport, except the target skill. This means only the same sport is seen during training.
Novel sport zero-shot (ZS-3): Train only on n sports, test on skills from the n-th unseen sport. This means other skills from the same sport are not seen during training.
For each controlled experiment, we retrain the model to avoid any information leak between the train and the test splits, and between the seen and novel sports.
Model architecture. fo is EgoVLPv2 [74] for Ego-Exo44 [37] (precomputed with dataset) and CLIP [75] for QEVD [68] (lightweight extract). We use feature per second with 4096-d output. For Ego-Exo4D [37], we use the ego and four exo views, while QEVD [68] is single-view. fm is layered MLP with GELU activation, consistent with [52]. The MLP+GELU module takes in 4096-d representation, consistent with the embedding dimension of the multimodal LLM C, -lama-3.1-8B-Insteee [5]. fy is linear layer with output size 4, the number of proficiency levels.
Training details. We train both F, and J, in LoRA setting [38], with rank 128, alpha 256, and dropout 0.05, for efficiency. The best performance is obtained with learning rate of 2 x 10 for fm and 2 x 10 for C. Recall that fo is kept frozen. The model is trained for 2 epochs or till convergence. Total training time depends on the dataset setting, varying between 1 hours. All experiments are performed on one GH200 NVIDIA node.
7

---

Dribbiing
Penalty
Milkan
Mid Shooting
25
25
25
30
140%
20-
20
25
1001 88 0.7
20%
23%
20
2.5
15-
15-
15%
15
1026
27%
2006
10
4656
F5
25:1
25/2
F5
25-1
25/2
P5
25-1
252
25
P5
25-1
25/2
25/3
P5
25:1
Z5.2
25
54-
0%
0%
51
0%
51-
51-
54-
METROR
48
4B
48-
51-
48
4%
45-
10%
49-
45
45
42
F5
25-1
25-2
25-3
F5
25-1
25-2
25-3
P5
25-1
252
25.3
42
F5
25-1
25-2
25.3
42
F5
25-1
25.2
2
Figure 4: Zero-shot performance. Performance trend when testing on various skills (dribbling, penalty, etc.) for different in-domain and zero-shot training settings (FS, (Sc.,,tc.) for skill-attribte generation (top) and actionable feodback generation (bottom) for Exgo-ExoDD. The relative drop in performance w.r.t. FS is shown percentage. Our method is consistently the best for all methods, and the relative drop is the least for all zero-shot variants (ideal curve would be flat and high). See Supp. for QEVD, Legend: CROSSTRAINER, Intern Video2 [88].
ExpertAF [9] and
4 Experiments and Results
We first present actionable feedback generation (Sec. 4.1) for its focus in prior work and directly comparable SOTA methods. Next, we validate the skill-attributes pretraining (Sec. 4.2). Finally, we present proficiency estimation (Sec. 4.3) and explore long-tail in-the-wild YouTube videos (Sec. 4.4).
4.1 Generating actionable feedback
First we evaluate actionable feedback generation: given video, generate the natural language commentary explaining what to correct.
Baselines. We compare against the SOTA ExpertAF [9] and Stream- VLM [68] models well the original baselines from their respective experiments, which include strong multimodal LLMs with has fewer trainable parameters due use of LoRA [38], but
model size 7B-8B. CROSSTRAINTE we leave the baselines in their full (non-LoRA) form to report their most accurate numbers. "w/o two-stage" is the end-to-end ablation not using skill-attributes.
[11], BLEU-4 [69], ROUGE [78] and BERT-score [25], reported out of 100; higher is better. While establishing ground truth commentary is nuanced and there could be multiple valid feedback statements for a given video, prior work [9] shows that this evaluation paradigm correlates strongly with human subjects' evaluation of the generated commentary.
Metrics. Following [9, 68]. we report language
(top right and bottom left) shows the results. PROSSTRAINER clearly outperforms all prior work and strong baselines. Standard error is less than 0.1 for all metrics. This result clearly supports using skill-attributes as an intermediate representation for actionable foedback. We investigate the robustness of actionable feedback generation to the choice of the LLM L and the noise level in the Supp. Fig. 5 (first two rows) shows qualitative outputs for both datasets, where our model correctly captures the expert's ''s feedback.
Fully supervised in-domain results. Tab. 1
Zero-shot transfer. Fig. 4 (bottom row) shows the trend when transferring the knowledge from one domain to another in Ego-Exo4D [37] (see Supp. for QEVD [68]; results are similar). We plot the best M-LLM (ExpertAF [9]) and retrieval (Intern Video2-FT [88]) baselines; we see a similar trend in all other baselines. Our method obtains the best performance for all training settings and degrades most gracefully as training data coverage diminishes in the increasingly difficult zero-shot settings (7S-1, 7S-2, 7S-3). Our max drop 4%, vs. 17% for the baseline. Fig. 5 (bottom left), we show confusion matrix denoting better transfer from soccer to basketball, and vice versa,
8

---

GT. The player naods to improve by jumbiig on the roloase the jump shot get [hi recausseryy Iif, power art. and sccurncy.
GT: The player needs to adjust their hip and shoulder aligmmen to receive the ball jorr offectionnly with this putsida of their foot
Fgg-as4th
Fully sipervised
Towneed on woooer
Fgn-PoodD
Fulyy supervoed
Treene on besteibel
Ni batriboos sump
Shillattr dens sump
Ekill-ati buses: Hip and
Skillarmruooss Shoulder
Acdocitil feedlacck
Acdionabla feechack
shoulder a ligmment
ocsitio
player jump
release
Axtioratie feedbacs:
Actionable feedback The player should ressiving
nelessing
some sump
player sowards
penerate correct power
Proficiency. Early Expert
Profiliency, Early Expert
Profisiency: Lata Expert
Proficlancy: Late Expart
few fligginng foot.
body weight falls away from
up engagee your
keep moving Let's gol
Egp-Cartm
Fully supery
Towined beokefbel
QnVB
Fuly auperty Seed
GPVD
Etill-tt bunss Body positening
SkIl buses: Balance
SkIl buses:
ighimess
Acpionttle feedback The climber should not loavo a
Actooaabl fcedback oxtendied causing the fall off due intetenne.
Actioratie fredbark Good
Make your coros Sil toc.
nanginng foot as incresses
att bunas
pressure on hands.
Proficiennn Early Expert
Preficency:
Earty Expert
Actooretie faecherk el Keep up
YouTube
ASR
This big leopy motion dows not help deriva power, and It' too long a throw.
You Tube
ASR: over rotate
ASR: keep your knees stright this
your winst.
Sllttttibatess Hand pos lloning
Skili atrributes: bend
Skil-attibuuag Power
Artionaate freditack The player should loosen and band the kniws more adjust to the heil movemtent
Actionable feedhauce: The player should have less rotation the what tn now bether grip and throw powerr
Acdorattle foadbazk: Keep the hand plsser the body generate power
Novel Fistoos
Jugg rg
Theblig toxtt jewe- bariettor dirbing Soccar
Fadure Cieas
GT. The player should not look down when onbbling avoid loosing track of teammates.
49.9
50.2
$2.0
55.8
49.2
Skil-allribuler Knee bend, Leg posilioning
Actionable feedback: player should knaes more.
-50
GT: The player lacks intention while hitting the ball
47.6
49.5
492
47.7
48.7
-48
Sell--ottibuuee Control
46
Drbbling Panalty
Mayyupp Jump Shooting Rayyu
Actionable feedbadks The player should bend the
hody a bit more towards right for more control.
Figure 5: Qualitative results. CROSSTRAINER generates skill-attributes, actionable feedback, and proficiency for samples from both Ego-Exo4D [37] and QEVD [68). The outputs are meaningful even in the zero-shot setting (first two rows). Our method is also applied to in-the-wild vidcos from YouTube with novel sports (frisbee and water polo) and even new drills (jugling soccer) with feedback matching the YouTube expert's s comments (transcribed with ASR) (third row). Confusion matrix shows better transfer between related sports (bottom left). Failure cases here and in Supp. show the difficulty of the task, especially non-visual feedback like lacking intent (bottom right).
compared to rock climbing Hrimbinn-rrinforccinn reinforcing observations in cognitive science [43, 65, 81]. In summary, CROSSTRAINER demonstrates robust to transfer to novel sports and drills.
4.2 Learning skill-attributes in pretraining
Next we evaluate the skill- hill--ttribbte prediction (c.f. Sec. 3.3).
Baselines. We compare to retrieval baselines LintenVideed [88], nearest neighbor (NN) and finetuned (FT), and Attribute-Retrivaa trained with contrastive learning), zero-shot multimodal baselines (VideoChat2 [47], LLaVA [53]), and a finetuned version of the best multimodal baseline (LLFVA-FT). In addition, we again compare to ExpertAF [9], a SOTA model for generating actionable feedback, but here we convert its output to skill attributes by text-only LLM prompting [2]. All the baselines have access to the same actionable feedback data as ours [37, 68], and they directly supervise with it.
Metrics. Since skill-attributs a list of phrases, we employ IoU matching (a.k.a. Jaccard index) [21, 31] between the inferred annotations (Sec. 3.3) and generated skill skill-attributes. We call a pair a match if BERT-score is more than k and report l for k=0.7 here and {0.8, 1.0} in Supp.
Fully supervised in-domaín results. Tab. (top left) shows the results for Ego-Exo4D [37] and QEVD [68]. CROSSTR TRAIER outperforms all zero-shot and trained baselines that post process actionable feedback to obtain atrileats,, by more than 10%. Moreover, our method outperforms Attribute-Rctrivval by showcasinn the effectiveness of our generative approach. The standard

---

enror is less than 0.5 for all metries and all methods. Fig. 5 shows examples predicting the skill- attributes that the person should improve.
Zero-shot transfer: Fig. 4 (top row of plots) shows the zero-shot transfer results for Ego-Exo4D [37] (see Supp. for QEVD [68]; results are similar). As above, declines much more gracefully than the baselines. Overall,
clear advantage of our training
scheme. Moreover, CROSSTRAINGS these results show the effectiveness of our skill-attribute guided pretraining for zero-shot transfer.
4.3 Estimating demonstrator proficiency
Next we evaluate the quality of the learned video representation for proficiency estimation.
Baselines and metrics. We compare the strength of the video representation with respect to the frozen representation fu, i.e. VL [74] for all scenarios. We report classification accuracy.
Results. Tab. 1 (bottom right) shows the results. We see a clear gain of up to 6% when using the pretrained video representation, with standard error <1. Fig. 5 shows some example predictions, where our model can distinguish between early and late expert. This experiment suggests another potential use case in pretraining skill-centric representations for better assessment and feedback.
4.4 Testing long tail in-the-wild YouTube sports videos
Finally, we explore applying CROSSTRAINER novel sports and drills in in-the-wild videos from YouTube. We extract 10 clips from tutorial videos where an expert tutor demonstrates the incorrect way of doing a drill, while also explaining the incorrectness (i.e., the target feedback, which is withheld from our model). Details of obtaining the videos from YouTube are given in the Supp. We use the expert's 's transcribed ASR text to compare with the output of our model.
Results. Fig. 5 (fourth row) shows example results (rest in Supp.). We see that a model trained on Ego-Exo4D [37] videos (soccer, baskethall, rock climbing), is able to predict the issue with demonstrations in novel long-tail sports frisbee and water polo. Moreover, we also see correct feedback in novel juggling drill in soccer, which CROSSTRAINES is not trained with. We observe that learning a sport-specific vocabulary is difficult in zero-shot transfer. Nonetheless, the essence is captured and described in text, e.g., even though the model does not understand the loopy motion, understands that keeping the hands closer to the body helps in generating more power-a fact known and applied in various physical scenarios. We also verify the generations with user study. Human subjects not associated with the project rated 75% of the generations as actionable and correct. Every generation is judged by three raters, and we take majority vote.
Failure cases in Fig. 5 (bottom right) showcase the difficulty in capturing subtle mistakes, especially non-visual feedback like intent, decision. We further discuss limitations and societal impact in Supp.
5 Conclusion
We introduced CROSSTRAINER a novel approach that discovers skill-atributes from video demon- strations. These akill-attributes are generalizable, enabling zero-shot transfer of skill assessment to novel sports and drills. Our experiments show notable gains in actionable feedback generation and proficiency estimation for both in-domain and zero-shot settings. In the future, we will explore ways to quantify sport relatedness to predict the transferability between sports, as well as explicit representations to capture the environmental context.
Acknowledgements
Thanks to the anonymous NeurlPS reviewers for their valuable feedhack. This research is supported in part by the UT Austin IFML AI Institute. Compute is from the Vista GPU Cluster through the Center for Generative AI (CGAI) and the Texas Advanced Computing Center (TACC) at the University of Texas at Austin.
Note that QEVD does not provide proficiency labels.
10

---

References
[1] Yazan Abu Farha, Alexander Richard, and Juergen Gall. When will you do whatt-anticipating temporal occurrences of activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5343-5352, 2018. 3
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 1rXXi:3303308774 2023. 5, 9, 19
[3] Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagarajan, Huiyu Wang, and Lorenzo Torresani. Ht-step: Aligning instructional articles with how-to videos. In NeurIPS, 2023. 3
[4] Mistral AL Mistral 8b: A large language model. Online model release. 2024. Version vX. Y (check exact version number) under license (e.g., Apache-2.0). Available at: https://moddels.mistr hasta//assttittaaaaa/mastsaaall ai/mmstraal-8. 20
[5] AI@ Meta. Llama model card, 2024. 7, 19
[6] Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hierarchical video-language embeddings. In IEEE/CVE Conference on Computer Vision and Pattern Recognition
(CVPR), pages 23066-23377 2023. 3
[7] Kumar Ashutosh, Santhosh Kumar Ramakrishnan, Triantafyllos Afouras, and Kristen Grauman. Video- mined task graphs for keystep recognition in instructional videos. Advances in Neural Information Processing Systems, 36, 2024. 3
[8] Kumar Ashutosh, Zihui Xue, Tushar Nagarajan, and Kristen Grauman. Detours for navigating instructional videos. In IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 18804- 18815, 2024. 3
[9] Kumar Ashutosh, Tushar Nagarajan, Georgios Pavlakos, Kris Kitani, and Kristen Grauman. ExpenAF: Expert actionable feedback from video. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 1, 2, 3, 5, 7, 9, 20, 21
[10] Max Bain, Arsha Nagrani, Gul Varol, and Andrew Zisserman. Frozen time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7728-1738, 2021. 3
[11] Satanjeev Banerjee and Alon Lavie. Metcor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the act workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, 2005. 8
[12] Gedas Bertasius, Hyun Soo Park, Stella X Yu, and Jianbo Shi. Am i baller? basketball performance assessment from first-person vidcos. In Proceedings of the IEEE international conference computer vision, pages
2017. 1, 2, 3,
[13] Jing Bi, Jiebo Luo. and Chenliang Xu. Procedure planning in instructional videos via contextual modeling and model-based policy learning. In Proceedings of the IEEE//CV International Conference on Computer Vision, pages 1561 1-15200, 2021. 3
[14] Jing Bi, Jiebo Luo, and Chenliang Xu. Procedure planning instructional videos via contextual modeling and model-based policy learning. In Proceddings of the IEEE//VV International Conference on Computer Vision pages 1561 1-15200, 2021. 3
[15] Maxime Bucher, Tuan- Hung Vu, Matthicu Cord, and Patrick Pérez, Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019. 3
[16] James Burgess, Xiaohan Wang, Yuhui Zhang, Anita Rau, Alejandro Lozano, Lisa Dunlap. Trevor Darrell, and Serena Yeung-Levy. Video action differencing. arXiv preprint arXiv:2503.07866 2025. 1
[17) Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017. 1
[18] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv: 013400 2018. 1
[19] Joe Causer and Paul R Ford, "decisions. decisions, decisions": transfer and specificity of decision decision-making skill between sports. Cognitive Processing, 15:385-335 2014. 2.3

---

[20] Chien-Yi Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure planning in instructional videos. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI. pages 334-350. Springer, 2020.
[21] Yunmo Chen, William Gantt, Tongfei Chen, Aaron Steven White, and Benjamin Van Durme. unified view of evaluation metrics for structured prediction, arXiv preprint arX11 2023.
[22] CVsports, 2025. 3
[23] Ginger Delmas, Philippe Weinzaepfel. Thomas Lucas, Francesc Moreno-Noguet and Grégory Rogez. Posescript: 3d human poses from natural language. European Conference on Computer Vision, pages 346-362. Springer, 2022. 7
[24) Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno-Noguer, and Grégory Rogez. Posefix: Conrectig 3d human poses with natural language. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15018 15028. 2023. 7
[25] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018. 8, 20
[26] Zheng Ding, Jieke Wang. and Zhuowen Tu. Open-vocabulary universal image segmentation with maskelip. arXiv preprint arXiv:2208.0896: 2022.
[27] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11990 2020.
[28] Hazel Doughty, Dima Damen, and Walterio ranking for skill determination. In Proceedings of the IEFF Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Who's better? who's best"? pairwise deep
[29] Hazel Doughty. Walterio Mayol-Cuevas, and Dima Damen. The pros and cons: Rank-aware temporal attention for skill determination in long videos. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3
[30] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint arXiv: 2173, 2024.
[31] Sam Fletcher, Md Zahidul Islam, et al. Comparing sets of patterns with the jaccard index. Australasion Journal of Information Systems, 22, 2018. 9
[32] Antonino Furnari and Giovanni Maria Farinella. Rolling- colling-unroling Istms for action anticipation from first-person video. IEEE transactions on pattern analysis and machine intelligence, 43(11):4021-4336 2020.
[33] Jiyang Gao, Zhenheng Yang. and Ram Nevatia. Red: Reinforced encoder-decode, networks for action anticipation. arXiv preprint arXiv: 2017. 3
[34] Silvio Giancola, Mohieddine Amine, Tarek Dghaily, and Bernard Ghanem. Soccemet: A scalable dataset for action spotting in soccer videos. In Proceedings of the IEEE conference on computer vision and puttern recognition workshops, pages 1711-1721, 2018. 3
[35] Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 13505- 2021. 3
[36] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16102 16112, 2022.
[37] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exodd: Under- standing skilled human activity from first-and third-person perspectives. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 1, 2, 3, 4, 7, 8, 9, 10, 18, 20, 21, 27
[38] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li. Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022. 7,
12

---

[39] Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong. Yali Wang. Limin Wang. et al. Egoexolearn: dataset for bridging asynchronous ego-and exo-centric view of procedural activities real world. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20026, 2024.
[40] Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, and Ehsan Elhamifar. Open-vocabulary instance segmentation via robust cross-modal pseudo-laheling. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7020-7031, 2022. 3
[41] D. Jayaraman and K. Grauman. Zero-shot recognition with unreliable attributes. In NeurlPS, 2014. 3
[42] H. Larochelle, D. Erhan, and Y. Bengio. Zero-data learning of new tasks. In AAAI, 2008. 3
[43] Daniel Leach, Zoc Kolokotroni, and Andrew D Wilson. Perceptual information supports transfer of learning in coordinated rhythmic movement. Psychological Research, 85(3):1167-1122, 2021. 3, 4,9
[44] Chunyuan Li. Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large aanguage-andvviion assistant for biomedicine one day, arXiv preprint urXiv:2306.0088 2023.
[45] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint arXiv:2201. 009450, 2022. 3
[46] KunChang Li, Yinan He, Yi Wang, Yizhuo Li. Wenhai Wang, Ping Luo. Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, arXiv preprint arXiv:2305.0633 2023. 5, 28
[47] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video understanding benchmark, 2023. 7, 20
[48] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical encoder for video+ language oeni-repprssnntation pre-training. arXiv preprint Xixx:20055002000 2020. 3
[49] Yanghao Li. Chao-Yusa Wu, Haoqi Fan, Karttikeya Mangalam, Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4804-4814,
2022. 3
[50] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024.
[51] Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray. Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongeheng Tu, Wenzhe Zhao, Weijie Kong, al. Egocentric vidco-language pretraining. In Advances in Neural Information Processing Systems (NeurIPS), 2022, 3, 5
[52] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
[53] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint (Ci1:2304008483 2023. 5, 6, 7. 20, 28
[54] Jingen Liu, Ben Kuipers, and Silvio Savarese. Recognizing human actions by attributes. In CVPR, 2011.
[55] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353. 2020.
[56] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.0544 2023. 5,
[57] Esteve Valls Mascaro, Hyemin Ahn, and Dongheui Lee. Intention-condttionedd long-term human egocen- tric action forecasting @ ego4d challenge 2022. arXiv preprint wrxiv:2207.120000 2022. 3
[58] Effrrsyyi Mavroudi, Triantafyllos Afouras, and Lorenzo Torresani. Learning to ground instructional articles in videos through narrations, 2023.
13

---

[59] Antoine Miech, Dimitri Zhukov, Jean- Caan-Baptstte Alayrac, Makarand Tapaswi. Ivan Laptev, and Josef Sivic. Hootoo000:: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-2640, 2019.
[60] Antoine Miech, Jean- Jean-Baptinn Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisser- man. End-to-end learning of visual representations from uncurated instructional videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9879-9889, 2020. 1
[61] Tushar Nagarajan and Lorenzo Torresani. Step differences in instructional vidco. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024. 3
[62] Zwe Naing and Ehsan Elhamifar. Procedure completion by learning from partial summaries. In British Machine Vision Conference, 2020. 3
[63] OpenAL. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shappe
Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luiss Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson. Rory Carmichael, Brooke Chan, Che Chang. Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman. Juston Forte. Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni. Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo. Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billic Jonn, Heewoo Jun, Tomer Kaftan, Eukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretehen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung. Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez. Ryan Lowe. Patricia Lue, Anna Makanju. Kim Malfacini. Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McI.cavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reicchrr Nakano, Rajeev Nayak, Arvind Nee- lakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang. Cullen OeKeefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo. Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rac, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher. Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang. Nikolas Tezak, Madeleine B. Thompson. Phil Tillet. Amin Tootoonchian. Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergyya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wicthoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report. 2024. 6,
[64] OpenFAD, 2023. 3
[65] Luca Oppici and Derek Panchuk. Specific and general transfer of perceptual- skills and learning between sports: A systematic review. Psychology of Sport and Exercise, 59:102118, 082188,220001 2022. 2, 2.9
[66] M. Palatucci, D. Pomerleau, G. Hinton, and T. Mitchell. Zero-shot learning with semantic output codes. In NewrlPS, 2009. 3
14

---

[67] Yulu Pan, Ce Zhang. and Gedas Bertasius. Basket: A large-scale video dataset for fine-grained skill estimation. arXiv preprint wrXiv2553.20788) 2025. 1,3
[68] Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Böhm, Florian Dietrichkeit, Reza Pourrzaa, Xuanlin Li, Pulkit Madan, Mingu I Lee, et al. What to say and when to say it: Live fitesss coaching as a testhed for situated interaction. Advances in Neural Information Processing Systems, 777775833778882 2024, 1, 3, 4, 7, 8, 9, 10, 18, 19, 20, 21, 27
[69] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[70] Paritosh Parmar and Brendan Morris. Action quality assessment across multiple actions. 2019 IEEE Winter Conference on Applications of Computer Vision ( (WACV), pages 1468-141 1476, 2019. 2,
[71] Paritosh Parmar and Brendan Tran Morris. What and how well you performed? multitask learning approach to action quality assessment. Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 1,3
[72] Paritosh Parmar, Amol Gharat, and Helge Rhodin. Domain knowledge-informed self supervueee repre- sentations for workout form assessment, In European Conference on Computer Vision, pages 105-123. Springer, 2022. 1,2
[73] Hamed Pirsiavash, Carl Vondrick, and Antonio Torralba. Assessing the quality of actions. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 556-571, Springer, 2014. 1,3
[74] Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpy2: Egocentric vidco- video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
5285-5297, 2023. 3, 6, 7, 10
[75] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748-878 8763. PMLR, 2021. 3, 6,7
[76] Santhosh Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Naq: Leveraging narrations as queries to supervise episodic memory. In IEEF/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 5
[77] André Roca and A. Mark Williams. Does decision making transfer across similar and dissimilar sports? Psychology of Sport and Exercise, 31:40-43, 2017. 2
[78] Lin CY Rouge. A package for automatic evaluation summaries. In Proceedings of Workshop on Text Summarization of ACL, Spain, 2004.
[79] Soyong Shin, Juyong Kim, Eni Halilaj, and Michael Black. Wham: Reconstructing world-grounde humans with accurate 3d motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2070-2080, 2024. 6
[80] Nina Shvetsova, Anna Kukleva, Xudong Hong, Christian Rupprecht, Bernt Schiele, and Hilde Kuehne. Howtocaption: Prompting llms to transform video annotations at scale. European Conference on Computer Vision, pages 1-18. Springer, 2024. 3
[81] Winona Snapp-Childs, Andrew D Wilson, and Geoffrey P Bingham. Transfer of learning between unimamual and bimanual rhythmic movement coordination: Transfer is a function of the task dynamic. Experimental brain research, 233(7):2225 2238,2238822220 2015. 3, 4,9
[82] Ben William Strafford, Pawel Van Der Steen, Keith Davids, and Joseph Antony Stone. Parkour as donor sport for athletic development in youth team sports: Insights through an ccological dynamics lens. Sports medicine-open, 4:1-6, 2018. 2, 3, 4
[83] Tristan Sylvain, Linda Petrini, and Devon Hjelm. Locality and composiitionality in zero-shot learning. In ICLR, 2020. 3
[84] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1217-1216, 2019. 3
15

---

[85] Yansong Tang, Zanlin Ni. Jiahuan Zhou, Danyang Zhang, Jiwen Lu, Ying Wu, and Jie Zhou. Uncertainty- aware score distribution learning for action quality assessment. In Proceedings of the IEEE/CVF conference un computer vision and pattern recognition, pages 9839-9848, 2020. 1,3
[86]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Maya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao. Xavier Martinet, Todor Mihaylov. Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungla, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Mclanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[87] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang. Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang. Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. 2022. 1,3
[88] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Internvide02: Scaling video foundation models for multimodal video understanding.
Xu, Zun Wang, arXiv preprint WtXiv:2403.13377 2024. 3, 7, 8, 9, 19, 20
[89] Chao-Yuse Yuan Wu, Yanghao Li. Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mcmvit: Memory-augmenter multiscalc vision transformer for efficient long-term vidco recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattenn Recognition,
pages 13587-13597, 2022.
[90] Yonggin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning-a compre- hensive evaluation the good, the had and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251 -2265,,, 2018. 3
[91] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision (ECCV), pages 305 3211 2018.
[92] Hu Xu. Gargi Ghosh, Yao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoelip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprins arXiv: 14084, 2021. 1,
[93] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data, arXiv preprint 2rXiv:2309.1660 2023. 3
[94] Wenjia Xu, Yongqin Xian, Jiuniu Wang. Bernt Schiele, and Zeynep Akata, Attribute prototype network for zero-shot learning. In NeurlPS, 2020. 3
[95] Xun Xu, Timothy Hospedales, and Shaogang Gong. Semantic embedding space for zero-shot action recognition. In 2015 IEEE International Conference on Image Processing (ICIP), pages 63 67. IEEE, 2015. 3
[96] Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Vidco-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:22/2.0499 2022. 3
[97] Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang. Weina Ge, David Ross, and Cordelia Schmid. Unloc: A unified framework for video localization tasks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 13623-13633, 2023. 3
[98] Xumin Yu, Yongming Rao, Wenliang Zhao, Jiwen Lu, and Jie Zhou. Group-aware contrastive regression for action quality assessment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7928,7988 2021. 1, 2,3
[99] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. arXiv preprint arXiv:2004. 13931, 2020. 3
16

---

[100] Songyang Zhang, Houwen Peng. Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. In Proceedings of the AAAI Conference Artificial Intelligence, pages 12870-12877, 2020.
[101] Yiwu Zhong, Licheng Yu. Yang Bai, Shangwen Li, Xueting Yan, and Yin Li Learning procedure-aware video representation from instructional videos and their narrations. arXiv preprint arXiv:333.11833, 2023.
[102] Honglu Zhou, Roberto Martin Martinn Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Niebles. Procedure- aware pretraining for instructional video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10727-10736 2023. 3
[103] Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Foubey, Ivan Laptev, and Josel' Sivic. Cross-task weakly supervised learning from instructional vidcos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3537-3545, 2019.
17

---

A Supplementary material for Learning Skill-Attibbtes for Transferable Assessment in Video
A.1 Table of content
This supplementary contains the following:
A supplementary video that first motivates the problem with some example actionable feedback requests from learners on Reddit (discussed in Sec. 3.3). Next, we show qualitative results from Ego-Exo4D [37]. QEVD [68], and YouTube videos. Finally, we also show some failure cases.
Sec. A.2: LLM prompt that we use to obtain skill-commentary from expert commentary.
Sec. A.4: Additional quantitative results to show metrics for skill-attribute generation, and QEVD [68] zero-shot results.
We discuss limitations in Sec. A.5.
We also discuss societal impact in Sec. A.6.
A.2 LLM prompt for obtaining skill-attribute from expert commentary
In Sec. 3.3, discuss using a large language model (LLM) to extract skill-attributes that suboptimally performed. We use the following prompt:
System: Answer the question regarding a commentary about sports drill. Do not add information not present in the question.
The transcript of an expert commentating on a SPORT NAME COMES HERE drill given below. List down the concepts that are correct and incorrect in the drill, as noted by the expert. The concepts are distinct aspects of the skill, e.g., control, body positioning, speed, body movement, hand position, and so on. Feel free come up with newer concepts and write the response in two lines. The first line should contain the correctly shown concepts, and the second line should contain the incorrectly shown concepts. should be this format. Correct comma separated concepts. Incorrect comma separated concepts. Here is the expert feedback:
User:
NARRATION ATIOATION COMES HERF Assistant:
We prompt the LLM to provide both correctly and incorrectly demonstrated skill-attributes. Our Attribute-Retries baseline (Sec. 4.3) uses both of them.
A.3 Obtaining YouTube videos for testing long tail in-the-wild sports and drills
We evaluate the performance of our model on zero-shot sports and drills from in-the-wild YouTube videos in Sec. 4.4. To obtain dataset for this test, we create list of novel sports and novel drills within the Ego-Exo4D [37] and QEVD [68] sports (basketball, soccer, rock climbing, exercise), and search for videos YouTube with their coaching videos. Our criteria is that the vidco should contain a mistake done by a learner, and a coach giving feedback. Note that many videos show only the correct demonstration; hence, obtaining videos for task is challenging.
Specifically, we randomly selected some common sports soccer (juggling), basketball (dribbling) and some rarer ones like water polo, korfball, polo, jai alai. kin ball, frisbee. We search more than 50 videos with keywords like "Coaching video of <sport/drill>", "<sport/drill> training session for beginners", "<sport/drill> common mistakes for beginners", "Dos and don't it in <sport/drill>", The videos are manually watched to find the desired coaching instances. The process took overall 12 hours, and was done by graduate students not associated with this project.
18

---

Table 2: Buckets of exercises in QEVD [68] grouped by similarity in execution and effect.
Group name Stretches & mobility
Exercises
Remark
quad stretch, armcrosschest, good morning beginner, floor touches, toe touchers
Focused on fiexibility and range of mo- tion; often used in warm-up or cooldown phases. Involves static or slow dynamic movement.
Cardio & agility
high knees, quick feet, jumping jacks, air jump rope, butt kick- ers, puddle jumps
Elevates heart rate with low to moderate resistance; emphasizes agility and coor- dination with repetitive footwork. Targets glutes, quads, hamstrings through controlled or explosive leg movements. Builds strength and endurance.
Leg strength & lower- body
squats, squat jumps, squat kicks, walking lunges, lunge jumps, standing kicks
Core upper-hody
plank taps, moving plank, pushups, shoulder gators
Focuses on core stabilization and up- per hody strength, particularly arms, shoulders, and chest. Often bodyweight- based.
Full-body
boxing squat punches, mountain climbers
High-intensity, compound movements that engage multiple muscle groups while promoting coordination and rhythm.
Stretches
42
Cardle
Leg strength
Core
Full-body
36
39
6%
4%
7%
33
33
27
30
27
27
33
3
30
21-
24
21
18-
2514
21
21
15
25:1
25:1
15
25-1
254
25-2
25-1
Z5.
18
375
8%
3%
15-
6%
18
12
METFOR
12
10%
8%
24%
10%
12
8%6
115
12%
16%
9-
15%
12
1.05
242
345
24%
F5
25.1
25-2
F5
25:1
25-2
P5
25-1
252
25.3
F5
25-2
25/2
25.3
25-2
25.2
Z533
Figure 6: Zero-shot performance. Performance trend when testing on various skills (stretches, cardio, etc.) for different in-domain and zero-shot training settings (FS, ZS-1, etc.) for skill-atrribtte generation (top) and actionable feedback generation (bottom) for QEVD [68]. and Intern Video2 [88].
Legend: CROSSTRAINGE
Stream- Poream-LMM [68]
A.4 Additional quantitative results
In Sec. 4.2 and Tab. 1 (top left), we show results for generating skill-attributes for loU@0.7 We extend the table to show results on loU@k for & E {0.7, 0.8, 1.0] in Tab. 3.
Next, we show zero-shot transfer performance on the QEVD [68] dataset (summarized in main paper, and detailed here due space limitations). Note that all videos in QEVD are for fitness exercises, and they are not as distinct as different sport. Moreover, every video contains multiple exercises, with the transition labeled as "Moving (EXERCISE NAME)..." We use these labels to split the videos per-exercise. We discard instances that are before the start of any labeled exercise. Next, we create sport labels based on similarity in execution and effects. We create this division for the purpose of zero-shot transfer experiments. This split is created using consensus from ChatGPT-44 [2] and 3.a [5], and finally manually verified. The splits and the reasonings are given in Tab. 2. total of 23 unique exercises divided into 5 groups
19

---

Table 3: Quantitative results. Skill-atrribute generation results for loU@k for & E (0.7, 1.0) for Ego-
Exo4D [37] and QEVD [68], extension of Tab. (top left) on remaining values.
Method
IoU
@0.7
@0.8
1.0
Method
IoU
@0.7
@0.8
@ 1.0
Video--NNN [88]
14.0
8.9
7.7
Intern Video2-NN [88]
23.8
16.3
14.8
Intern Video2id [88]
15.0
9.4
8.2
Intern Video2-FT [88]
24.5
16.6
15.3
VideoChat2 [47]
9.3
6.6
4.0
VideoChat2 [47]
16.9
11.4
10.1
LLaVA [53]
9.7
7.2
LLaVA [53]
17.3
12.5
11.8
LLaVA-FT [53]
14.6
9.1
8.1
LLaVA-FT [53]
26.9
19.2
18.2
Stream-VI.M [68]
14.5
9.1
8.3
Stream-VILN [68]
28.0
19.9
18.6
ExpertAF [9]
15.0
9.5
8.4
ExpertAF [9]
28.1
19.7
18.3
Attribute-Retric
19.7
12.7
10.7
Muribute-Retrieval
32.4
24.7
23.0
CROSSTRAINER
25.7
15.9
CROSSTRAINER
37.6
29.8
28.1
Table 4: Robustness and sensitivity of LLM: Comparison of actionable feedback generation with various
sources of skill-attributes (left). Comparison BERT similarity score skill-attributes generated by our method with skill-attributes obtained from different LLMs and prompts (right).
Skill-attie attrib. source
Test-set C
B@4
M
R-L
Llama-3 8B (orig)
Llama-3 8B (orig)
45,6
51.7
57.8
Mistral 8B [4]
Llama-3 8B (orig)
45.3
51.8
57.5
Llama-3 8B (orig)
Mistral
45.8
51.8
57.8
Score
Llama-3 8B (orig)
Llama-3 prompt choice
0.99
w/ 10% noise
Llama-3 8B (orig)
45.2
50.5
56.2
Llama-3 prompt choice
0.98
w/ 20% noise
Llama-3 8B (orig)
45.1
50.0
54.9
Mistral 8B [4]
0.98
w/ 30% noise
Llama-3 8B (orig)
44.3
49.4
53.1
w/ 50% noise
Llama-3 8B (orig)
42.7
47.3
50.2
w/ 70% noise
Llama-3 8B (orig)
41.9
46.3
49.6
Fig. shows the results. First, in skill-attribute generation (top row), we see that our method outperforms both Stream-VLLN [68] and InterVVidco [88] baselines. Moreover, as seen in Ego- Exo4D [37]. the performance decrease in the zero-shot setting is milder than the drop observed in the baselines. Finally, we see similar trend in actionable feedback generation (bottom row). Overall, zero-shot results in both Ego-Exo4D [37] and QEVD [68] show that our idea of learning generalize using skill-attribun is effective.
Finally, to show the robustness and sensitivity, we perform the following experiments Ego- Exo4D [37] actionable feedback generation:
Robustness the choice of the language model: We train and evaluate skill- attributts using different language models. We first train the model using Mistral's 8B language model (mistralai/Ministrr IB-Inutruct-244 [4] and compare it against the skill-atrrbbues test set obtained using Llama-3. --B- Instruct, and vice-versa. Tab. 4 (left) shows the results. We see that the model robust to the choice of the language model, and using any strong language model helps achieve good performance.
Actionable feedback generation w/ noisy aill-aatributes: We inject noise in the actionable feedback generation evaluation. We replace X% of inferred skill-atribbues with a random skill-attribute and observe the performance at various levels noise See results in the table below. We observe that adding noise degrades the performance, with the performance matching that of end-to-end direct training at = 20%. We can conclude that the performance is positively correlated to the quality of the generated skill-attribute. Improving that will also improve the actionable feedback performance. These ablation studies further showease the effectiveness of using skill-atributes for actionable feedback.
Correlation between skill-attribute generated using different LLMs: We try two prompt variants, and different language model (Mistral 8B, MoaillaaiMiiiiaaalaaa -nnistral--B--nstrute--2-- (111 [4] to compare similarity between generated
our idea is independent the chosen language model. We use Hungarian matching to find the most-similar match between the new skill-atribbute set and our original skill-atrribtee set. Next, we find the average BERT score [25] between the sets. Tab 4 (right) shows the results. We see very high similarity between skill-attributes
-aeribbttss-checkingg
20

---

generated from different prompts, and a different language model. This result implies that our idea is independent of the choice of reasonable language model.
A.5 Limitations
We observe that the proposed model struggles with feedback that is about aspects not directly visible in the video. Phrases like lacking intent is not groundable definitively and hence, is not captured. Nevertheless, this inability to capture abstract notions is also observable in all the baselines. and in general, vision encoders. Secondly, as we discuss in Sec. 4.1, commentary is subjective and there can be various correct ways of providing feedback that improves a learner's 's performance.
Moreover, in this work, do not factor aspects like terrain and opponent behavior. This assump- tion is consistent with the datasets Ego-Exo4D [37] and QEVD [68] that are both single-person, and do not consider external factors. Furthermore, prior work also considers single-person skill assessmeen/feedbaekk [9, 12, 37J. There are research works in multi-agent cooperation, but they are restricted simulation and simpler objectives than performance feedback.
A.6 Societal impact
Our CROSSTRAINER can be used for learning skills, especially long-tailed low-resource sports like kho-kho, shinty. On the positive side, model democratizes access to skill coaching, and it promotes inclusivity in underrepresented sports. More learning will promote more people playing the sport, and eventually, more data for training and expansion of knowledge. However, the model is trained with Ego-Exo4D [37] and QEVD [68] that might have regional bias. We believe as more data is available, the biases will go down, and we will move closer towards full physical skill understanding.
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [Yes] We show results on transferring performance across various sports on multiple datasets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Justification:
[Yes] Sec. 4.4 discusses the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: [NA] We do not propose any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [Yes] Sec. 3.5 describes all the information required to reproduce the code. Furthermore, will release the code and the data upon paper acceptance.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer:
[No]
Justification:
[No] The code and data will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Beteiii/guudde//ooeeeue wttpssammisiioooooooiopooopoioii for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n cts.pss.cc/uubic/uuuuuu. gewds//////eeeeeee/o /CodeSubmian ss ierionnoolliii onoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
24

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: [Yes] The method section (See. 3) discusses the training and details, along with all the design choices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
[Yes] All result sections have the maximum standard error reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [Yes] The details compute resource and training duration given in Sec, 3.5. We did not have significant computational overhead due to failed experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htapps:////uripiie http:pp//aurrpps.pppppiiii////// wtw.hhiiiiciclllaaneeeeeaa
Answer: [Yes]
Justification:
[Yes] The authors confirm that this research follows the NeurIPS code of
ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
[Yes] Supplementary discusses societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
26

---

Justification: [Yes] We only use Exo-Exo4D [37] and QEVD [68] dataset for the research. Moreover, the generations of the language model is finetuned only generate expert commentary. We believe our trained model cannot be misused.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [Yes] have cited and ensured the codes and models have licenses that we can use for this research.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, shporswithcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: [Yes] All the details of the weakly-supervised dataset is provided in Sec. We will provide the documentation of the code along with its release to the community.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
27

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMss if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: [Yes] Our model design use LLM as the core component. This is a standard in recent video understanding methods [46, 53].
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: : htarrrpss.cc//ooneeenneee/////// for what should or should not be described.
28


Reviews:
[{"review_id": "6WLOT6mtSu", "reviewer": "Reviewer_98Cb", "summary": "The article suggests to use cross training approach as multiple sports have common skill set requirements. The model trained performs better as it has already learned the features which are common in the sport. Multiple set of experiments are conducted: fully supervised, zero shot, similar sport zero shot and unfamiliar sport zero shot. The results show that the approach works well and results are presented in quantitative as well as qualitative forms. The results demonstrate that approach is effective and it can help in real world scenarios where datasets for all different sports are not available.", "questions": "Please check comments above", "limitations": "Please check weaknesses mentioned above.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "strength\n- paper is well written\n- thorough experiments are conducted under different set of conditions to demonstrate the effectiveness of the approach\n- the approach is simple and is represented well by mathematics in the article. \n- the approach has many implementations as it is a concern in the literature to create individual datasets for each sport.\n\nweaknesses\n- the idea is not novel and has been used in many other fields in different forms such as computer vision and medical imaging\n- the approach is pretty much similar to fine-tuning of the existing models. It is slightly different as it is learning from the other datasets.", "was_revised": true, "final_justification": "Authors have submitted the rebuttal and clarified the comments. I have decided to raise the rating. However, authors need to embed the changes as mentioned in the rebuttal.", "rebuttal": "We thank the reviewer for their feedback. \n\n---\n\n**1: The idea is not novel and has been used in many other fields in different forms such as computer vision and medical imaging**\n\n\n\nOurs is the first work to specifically design transferable video representations for skill assessment (L5). We identify skill-attributes as the intermediary to help transfer feedback and proficiency between sports and drills (L40-43).\n\nSpecifically, our novelty lies in discovering the physical skill-attributes, and learning video representations that transcend the boundary between sports. We acknowledge and contrast with existing works in L94-103 using attributes in image recognition (L98), and general purpose zero-shot learning methods (L96-98). \n\nIf the reviewer could provide specific reference(s) that conflict with our novelty claims, we would be happy to contrast them with our key contributions.   \n\n---\n\n**2: The approach is pretty much similar to fine-tuning of the existing models. It is slightly different as it is learning from the other datasets.**\n\n\nOur implementation is inspired by the success in recent multimodal architectures [43, 45, 52, 55] (L170). However, our key novelty is in identifying skill-attributes as the medium to unlock transferability between sports (L5-8). \n\nMoreover, using a different dataset is not our contribution. We have existing fine-tuning methods that use the same dataset as baselines, and we outperform them because of our two-stage training approach that leverages skill-attributes (Sec 3 and Fig. 3). We outperform vanilla fine-tuning of existing models by a significant margin—retrieval baselines by >7% and multimodal finetuning by >3%, even when trained with the same dataset (Tab 1, right)."}, {"review_id": "1QJUjEKDib", "reviewer": "Reviewer_PsTp", "summary": "The paper addresses video-based skill assessment—including incorrect skill-attribute detection, actionable feedback generation, and proficiency estimation—and introduces the CROSSTRAINER framework. The core idea is to incorporate an intermediate representation, namely skill attributes, into the multimodal LLM (MLLM) pretraining and prediction process. Results show that CROSSTRAINER outperforms existing methods across various tasks, datasets (Ego-Exo4D, QEVD, and self-collected YouTube videos), and settings, particularly in diverse zero-shot scenarios such as testing on unseen skills from previously unseen sports during training.", "questions": "1. Please refer to Weakness 1 and provide additional training details for the zero-shot setups. Specifically, clarify whether any skill attributes derived from the test sports or skills were used during training, including the pretraining stage, to rule out potential information leakage.\n\n2. Please address Weakness 2 by clarifying whether the activity name (e.g., dribbling) is provided in the prompt during both training and testing. Additionally, indicate whether competing methods listed in Tab. 1 use a similar strategy, to ensure a fair comparison in actionable feedback generation.\n\n3. In response to Weaknesses 3 and 4, please explain why testing on a limited number of sports and skills is still considered meaningful. Furthermore, clarify the rationale behind selecting the three specific sports for YouTube video collection.\n\n4. Please address Weakness 5 by including additional ablation studies on skill attributes, if possible. In particular, analyze how variations in their diversity or quality influence overall performance.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper establishes a strong connection between the proposed method and cognitive science (e.g., the transferability of skills across different sports), enhancing its overall coherence and plausibility (Ln 55-62).\n\n2. The experimental results presented in Tab. 1, Fig. 4, and Fig. 5 are promising, highlighting the superiority of CROSSTRAINER over many state-of-the-art approaches, including ExpertAF [1], in both in-domain and zero-shot setups.\n\n3. A few failure cases are presented in Fig. 5 and discussed in the supplementary material, which is reasonable and highly helpful for understanding the method.\n\nMajor Weaknesses:\n\n1. Lack of essential training details for zero-shot setups: According to Section 3.3, the proposed method derives skill attributes from 32k Ego-Exo4D and QEVD commentaries. In zero-shot setups, were the skill attributes extracted from the “test” sports or skills excluded from all stages of training, including pretraining? Clarifying this is crucial to ensure that no information from the test sports or skills leaks into the training process.\n\n2. Potential unfair comparison: For actionable feedback generation, the proposed method utilizes video features, predicted skill attributes, and a prompt (Lines 204–206) as input. This prompt appears to include a placeholder for the activity name (e.g., &lt;activity&gt;). Does this imply that the actual activity name (e.g., dribbling) is inserted during both training and testing? If so, do the competing methods listed in Tab. 1 employ a similar mechanism for generating actionable feedback? Otherwise, the proposed method may have an advantage, as it is already provided with contextual information (e.g., the sport or skill) and only needs to generate feedback based on the attributes within this known context.\n\n3. Limited number of tested sports and skills: Although the proposed method aims to transfer skill knowledge across different sports and skills, the range of evaluated activities is quite narrow, comprising only 3 sports and 5 skills in Ego-Exo4D, 23 workout moves in QEVD, and 3 sports in the collected YouTube dataset. This limited scope makes it difficult to assess whether the observed performance gains are genuinely attributable to the method’s capabilities or simply to the specific choice of tasks. In fact, it is easy to imagine scenarios where skill transfer between sports may not be effective. For example, leaning forward may be beneficial in skiing but detrimental in basketball defense. Negative transfer [2] is a well-known challenge in machine learning, further underscoring the need for the proposed method to be evaluated across a broader and more diverse set of sports and skills to validate its generalizability.\n\n4. The paper does not provide justification for the selection of the three sports, frisbee, water polo, and a juggling drill in soccer, for collecting the YouTube videos. Clarifying the rationale behind choosing these specific activities is important to assess whether the proposed method can truly generalize to a broader range of sports and skills.\n\nMinor Weaknesses:\n\n5. The paper lacks ablation studies on skill attributes, particularly regarding how their diversity or quality impacts final performance. This analysis is important to support the claim in Ln 155–157 that the proposed method can effectively leverage commentaries collected from online video-sharing platforms. Without such an evaluation, it remains unclear how robust or sensitive the method is to variations in the source and quality of skill attributes.\n\n[1] Ashutosh, Kumar, et al. \"ExpertAF: Expert actionable feedback from video.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Zhang, Wen, et al. \"A survey on negative transfer.\" IEEE/CAA Journal of Automatica Sinica 10.2 (2022): 305-329.", "was_revised": true, "final_justification": "Although I still believe it would be better to evaluate the proposed method on a more diverse set of sports and skills to fully validate its claims on transferability, most of my concerns have been addressed within the authors’ rebuttal, including the addition of ablation studies on skill attributes, a key component of the proposed method. As a result, I have decided to slightly raise my score for the paper.", "rebuttal": "We thank the reviewer for their feedback. All the requested clarifications, and additional ablations for the choice of LLM are addressed below:\n\n---\n\n**1: Lack of essential training details for zero-shot setups.**\n\nThis is a possible misunderstanding. Our model **generates** skill-attributes, as opposed to _retrieving_ them. Thus, we do not need a list of skill-attributes at test time; the model is capable of generating it (Sec. 3.1 and 3.3). We do not maintain any list of skill-attributes to use with the test or train set. In Sec. 3.3, we mention 32k skill-attributes to signify the scale and diversity of the skill-attributes from all the training samples. In summary, there is no information leak between the training and the testing set.\n\n----\n\n**2: Clarification regarding the use of \\<activity\\> in the prompt.**\n\nIt is correct that the activity name is inserted during both training and testing. Importantly, the same is done for all the baselines. In particular, the prompt format is the same in ExpertAF [8], LLaVA [52], VideoChat2 [46], and Steam-VLM [67]. In retrieval baselines, the candidates are only from the mentioned activity. Note that \\<activity\\> here refers to the sport, e.g., soccer, and not the particular drill, e.g., dribbling (same as L189). Knowing which sport is being played is a reasonable assumption that real use cases would make, i.e., the system would know it’s being asked for feedback on tennis or basketball etc. \n\n\n----\n\n**3: Limited number of tested sports and skills, and potential negative transfer.**\n\nWe believe this total of 31 skills/activities with ground truth commentary is sizable and represents the total available relevant data for evaluation.  We welcome pointers from the reviewer if we have overlooked any additional dataset that would broaden evaluation even further.  \n\nTo evaluate coaching feedback, the required ground truth must be of ‘expert quality’. Ego-Exo4D [36] and QEVD [67] hired sports professionals for curating the dataset (both papers are from industry labs). We leverage these datasets for our task. Prior work evaluates physical skill at the same or much lower scale— [11] uses only 48 single sport (basketball) videos; AQA [69] contains six activities totaling 803 videos; [36] and [8] are comparable since they use Ego-Exo4D [36], but we additionally use QEVD [67]. In comparison, we have 31 skill/activities, spanning 2800 videos, if anything increasing the breadth of testing for this area.\n\nWe agree that negative transfer is an important risk in transfer and multi-task learning.  Precisely because of this variability between sports, we perform zero-shot transfer at the skill-attribute stage, as opposed to performing zero-shot transfer from the actionable feedback. We hypothesize that the exact actionable feedback depends jointly on the incorrect skill-attribute and the current sport. For example, for the reviewer’s example of skiing/basketball, we expect the model to flag ‘leaning’ as an incorrect skill-attribute. Next, actionable feedback for skiing would indicate leaning to be better, vs in basketball, it would advise against leaning when defending. This observation is discussed in L209-212.\n\nNevertheless, we acknowledge that transfer between certain sports is more suitable than others. Fig. 5 (bottom left) shows a better transfer between soccer and basketball, than rock climbing.\n\n---\n\n**4: Justification for the selected sports for the YouTube dataset (3rd dataset we tested).**\n\nWe create a list of novel sports and novel drills within the Ego-Exo4D and QEVD sports (basketball, soccer, rock climbing, exercise), and search for videos on YouTube with their coaching videos. Our criteria is that the video should contain a mistake done by a learner, and a coach giving feedback. Note that many videos show only the correct demonstration; hence, obtaining videos for our task is challenging. \n\nSpecifically, we randomly selected some common sports—soccer (juggling), basketball (dribbling) and some rarer ones like water polo, korfball, polo, jai alai, kin ball, frisbee. We search more than 50 videos with keywords like “Coaching video of <sport/drill>”, “<sport/drill> training session for beginners”, “<sport/drill> common mistakes for beginners”, “Dos and don’ts in <sport/drill>”. The videos are manually watched to find the desired coaching instances. The process took overall 12 hours, and was done by graduate students not associated with this project. We will add this detail in the paper.\n\n---\n\n**5. Ablation studies on skill attributes**\n\nWe deployed only one SOTA LLM in our pipeline (Llama 3.1 8B), for simplicity, but we appreciate interest in knowing how sensitive our full pipeline is with respect to that choice. To show the robustness and sensitivity, we perform the following additional experiments on Ego-Exo4D actionable feedback generation (Tab 1, right):\n\n- Robustness to the choice of the language model: We train and evaluate skill-attributes using different language models. We first train the model using Mistral’s 8B language model (mistralai/Ministral-8B-Instruct-2410) and compare it against the skill-attributes test set obtained using Llama-3.1-8B-Instruct, and vice-versa. See the table below for results. We see that the model is robust to the choice of the language model, and using any strong language model helps achieve a good performance.\n\n\n- Actionable feedback generation w/ noisy skill-attributes: We inject noise in the actionable feedback generation evaluation. We replace X% of inferred skill-attributes with a random skill-attribute and observe the performance at various levels of noise X. See results in the table below. We observe that adding noise degrades the performance, with the performance matching that of end-to-end direct training at X=20%. We can conclude that the performance is positively correlated to the quality of the generated skill-attribute. Improving that will also improve the actionable feedback performance.\n\n\n| Skill-attribute source | Test-set LLM | BLEU-4 | METEOR | ROUGE-L\n|-------------------------------|-------|-------|-------|-------|\n| Llama-3 8B (original)| Llama-3 8B (original)   | 45.6  | 51.7 | 57.8  |\n| Mistral 8B | Llama-3 8B (original)   | 45.3 | 51.8 | 57.5 |\n| Llama-3 8B (original)| Mistral 8B   | 45.8 | 51.8 | 57.8 |\n| **===========================================** | **======================** | **====** | **=====** | **======** |\n| Llama-3 8B (original) w/ 10% noise                    | Llama-3 8B (original)  | 45.2  | 50.5   | 56.2  |\n| Llama-3 8B (original) w/ 20% noise                    | Llama-3 8B (original)  | 45.1  | 50.0  | 54.9  |\n| Llama-3 8B (original) w/ 30% noise                    | Llama-3 8B (original)  | 44.3  | 49.4  | 53.1  |\n| Llama-3 8B (original) w/ 50% noise                    | Llama-3 8B (original)  | 42.7  | 47.3  | 50.2  |\n| Llama-3 8B (original) w/ 70% noise                    | Llama-3 8B (original)  | 41.9  | 46.3  | 49.6  |\n\nThese ablation studies further showcase the effectiveness of using skill-attributes for actionable feedback. We will add this ablation in the paper."}, {"review_id": "Opls24SFlw", "reviewer": "Reviewer_ph2p", "summary": "This paper tackles the problem of video-based skill assessment, aiming to overcome a major limitation of existing methods: their poor transferability across different sports or activities. The authors propose an approach centered on the idea of learning a set of universal, sport-agnostic skill-attributes like balance, control and body positioning. The core contribution is a two-stage training paradigm. First, they cleverly bootstrap these skill-attributes by using a LLM to parse existing expert commentaries from datasets, thus creating weakly-supervised labels. Second, they pre-train a multimodal model to predict these attributes from video, and then finetune it to generate actionable feedback and estimate a performer's proficiency level. The experiments conducted on datasets like Ego-Exo4D and QEVD, demonstrate that this approach  improves in-domain performance and shows significantly better generalization to novel sports and drills in various zero-shot settings compared to state-of-the-art methods.", "questions": "1.  The paper compellingly argues that the two-stage, attribute-centric approach is superior to other SOTA models. To more directly isolate and quantify the benefit of the proposed pre-training stage, have you considered running an ablation study where the exact same  architecture is trained end-to-end? Specifically, a baseline trained to go directly from video to actionable feedback, without the intermediate skill-attribute discovery and pre-training stage. This would provide a very clean and powerful demonstration of the value added by your proposed pre-training methodology.\n\n2. Regarding the crucial skill-attribute discovery stage (Sec 3.3): Could you provide more analysis on the quality and stability of the LLM-generated attributes? How consistent are the extracted attributes if a different LLM or slightly different prompts are used? \n\n3. The paper mentions creating open-vocabulary skill-attributes. Is it a fully open vocabulary discovered on the fly, or is there a process to consolidate or cluster similar LLM outputs into a more canonical set of attributes?\n\n4. Have you conducted any human studies to assess whether the generated feedback is perceived as actionable and correct, especially for novel sports?\n\n5. In the ablation-like discussion, you mention that adding explicit body pose features didn't provide a significant accuracy boost relative to the computational cost. Could you quantify this trade-off?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses the problem of the lack of transferability in action quality assessment. The core idea of identifying and using intermediate skill-attributes as a transferable representation is insightful. It provides a structured way to reason about why a performance is good or bad, which aligns well with the principles of AQA and effective coaching.\n\n2. The setup for testing transferability with its multiple zero-shot configurations (ZS-1, ZS-2, ZS-3), is rigorous and provides strong evidence for their main claim. The results show their method outperforming strong baselines, especially in the challenging zero-shot scenarios where it degrades much more gracefully.\n\nWeaknesses:\n\n1. The quality of the bootstrapped skill-attributes is fundamental to the entire pipeline, yet this process relies heavily on an LLM prompted to extract concepts from text. The robustness of this extraction process to different LLMs, prompts, or noisy/ambiguous commentary is not deeply analyzed. \n\n2. The overall framework of generating soft labels using LLM then fintune LoRA and the underlying model architecture like a frozen video encoder, a trainable mapping network, and a frozen LLM is a relatively standard pattern in recent multimodal research and lacks novelty. \n\n3. While the experimental evaluation is extensive, the central claim about the necessity of the skill-attribute pre-training stage could be further substantiated with more direct ablation studies, as detailed in the questions below.", "was_revised": true, "final_justification": "The rebuttal addresses most of my concerns, I will raise my assessment of the paper to 4.", "rebuttal": "We thank the reviewer for their feedback and questions. We provide the clarifications below, including a requested baseline already implemented:\n\n\n---\n\n**1. Ablation studies on skill attributes**\n\nPlease see our response to reviewer PsTp’s #5.\n\n---\n\n**2. Overall framework lacks novelty**\n\nWhile our implementation is indeed inspired by the success in recent multimodal architectures [43, 45, 52, 55], our key novelty is in identifying skill-attributes as the medium to unlock transferability between sports (L5-8). Moreover, we outperform vanilla fine-tuning of existing LLMs by a large margin—LLaVA-FT [87] (Tab 1). These methods also use the standard pattern of using LLM and LoRA finetuning, even when trained with the same dataset. Overall, our method correctly uses skill-attributes, both to improve in-domain performance, and generalize better to novel scenarios—more than the vanilla LLM finetuning framework.\n\n---\n\n**3. End-to-end training vs two-stage, attribute-centric approach**\n\nGood point, and please note that we have this exact baseline. ExpertAF [8] and Stream-VLM [67] precisely do what the reviewer described (Tab 1). In actionable feedback generation, these methods train an end-to-end model — going directly from video to actionable feedback. The implementation in [8] fully fine-tunes the LLM; reproducing that in our LoRA training setting with a more recent LLM gives slightly lower performance. Thus, we compare directly with the reported number and outperform it by up to 3% (L254-256).\n\nFurthermore, for skill-attributes generation, we use the end-to-end actionable feedback output to extract the skill-attributes.\n\n---\n\n\n**4. Quality and stability of LLM-generated attributes with different LLM or different prompt**\n\nPlease see our response to reviewer BeW2’s #3 where we show high similarity between the generated attribute set with different prompts, and a different LLM (Mistral 8B).\n\n---\n\n**5. Are the skill-attributes fully open vocabulary discovered on the fly?**\n\nYes, it is a fully open vocabulary. At the test time, we use a generative model (Sec 3.3) to generate skill-attributes. This ensures the model is free to generate skill-attributes that are not verbatim seen in the training, but related to it (L164-166).\n\n---\n\n**6: Have you conducted any human studies to assess whether the generated feedback is perceived as actionable and correct, especially for novel sports?**\n\nFor the novel sports collected from YouTube, the data collection process ensures every video contains ground truth description of the actionable feedback. Human subjects not associated with the project rated 75% of the generations as actionable and correct, compared to the ground truth—even when the sport is unseen during training. Additionally, for actionable feedback generation, ExpertAF [8] conducted a human study, and found a positive correlation between the automatic metrics and human preference.\n\n---\n\n**7. Trade-off in using explicit body-pose**\n\nConsistent with the observation in ExpertAF [8], using explicit body-pose increases the performance by ~0.5% in BLEU-4, ROUGE and METEOR (Tab 1, right). However, in order to use the additional pose input, we need to train a pose encoder (similar to PCT, Geng et al. CVPR 2023, “Human Pose as Compositional Tokens”), and a pose mapper, similar to the visual mapper (L193). Since explicit body-pose is not the primary contribution of this work, and we obtain stronger results than ExpertAF [8] without pose, we do not use it."}, {"review_id": "axTO02D2BI", "reviewer": "Reviewer_BeW2", "summary": "The paper proposes CROSSTRAINER, a method for video-based skill assessment that learns transferable skill-attributes (e.g., control, balance) shared across sports. It uses a multimodal LLM to predict incorrect attributes, generate actionable feedback, and estimate proficiency", "questions": "How do you ensure the extracted skill-attributes are accurate and consistent? Are they verified or clustered?\nDo you expect CROSSTRAINER to work for non-physical tasks (e.g., surgery, industrial skills)? Any thoughts on adapting it?", "limitations": "The method still depends on expert commentary for supervision. Addressing how this could be scaled or replaced with weak/noisy supervision would be useful.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Demonstrates consistent improvements over SOTA baselines across skill-attribute prediction, feedback generation, and proficiency estimation.\n2. Handles unseen skills and sports with minimal performance drop, showing strong generalization across tasks and domains.\n3. Decomposing feedback into generalizable “skill-attributes” makes the model more interpretable and adaptable.\n\nWeakness:\n1. While skill-attributes reduce label cost, they still rely on curated expert feedback for supervision, which may not scale well.\n2. Current formulation lacks mechanisms for reasoning about external factors (e.g., terrain, opponent behavior) that may influence skill.\n3. Focuses only on individual physical tasks; unclear if the approach extends to multi-agent, strategic, or cognitive skills.\n4. Heavy reliance on LLM prompting and proprietary components (e.g., LLaMA-3) without detailed insight into prompt design or tuning.", "was_revised": true, "final_justification": "The rebuttal is clear to me, I will raise the score.", "rebuttal": "We thank the reviewer for the feedback. All the concerns are addressed below:\n\n---\n**1: While skill-attributes reduce label cost, they still rely on curated expert feedback for supervision, which may not scale well.**\n\nCurated expert feedback is indeed expensive, which motivates our approach for zero-shot transfer of feedback across sports—effectively reducing the need for curating supervision. That is, our proposed generalization ensures we scale better than all the baselines that are tied to the specific sport [87, 46, 52, 67, 8] (Tab 1, Fig. 4). Our method achieves better performance in novel scenarios and sports without explicit expert knowledge of the sport (Fig. 4 and L267-275).\n\n---\n\n**2: Can the current formulation handle external factors (e.g., terrain, opponent behavior), multi-agent, strategic, cognitive skills, surgery, industrial skills?**\n\nTerrain and multi-agent opponent behavior: That’s a good observation. We do not factor terrain and opponent behavior in this work. This assumption is consistent with the datasets Ego-Exo4D [36] and QEVD [67] that are both single-person, and do not consider external factors. Furthermore, prior work also considers single-person skill assessment/feedback [11, 36, 67, 69, 70, 72]. There are research works in multi-agent cooperation [A, B], but they are restricted to simulation and simpler objectives than performance feedback.\n\nExtending this method to external factors and opponent behavior would require additional skill-attributes like coordination, relative placement, spatial awareness, etc. and a similar training approach as ours. We do not consider this scenario due to the lack of training data, and prior methods; this is an interesting extension of our work.\n\nStrategic and cognitive skills: We do not focus on strategic or cognitive skills since they are not necessarily visual—hence using visual encoders and visual models might not be required. \n\nSurgery and industrial skills: We believe our method can be extended to surgery and industrial skills. However, adapting it to different domains would require a careful choice of visual encoders, e.g., surgery may only need hand pose encodings, and corresponding skill-attributes. Nonetheless, it is an interesting future work, provided there are datasets for such supervision with actionable feedback.\n\n\nReferences:\n\nA: Liu et al., ICLR 19, “Emergent Coordination Through Competition”\n\nB: Lowe et al., NeurIPS 17, “Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments”\n\n---\n\n**3: Insights into the generated skill-attributes — prompt design tuning, accuracy and consistency, and if they are verified and clustered.**\n\nPrompt design tuning and consistency w/ different language model: Thanks for the suggestion for prompt design tuning and checking for the consistency by choosing a different open-sourced language model (Llama-3 is an open sourced model). We deployed only one SOTA LLM in our pipeline, for simplicity, but we appreciate interest in knowing how sensitive our full pipeline is with respect to that choice. \n\nWe try two prompt variants, and a different language model (Mistral 8B, mistralai/Ministral-8B-Instruct-2410) to compare the similarity between generated skill-attributes—implying our idea is independent of the chosen language model. We use Hungarian matching to find the most-similar match between the new skill-attribute set and our original skill-attribute set. Next, we find the average BERTScore (Zhang et. al, ICLR 20) between the sets. The similarity is tabulated below:\n\n| Comparing model (Ours vs)  | Score |\n|-------------------------------|-------|\n| Llama-3 8B w/ prompt choice 1 | 0.99   |\n| Llama-3 8B w/ prompt choice 2 | 0.98   |\n| Mistral 8B                    | 0.98  |\n\nWe see a very high similarity between skill-attributes generated from different prompts, and a different language model. This result implies that our idea is independent of the choice of a reasonable language model.\n\nAccuracy of the skill-attributes: We ensure the accuracy of the skill-attributes by manually verifying the skill-attributes for the videos in the test set. The manual verification process involves ensuring the skill-attribute correctly reflects actionable feedback mentioned by the expert. The test set contains 1272 videos, and we only had to edit only 0.2% of the skill-attributes. We will add this in the paper.\n\nAre they clustered?: We do not restrict the scope of the skill-attributes, and hence, we do not cluster the skill-attributes. The skill-attributes are generated by our proposed model (Fig 3 and Sec. 3.3), and then the predicted skill-attribute is used for actionable feedback generation, and proficiency estimation (Sec. 3.4).\n\n\n---\n\n**4. Addressing how expert feedback training supervision could be scaled or replaced with weak/noisy supervision would be useful.**\n\nIn this work, we focus on maximizing the ability to provide skill assessment given the current scale of the expert commentary (L45-47). We also demonstrate the ability to transfer the assessment across the long tail of sports (L33-35).\n\nThe scale of the expert feedback can be increased by having more annotations by subject experts (costly), or leveraging knowledge from the internet, e.g., YouTube, Reddit (L155-156). Increasing the expert feedback scale for training and evaluation is an orthogonal line of research; any improvement on that front will further expand the applicability of our approach. Moreover, the supervision cannot be scaled equally for all sports, due to the interest and the availability of the data, e.g., soccer is more popular, and has more data, than korfball. Therefore, the problem of long-tail transfer will always be crucial, as shown in our work."}]

Author rebuttal:

Meta-review: This paper proposes "CrossTrainer", a method for video-based skill assessment that learns transferable "skill-attributes" (e.g., balance, control) to generalize across different sports. The core idea is to discover these sport-agnostic attributes from expert commentaries and then use them to train a multimodal model to provide feedback and proficiency scores.

The main strength is the novel concept of skill-attributes, which provides an more interpretable and effective way to achieve transfer learning in skill assessment, a significant challenge in the field. The paper is supported by thorough experiments, demonstrating substantial gains over state-of-the-art methods, especially in various zero-shot settings.

Initial weaknesses raised by reviewers included concerns about the novelty of the overall architecture (Reviewers 98Cb, ph2p), the robustness of the LLM-based attribute generation (Reviewer ph2p), and the need for more direct ablations (Reviewer ph2p). During the rebuttal, the authors successfully clarified that the novelty lies in the concept of skill-attributes, not the architecture. They provided new ablation studies showing the robustness of their attribute generation process to different LLMs and prompts. These clarifications and additional results effectively addressed the reviewers' concerns, leading multiple reviewers to raise their scores and resulting in a consensus for acceptance.
Decision: Accept (poster)