Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understanding
Authors: Kaiting Liu
Hazel Doughty
Venue: ICLR 2026 Poster (2026)
Keywords: Video Understanding
Representation Learning
Action Recognition
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Video recognition models are typically trained on fixed taxonomies which are often too coarse, collapsing distinctions in object, manner or outcome under a single label. As tasks and definitions evolve, such models cannot accommodate emerging distinctions and collecting new annotations and retraining to accommodate such changes is costly. To address these challenges, we introduce category splitting, a new task where an existing classifier is edited to refine a coarse category into finer subcategories, while preserving accuracy elsewhere. We propose a zero-shot editing method that leverages the latent compositional structure of video classifiers to expose fine-grained distinctions without additional data. We further show that low-shot fine-tuning, while simple, is highly effective and benefits from our zero-shot initialization. Experiments on our new video benchmarks for category splitting demonstrate that our method substantially outperforms vision-language baselines, improving accuracy on the newly split categories without sacrificing performance on the rest. Project page: https://kaitingliu.github.io/Category-Splitting/.

Published as a conference paper at ICLR 2026
LET'S SPLIT UP: ZERO-SHOT CLASSIFIER EDITS FOR FINE-GRAINED VIDEO UNDERSTANDING
Kaiting Liu, Hazel Doughty Leiden University
ABSTRACT
Video recognition models are typically trained on fixed taxonomies which are of- ten too coarse, collapsing distinctions in object, manner or outcome under single label. As tasks and definitions evolve, such models cannot accommodate emerging distinctions and collecting new annotations and retraining to accommodate such changes is costly. To address these challenges, we introduce category splitring, a new task where an existing classifier is edited to refine a coarse category into finer subcategories, while preserving accuracy elsewhere. We propose a zero-shot edit- ing method that leverages the latent compositional structure of video classifiers to expose fine-gree fine-grained distinctions without additional data. We further show that low-shot fine-tuning, while simple, is highly effective and benefits from our zero- shot initialization. Experiments on our new video benchmarks for category split- ting demonstrate that our method substantially outperforms vision- caiionlanguage base- lines, improving accuracy on the newly split categories without sacrificing perfor- mance on the rest. Project page: https:/kaiiiiglinngiinn wtw.gggthhhb..hggggoooo Coaaeggr Spllttiinn/ggg/////
INTRODUCTION
Categorization underlies recognition in vision, yet most models assume a fixed taxonomy that rarely matches real-world complexity. A single label can cover many visually distinct cases, and as appli- cations mature, new distinctions often become important. In video understanding, such refinements are especially common: subtle differences in motion, timing, or object interactions can completely change the meaning of an action. For example, the broad label open can mask distinctions by object (open cupboard), manner (open by pushing), speed (open quickly) or outcome (open halfway). These examples highlight that action categories are inherently multidimensional, yet current recognition models commit to single, fixed partition.
A straightforward solution to retrain the model with new annotations. Yet this costly, requiring extensive labeled data and full training cycle. Vision-language models (VLMs) (Radford et al., 2021: Zhao et al., 2024) appear to offer a shortcut, allowing new categories at test time via text prompts. However, VLMS rely on massive video-text corpora that are rarely available in special- ized domains, and seldom capture the subtle temporal cues of fine-gran -raingriidd actions. Continual learn- ing (Wang al., 2024b) provides another angle, aiming to expand the label space without forgetting. However it assumes access to training data for each new class and targets entirely novel categories rather than considering relationships to existing ones. Current approaches therefore fall short when an existing category must be divided into finer subcategories with little or no supervision.
We address this challenge by introducing category splitting (Figure 1). The task is edit an existing classifier to refine a coarse label into fine-grained subcategories, while preserving other predictions. We focus on actions, where fine-grained distinctions are common and challenging, often hinging on subtle motion or temporal cues. Our key observation is that modern video backbones capture latent structure that can be decomposed to separate fine- aaee-granndd variations even without direct supervision. Building on this insight, we propose shot editing method to expose new subcategories without additional data. When limited supervision is available, simple low-shot fine-tuning proves highly effective, particularly when initialized from zero-shot edit. Experiments on our category splitting benchmarks SSv2-Split and FineGym-Split demonstrate substantial gains on newly split categories without sacrificing performance elsewhere, consistently outperforming VLM baselines.
In summary, we: (i) define the category splitting task, (ii) propose a zero-shot editing method, (iii) show that low-shot fine-tuning is effective and benefits from zero-shot initialization, (iv) introduce benchmarks and metrics for evaluation, and (v) analyze where category splitting succeeds and fails.

---

Published as a conference paper at ICLR 2026
Base Model
Prediction
Dropping something
Dropping something
Dropping something
Dropping something
Edit
Edited Model Prediction
Input videos
Dropping something into something
Dropping something behind something
Drogping something next to something
Dropping something onto something
Figure 1:
Category splitting aims to edit trained video classifier by dividing a coarse label into multiple fine-grained subcategories, while keeping all other predictions unchanged. The challenge to achieve this without retraining the full model and with zero or very few labels.
CATEGORY SPLITTING PROBLEM DEFINITION
We address the problem of category splitring as illustrated in Figure 1. The goal is to refine a chosen coarse category into finer- gner-grained subcategories, while preserving the performance others. Con- cretely, we start with classification model trained to predict a set of categories, some of which may be more fine-grained while others current model that needs to be split into several new fine-grained subcategories. A naive approach would be retrain the model from scratch with newly annotated data for these subcategories, gether with the original training set. However, this is costly, requiring time-intensive data collection and annotation as well as full training cycle. Our goal is enable category splitting efficiently, operating with little or no labeled data. This allows rapid model adaptation in resource-constraingg or time-sensitive scenarios and supports rare categories that would be absent from the training set.
coarse. We given coarse-grained category within the
Formally, let A' be the input space, y the label space, with the base classification model: fo :X y Given coarse category e E V. and its desired fine-grained subcategories =s s2...... updated label space removes the coarse category and adds the fine-grained subcategories:
........))) .%), the
= (3) US where S nyy=0
(1)
We seek an editing method E, that locally and efficiently edits the original model fa. yielding an updated model that supports fine-grained classification within the selected coarse category e:
E(Se) = Se,, where fe X < )
We desire two properties for the updated model for: generality and locality, adapted from model editing in NLP (Mitchell et al., 2022a). Generality means the model edits should extend beyond any given training samples and correctly classify unseen examples of the new subcategories S Locality means the edits should preserve predictions of all other existing model categories y {c).
3 ZERO-SHOT CATEGORY SPLITTING
Fine-grainer categories may lack annotated examples, as with events anomalies. We there- fore focus on zero-shot category splitting (Figure 2). Our key insight is that modern video backbones already encode rich latent features that capture compositional fine-grained variations within coarse classes, even without explicit compositional labels or text supervision. Exploiting this, we design a simple but effective approach that edits only the classification head, leaving the backbone un- changed. We view each fine fine-grained subcategory as coarse category combined with modifier that specifies a particular variation. This exposes a compositional structure within the classification head that we can exploit. Building on this, Section 3.1 introduces a retrieval-based method that first decomposes existing fine-grained categories into coarse concepts and modifiers, and then reuses suitable modifiers to split existing coarse categories into previously unseen fine ine-grainee categories. Section 3.2 then extends this idea with a lightweight alignment model that maps textual modifier descriptions into the classification head weight space, enabling generalization to modifiers that do not appear in the original label set.
3.1 ZERO-SHOT EDITING: MODIFIER RETRIEVAL
We assume that the classifier we are editing is mixed- -raedd-gaaularity video classifier, that is, has been trained predict set of categories, coarse and some fine. To split selected coarse category,
2

---

Published as a conference paper at ICLR 2026
Modifier Retrieval
Modifier Dictionary
Edited Classification Hoad
Original Classification Head
Vectors
label
Nodifirr Text
Similanty
Poking someching 23 it
NN sanoeeiing s (ari
"6WE3 16 W'EERRRY Woves
Pishing wortthhing
Puahing sommehhngg spin
something
16W0gg Sareainng 95' Highty moves
Pusting
somsthingg
Pualting
holding sometting noxt semething
Holding something behind something
behind something
next to sometr ing
movis
Pushing something left to right
Fine Grained Subcotagorics Ics
Modiler Text
Pusning samething
Target coarse category: Pushing
Pushing something
Rusding something
Modifier Alignment
Inference
Edited Clessification Head
Nasiler Voctors
(Pocude /Catepory Labe's
Weight
Pushing Morwthingg
Pushiing something
50 it fa
Fushing something
Training
Predictions
stinss
+
poking something so spins Pokkng somerhing
Pusting
behind something
K roqves
next somsthhnng
Dropsing somathing ing
Push something
Modifier pairs
Category pairs
right.
Figure 2: Zero-shot Category Splitting, Given a trained video classifier, our goal is to split a coarse category (e.g. pushing) into -neggriieed subcategories without any video data. Modifier retrieval first exposes compositional structure in the video classifier's classification head from which it builds dictionary of modifier vectors. The classifier is then edited by retrieving the appropriate modifier vector and adding it to the coarse category's 's weight vector to create new fine-grained subcategory. To generalize to unseen modifiers, modifier alignment learns lightweight mapping from modifier text to modifier vectors, using category text/weight vectors as additional supervision.
we build on the principle of compositionality: fine- -raiinr concepts can be expressed as coarse base concept combined with modifier that specifies particular variation. For example, pushing left to right can be viewed as the coarse action pushing composed with the directional modifier left to right. If the model already distinguishes between related classes such as throwing left to right and throwing right to left, the relevant modifier to split the pushing category is likely already encoded in the model. We exploit this by extracting modifier vectors from classifier weights, constructing modifier dictionary and transferring the appropriate the modifiers to the coarse category being split.
Modifier Dictionary Construction. We extract modifiers by grouping existing categories in the classifier's label space Y that share a base concept, forming a pseudo coarse category ė with fine- grained variants Se C y. For example the set S {poking so spins, poking so it falls, poking so it slightly moves} would give a pseudo coarse category č that corresponds to poking. Since the classifier already distinguishes between the fine-gra -ra-gga variants in S the differences between their weight vectors reflect the modifier directions encoded in the classification head Opead- To expose this structure, we construct explicit modifier vectors representing each fine- inn-grained category's 's difference from the pseudo coarse cateory. Let wy denote the classifier weight vector of a category y. We approximate the weight vector of the pseudo coarse category Va as the mean of the associated fine- grained classifier weight vectors wy:
1
V2
W'y
S
MES*
The modifier vector Um for fine- fine-grained class y S is then obtained by subtracting this shared base component:
(4) We store these modifier vectors a dictionary Mmodd each paired with a descriptive key. this work, use text labels as convenient way to identify both base concepts and modifiers, through other forms of metadata such as attributes could also be used. Given the text description ty of a category y D, identify the shared base description of its pseudo coarse category The modifier text is then f. = by - fa. describing the semantic difference (e.g. left to right). Each entry in the dictionary stores the triplet:
Um wy Ua-
Mmood = {Imm by: {m))
(5)
Modifier Vector Transfer. Our goal is to split coarse category into new fine-grained subcate- gories S = {sf. sq.... } without any video examples, by editing only the classifier head, Using modifier dictionary Mmod we construct new classifier weight vectors for each new subcate- gory. We assume that the modifier vectors in Mmod are sufficiently disentangled from their original coarse concept across categories. Under this assumption, for each subcategory 86 we retrieve the
3

---

Published as a conference paper at ICLR 2026
most appropriate modifier vector in Mmod that best distinguishes s from the coarse category to be split c. To do so, we first derive the modifier text in describing the difference between sÿ and and encode it with a text encoder c. Each modifier text im in the dictionary Mood is likewise embedded, and we retrieve most similarity entry using cosine similarity (sim(;, +)) to give us the modifier vector umm
arg max sim( ((f,))
(fm
However, the same modifier can have different visual effects depending on the coarse concept, e.g. pushing left to right is more visually similar to pulling left right than looking left right. To account for this, we extend retrieval to consider both the modifier and the coarse concept together. Given the text descriptions of the full label t and modifier tmm for the target subcategory sj. the corresponding modifier vector em is retrieved as:
arg max sim (m(t),, ((t))) + sim(
To add the new fine- fine-grained subcategories f" sf. the model, we replace the orig- inal coarse class weight We in the classification head Oread with additional weights head [uxt,...,,xs]] w.:l Each subcategory weight Way is formed by adding the retrieved modifier vector to the coarse category weight:
Weg = We + Um
(8)
This local edit reuses modifier knowledge already present in the model, enabling recognition of new fine-grained categories while leaving the rest of the model unchanged. The entire process is zero-shot, requiring no annotated videos and no backbone retraining.
3.2 ZERO-SHOT EDITING: MODIFIER ALIGNMENT
While modifier retrieval enables zero-shot category splitting, it is limited to modifiers that already appear in the trained model's label space. To handle modifiers outside of our dictionary Minod we introduce an alignment module that maps text embeddings directly into the classifier weight space. This mapping allows us to synthesize modifier vectors directly from text, enabling generalization to unseen modifiers. We define an alignment function Je RT that projects text embeddings from the encoder & to modifier vectors in the classifier weight space. Learning this mapping re- quires pairs linking modifier text fm to its corresponding vector Um. Crucially, video data is needed, keeping our method zero-shot, as our modifier dictionary Mmoo naturally provides such supervision. From it, we obtain modiffer-level pairs:
Dmodd = {(o(fm), vm) (fm, Um)
vm) E Emmoj..
However, modifier pairs alone provide too little supervision to reliably learn mapping from text to classifier vector space. To enrich the training signal, we also align the text embeddings of existing categories y E y and pscudo-coarse categories E Vpseudo with their corresponding weight vectors:
Deat = {()(ty). myy) y E y} ((o(ta), va) | é }pscudo}
(10)
Here by and ta are the textual descriptions of the category and pseudo-coarse category respectively and Wy and Us are the corresponding weight vectors (Eq. 3). We train the alignment module gg using mean squared error over the combined supervision:
LMSE
Σ
[g)(((()) - =012
(11)
(o(t).))ED (.o)))
During training, only the alignment parameters vb are updated; the classifier and text encoder remain frozen. At inference, given a modifier text th for a target subcategory s7. we generate its vector th (o((t), )) and extend the classification head with w.; + u This enables generalization to unseen modifiers while remaining entirely zero-shot and requiring no backbone updates.
LOW-SHOT CATEGORY SPLITTING
While our method enables zero-shot category splitting, in practice small number of labeled ex- amples may be available. We therefore study low-shot category splitting (Fig. 3), using only a few videos from new subcategories. We find fine-tuning surprisingly effective even with extremely limited data, and that performance improves further with initialization from our zero-shot method.
Isolated Finetuning. A challenge in category splitting is preserving performance on the original categories. To avoid disrupting unrelated classes, we fine-tune only the new subcategory weights, keeping the rest of the model frozen. Let the original video classifier be ʃo with parameters
4

---

Published as a conference paper at ICLR 2026
Df
Sc Pushing right left
(bbackbone: 0head), corresponding to the parameters of the backbonn and the classification head. Given a coarse category encoded in heead, suppose wish to split e into fine-grained subcategories [" }. We construct a small fine-tuning dataset (z,; M4 { where y4 S We focus the extreme low-shot case with N example per subcategory. To accommodate the expanded label space, we remove the coarse weight we from the classifier head thead and replace it with new set of subcategory weights 8% "head=(w....... Weg We initialize each new subcategory weight Way from we, since the subcategories are subtle variations of their parent. The resulting model is for, with parameters O' = {Onackhones Ohrad, head]- Fine-tuning is then ap- plied only to hhead using cross entropy loss.
Pushing right left Pushing off sornething
Pushing right to left
Pushing right loft
Pushing
Dropping
Pushing off something Bbackbone
Bhoad
Low-shot Category Split- ting. edit the model by replacing the coarse category e with he This head is fine-tuned with little as one video per fine-grained subcategory, initialized with our zero-shot approach.
Figure 3:
Zero-Shot Initialization. While isolated fine-tuning is a strong baseline for low-shot category splitting, it can be further improved with our zero-shot ap- proach. For each fine-grained category sf. we initialize the classifier weight w.z=w,typ where om either be obtained from our modifier retrieval (Sec. 3.1) modifier alignment (Sec. 3.2). This hybrid strategy combines model's latent fine-grained structure with available labeled samples, improving performance in the low-shot regime.
CATEGORY SPLITTING DATASETS
Since no benchmarks exist for category splitting, construct two from Comething-Somethiee V2 (SSV2) (Goyal et al., 2017) and FineGym288 (Shao et al., 2020). SSV2 is a large-scale action dataset with 220K videos spanning 174 fine- fine-graing seeegraaned categories. These categories capture subtle differences in object interactions and motion and can be naturally grouped into coarser semantic concepts. Fin- eGym288 is fine-grained gymnastics dataset with 31 K videos annotated with 288 action categories organized hierarchically, providing a natural structure for evaluating category splitting.
Constructing Coarse and Fine Labels. We group the original fine-grained categories into seman- tically coherent coarse categories. For SSV2 we do this manually, for FineGym288 we follow the dataset's action hierarchy. The resulting coarse categories can be divided into fine- ine-grained variants that differ in subtle aspects such as spatial relation, motion type, intensity, completion, antonyms, repetitions or body pose. complete list of groupings is provided in Appendix A.
Mixed Granularity Base Model. Our task requires base models trained under mixed supervision, with some categories coarse and others fine. This mirrors real-world annotation settings where label granularity is inconsistent and creates the conditions to evaluate category splitting. Coarse categories serve as targets to be split, while fine-grained categories expose the backbone to subtle distinctions elsewhere label space. To construct this setup, we partition coarse categories into two sets, A and B (see Appendix A) and define two complementary subsets. In subset A, categories in set A are collapsed into coarse labels, while those set retain their fine-grained labels. In subset B, the roles are reversed. This design ensures that every category is evaluated as a coarse label to be split. Each subset is used to train base model, following the original SSV2/FineCtmm288 train and validation partition, which is then used to evaluate category splitting.
Dataset Statistics. Table 1 summarizes our benchmarks SSv2-Split and FineGym-Split SSv2-Split contains 54 coarse categories and FineGym-Split 42, each split into 2-19 subcategories. SSv2-Split emphasizes everyday actions distinguished by spatiality, state changes and object interaction, while FineGym-Split targets specialized domain with differences in body pose, motion and repetitions.
6 EXPERIMENTS & RESULTS
6.1 IMPLEMENTATION DETAILS.
Our base model uses ViT-Small (Dosovitskiy et al., 2021) pretrained with MVD (Wang et al., 2023) on Kinetics-400 (Carreira & Zisserman, 2017). We follow MVD's fine-tuning recipe for SSV2, training on 4 NVIDIA A100 GPUs with the same hyperparameters, except for larger batch size
5

---

Published as a conference paper at ICLR 2026
Table 1: Statisties of Category Splitting Datasets. We create meta-datasets SSV2-Split and FineGym-SSlit,, each with two subsets (A and B). For each subset we report the number of cate- gories and videos used train the base model, the number of coarse categories eligible for splitting, the number of resulting fine-gras rreeggraiedd subcategories we can split into, the average number of videos used to evaluate generality and locality per category split as well as an example coarse-to-fine split.
Base Model
Category
Splitting
Evaluation
Example
Benchmark
Subset
categories
videos
coarse
fine
generality locality
SSv2-Split
A
119
169K
27
92
429
24347
bending-> Baalligg>(beennng (bending so deforms, heading until it breaks, trying bend something unbendable)
B
121
169K
27
94
403
24373
Sine
A
178
23K
23
155
173
7988
handspring forward-> hrewardd-(handaee (handspring forward 1.5
B
162
23K
19
143
204
7957
handspring forward 1 tum, hundspring forward no tur)
of 18 for efficiency. We fine-tuning FineGym with the same setting. Low-shot fine-tuning uses AdamW (Loshchilov & Hutter, 2019) with learning rate 1 1x10-3 weight decay 1x10-3 and batch size 16. The learning rate follows cosine annealing, and training runs for up to 100 epochs with early stopping based on an exponential moving average (EMA) of training or validation loss (3=0.95, pa- tience=5, 8=1 x 10-5). For zero-shot methods, we adopt CLIP ViT-L/14 text encoder (Radford et al., 2021). In modifier aligmment, we use an MLP with one 384d hidden layer, trained using AdamW with learning rate of 1x 11003, cosine annealing. and batch size 10, for up to 100 epochs. Early stopping is applied with EMA of cosine similarity between predicted and reference embed- dings. For more details refer to Appendix B.
6.2 EVALUATION METRICS.
We evaluate category splitting methods with two criteria: generality and locality. Generality mea- sures the accuracy of the edited classifier S'g on distinguishing subcategories that coarse category c has been split into:
1 E M
Generality
1 1[b(z,) = ysl, yj E
(12)
Locality measures how well the edited classifier f/ preserves performance on non-target categories. It is defined as the ratio between the accuracy of S/ and that of the original classifier So on the unchanged portion of the test set:
Σe ([b(s) yil
Locality-
y4 E VEVioo {c)
(13)
E 1 [Jo(a;) yi]
A locality of indicates that the model edit has no negative impact on other categories. Each target category is split in isolation and results are averaged. Metrics are multiplied by 100 for readability.
6.3 COMPARATIVE RESULTS
Since category splitting a new task, there are no existing methods. As a point of comparison, build baselines using vision-language models (VLMs) external modules, which provide natural way to leverage semantic information for fine-grained recognition. For each video the base model classifies the target coarse category, we predict fine-grained subcategory by measuring similarity between the video embedding and the text embeddings of candidate fine- fine-grained labels. gre-ggrided, and video-specific models: We evaluate a range of VLMs, spanning general-purpose,
CLIP (Radford et al., 2021), FLAIR (Xiao et al., 2025), FG-CLIP (Xie et al., 2025), VideoCLIP- XL (Wang et al., 2024a), VideoPrism (Zhao et al., 2024), and Intern Video2 (Wang al., 2024c).
The results on both SSv2-Split and FineGym-Split are summarized in Table 2. All VLMs achieve perfect locality by construction, since they are applied externally and do not modify model pa- rameters. However, their generality remains low. On SSv2-Split there is little advantage to using video-text models, with VideoCLIP-XI, VideoPrism and Intern Video2 all performing worse than CLIP and FG-CLIP on subset B. On FineGym-Split video-text models are somewhat stronger with VideoPrism achieving 21.7% generality on subset A compared to CLIP's 12. 1% and FG-CLIP's 19.4. In contrast, our method achieves substantially higher generality across both datasets and sub- sets. For example, it reaches 45.9% on SSy2-Split subset A and 34.2% on FineGym- A, while maintaining near-perfect locality. These results show that category splitting can be accom- plished directly in video-only classifiers, and that exploiting the latent structure of the classifier is more effective than relying VLM pretraining.
subset

---

Published as a conference paper at ICLR 2026
Table 2: Comparative Zero-Shot Results. Comparison with baseline methods on SSv2 and Fine-
Gym. Our approach has much better generality than vision-language models
SSv2-Split
Sppltt
Subset
Subset B
Subset A
Subsct
Method
Gen.
Loc.
Mean
Gen.
Loc.
Mean
Gen.
Loc.
Mcan
Gen.
Loc.
Mean
CLIP (Radford ct al., 2021)
27.6
100.0
63.8
30.7
100.0
65.4
12.1
100.0
56.1
7.2
100.0
53.6
FLAIR (Xiao al., 2025)
30.6
100.0
65.3
28.4
100.0
64.2
18.0
100.0
59.0
11.6
100.0
55.8
FG-CLIP (Xic ct al., 2025)
30.9
100.0
65.4
30.8
100.0
65.4
19.4
100.0
59.7
13.9
100.0
56.9
VideoCLIP-XX (Wang et al., 2024a)
28.6
100.0
64.3
29.9
100.0
64.9
18.0
100.0
59.0
8.2
100.0
54.1
VideoPrism (Zhao et al., 2024)
28.2
100.0
64.1
29.3
100.0
64.7
21.7
100.0
60.9
114
100.0
55.7
InternVidoo2 (Wang et al., 2024c)
25.9
100.0
62.9
21.8
100.0
60.9
17.4
100.0
58.7
10.8
100.0
55.4
Ours
46.3
98.9
72.6
38.4
98.9
68.7
34.2
97.8
66.0
18.9
97.9
58.4
Table 3: Zero-shot Ablation. Our mod- ifier retrieval and alignment greatly im- prove generality by mining modifier vec- tors from the video-only classifier.
Method
Generality
Locality
Mean
Vision Laiion- Model Modifier Retrieval
27.61000
100.0100
63.8 1..8
45.0+0.0
98.9
71.9
Modifier Alignment
46.3
98.9+00
7..6
Table 4: One-Shot Finetuning Ablation. Constraining updates to the extended head (8head) avoids catastrophic forgetting when training with little data, and initializing with Knowledge Retrieval provides a stronger starting point that boosts generality without sacrificing locality.
Components Finetuncd
Initialization
Generality
Locality
Mcan
Full Data Finetuning
Oreaks of head
coarse category
86.7 0.1
2
$4.4
8
coarse category
86.7
19.2
52.9
One-Shot Finetuning
Ohackiine:: Bread: head
coarse category
33.6+ 1.9
0.0+00
16.8+
Speas: Draat
coarse category
5..7
96.4
73.5
random
45.0
98.8 1..1
71.9111
6 &read
coarse category
48.4+22
98.4+0.
73.4+13
modifier alignment
52.8
98.2
75.5
6.4 ABLATION STUDY
We perform ablations on SSv2-Split subset A, averaging results over three runs.
Zero-Shot Ablation. Table 3 shows zero-shot category splitting under different variants of our approach. As a baseline, we directly apply the vision-language model that supplies the text-encoder for our method: videos the base model predicts as coarse category c are reassigned to the fine-grained subcategory with highest video-text similarity. This achieves perfect locality since the base classifier remains unchanged. However, its low generality (27.6%) shows that pretrained vision-language embeddings are insufficient for capturing fine- fine-grained grained distinctions. In contrast, our modifier retrieval substantially improves generality to 45.0% while maintaining high locality (98.9%), showing can effectively re-purpose distinctions already encoded in the classifier without any new data. Adding modifier alignment yields a further gain in generality (+1.3%), enabling better fitting and improved generalization to new modifiers. Together, these results demonstrate that zero-shot category splitting not only possible but surprisingly effective when exploiting the classifier's 's latent structure.
One-Shot Finetuning Ablation. Table 4 compares different fine-tuning strategies with limited training data. Updating the full model (Obackbone: Ohead; (heaa) in a one-shot setting achieves mod- erate generality (33.6%) but destroys locality (0.0) as shared parameters are overwritten. Restricting updates the head ((fread, 8% (/ead) or (8%)ea)) restores high locality (98.4) and boosts generality (48.4%), showing the need to isolate edits. Initializing the extended head with our zero-shot mod- ifier alignment boosts generality (+4.4% over coarse category initialization, +7.8% over random), while maintaining locality, further indicating the effectiveness of our zero-shot approach. Notably, our one-shot approach outperforms full-data finetuning (75.5 vs 54.4 mean) as full-data finetuning creates a strong bias the new classes, severely reducing locality.
Base Model Pretraining. Table 5 examines how category splitting varies across different model pretraining. Editing model trained from scratch produces the weakest results, underscoring the im- portance of strong prior representations. Using the visual encoder of CLIP (Radford et al., 2021) im- proves performance, indicating that image-text pretraining provides useful compositional structure.
7

---

Published as a conference paper at ICLR 2026
Table 5: Base Model Pretraining. Stronger pretraining helps, but our method is effective even on models trained from scratch.
Table 6: Text Encoders from video-text models provide little benefit over CLIP.
Text Encoder
Generality
Locality
Mean
Base Model
Generality
Locality
Mean
Text-only
From Scratch
1.
97.7
u..4
RoBERTa
40.9
99.2
70.0
CLIP (Visual)
38.2
98.010.1
1.0.
Image-Ter
VidcoMAE
42.9+07
98.5+
70.7+04
CI.IP
46.3++
98.9+ 0.0
72.6+0.5
MME
42.6
99.010.0
70.8
Video-Text
SIGMA
4..
09.0=a.0
71.6=u6
Intern Vidco2
36.9
99.5
68.2 +
MVD
46.3
98.910.
72.610.
VideoCLIP-XL
15
98.91
72.2 1.055
VideoPrism
46.5
98.8 1.00
72.7 2.1.1
(a) Number Subcategories
(b) Subcategory Type
(c) Analogies
100
100
100
Score
Score
Score
Count
Sp Spatiality
Direction
Success
State
Motion
Interaction
without
Generality
Locality
Figure 4: Analysis over different category splits. (a) Locality decrease slightly with more subcat- egories in the split, while generality shows no trend. (b) Performance is highest for direction-basee splits and lowest for differences in object count, intent/success, and object interactions. (c) Existing analogous categories with the same modifier help, but our approach remains effective without them.
However, video-only pretraining proves most effective: VideoMAA (Tong et al., 2022), MME (Sun al., 2023), SIGMA (Salehi et al., 2024) and MVD (Wang et al., 2023) all outperform CLIP. Among these models SIGMA and MVD achieve the strongest results, likely due to objectives that empha- size semantic structure over pixel-level reconstruction. Importantly, our method successfully edits all pretrained models, demonstrating robustness even when pretraining is limited.
Text Encoder. Table 6 compares different text encoders used in our zero-shot modifier alignment. Text-only model RoBERTa (Liu et al., 2019) achieves moderate generality (40,9%) but lags behind multimodal encoders. CLIP (Radford et al., 2021) improves generality (45.9%) and maintains lo- cality, showing the benefit of visual alignment. However, video-text models do not consistently outperform CLIP: Intern Video2 (Wang al., 2024c) performs poorly, while VideoCLIP-XC (Wang et al., 2024a) and VideoPrism (Zhao et al., 2024) achieve similar results. We thus conclude that mul- timodal alignment helps, but image-text models are sufficient for identifying relevant compositions.
6.5 ANALYSIS
We next examine where our model excels and where the room from improvement lies, with the aim of guiding future research in category splitting and fine-grained video understanding.
Does performance degrade with more subcategories? Figure 4a shows results across varying numbers of subcategories. We observe a slight decline locality as the number of subcategories in the split increases, but generality has no clear trend. This suggests that while editing becomes some- what less precise with more splits, the ability of the model to recognize new categories unaffeccted. Which subcategories are easier split into? Figure 4b shows performance across subcategory types. Our model performs particularly well on direction as well as spatial position, motion and state change differences. Object count, action success and object interactions the most difficult. Are subcategories with analogies easier? Figure 4c compares subcategories where the relevant modifier is present in existing category. As expected, performance higher when analogies available, confirming the compositional structure we discover in video classifiers. However, even without analogies, our method performs well, demonstrating its capacity for generalization.
How much compositional variation is needed in the original label space? To assess our method's dependence on the compositional variation in the original label space, Table 7 varies the proportion of coarse- coarse-grained categories used in the original label space before splitting. Our default setting used so far uses 50% of coarse categories. Increasing this to 66% and 75% reduces the number of fine-grained categories observed during training, thereby reducing the compositional structure available for modifier extraction. Despite this reduction generality decreases only slightly, while locality remains stable across all settings. Importantly, our method consistently outperforms CLIP regardless of how much compositional structure is present.
8

---

Published as a conference paper at ICLR 2026
Table 7: Effect of Compositional Variation in the Original Label Space. Our method remains robust when reducing the number of fine-grained categories in the original label space and consis- tently outperforms CLIP
Original Cat.
Cat. After Split
Ours
CLIP
Coarse Cat.
Course
Fine
Total
Total
Gen.
Loc.
Mean
Gen.
Loc.
Mean
50%
92
119
174
46.3
98.9
72.6
27.6
100.0
63.8
66%
36
67
103
174
44.2
99.1
71.7
29.0
100.0
64.5
75%
41
55
96
174
44.6
98.8
71.7
29.4
100.0
64.7
(c) Laying something on the fable on side, not upright in front of something
(a) Throwing in the nir from left to right
(b) Twisting until flips
Figure 5: Qualitative Results on SSv2-Split with unseen category + modifier combinations.
GT Spinning something
Something being deflected from somthing
Bending something until
Letting something roll down slanted surface
continues sprinning
breaks 59%
Acc.
Video
Pred Spinning something that it quickly stops spinning
Something colliding with something and both are deflected
Bending something until it breaks
Letting something roll down slanted surface
Figure 6: New Visual Distinctions. We show failure and success cases of our model when categories require new visual distinctions.
6.6 QUALITATIVE RESULTS.
Unseen Modifiers. Figure 5 shows qualitative results on SSv2-Split. Our modifier alignment can successful classify videos with unseen category-modifie pairs (a) and completely unseen modifiers (b). Moreover, modifiers can be applied to to fine-grained categories further subdivide them (c). New Visual Distinctions. Figure shows examples that require genuinely new visual distinctions. challenging,
edits operate only the classification head; the backbone must already encode the visual cues needed to support the new subcategories. For instance, concepts such continues deflected introduce visual distinctions absent from the original training setup, leading to failure cases. However, our method also succeeds on some edits that introduce previ- ously unseen concepts, such as breaks slanted surface. This suggests that certain fine-grained cues are implicitly captured by the backbone despite not being required by the original label space. These successes highlight the stronger compositional structure provided by more capable pretrained backbones such as MVD.
These cases
7 RELATED WORK
Model Editing. Model editing studies how to efficiently modify model behavior while preserving performance on untargeted inputs. Prior works follows three strategies: (i) adding external modules, such as retrievers or auxiliary networks (Mitchell et al., 2022h; Zheng et al., 2023); (ii) augmenting
the model with extra parameters, like adapters extra neurons (Yu al., 2024; Huang al., 2023); and (iii) intrinsic edits that directly update the model weights (Meng et al., 2022; Fang et al., 2025; Meng al., 2024; Mitchell al., 2022a). Most of this literature focuses on LLMs, but there is growing interest in vision, especially in image generation models (Bau et al., 2020; Arad et al., 2024: Orgad al., 2023). Closest to us are works that edit image classifiers (Santurkar al., 2021; Yang et al., 2024). Santurkar et al. (2021) modify prediction rules by mapping one existing model concept to another (e.g- "snow"+'roan et al. (2024) learn hypernetwork to locate editable parameters without prior concept knowledge. Unlike prior work, which modifiers decision boundaries for existing labels to correct errors, we edit to expand the label space, replacing coarse category with multiple fine-gra fine-grained ones.
using a single exemplar and augmentations, while Yang
Continual Learning. Continual or class-ineremental learning aims to learn new tasks and cate- gories without forgetting prior knowledge. Approaches fall into three groups: (i) rehearsal, which trains the model with a mixture of old and new data to maintain past performance (Rolnick et al., 2019; Aljundi al., 2019); (ii) regularization, which penalizes changes parameters important for

---

Published as a conference paper at ICLR 2026
old tasks (Kirkpatrick et al., 2017; Mitchell et al., 2018); and (iii) architectural, which expand or modify the network to assign capacity to new tasks (Mallya & Lazebnik, 2018; Aljundi et al., 2017). Few-shot variants address adding new classes from limited data (Tao et al., 2020; Zhao et al., 2023; Zhang et al., 2021a; Xiang et al., 2022). Unlike continual learning, which expands the label space with new categories, we split an existing category into finer subcategories. We address this in zero-shot setting with no data from either existing or new categories.
Fine-Grained Video Understanding. Fine-grained video understanding aims to recognize subtle distinctions between visually and semantically similar actions (Goyal et al., 2017; Shao et al., 2020; Damen et al., 2018). Beyond action recognition, other works identify fine ane-grained differences in terms of skill (Doughty et al., 2018; Pirsiavash et al., 2014; Parmar & Tran Morris, 2017), repeti- tions (Hu et al., 2022; Zhang et al., 2020), adverbs (Doughty et al., 2020; Moltisanti et al., 2023), action attributes (Zhang et al., 2021b), action differences (Nagarajan & Torresani, 2024; Burgess et al., 2025), temporal prepositions (Bagad et al., 2023), chirality (Bagad Zisserman, 2025; Price & Damen, 2019) or how and why actions happen (Perrett et al., 2025). Progress spans both video- only models (Tong et al., 2022; Sun et al., 2023; Wang et al., 2023) and video-text pretraining (Xu et al., 2021: Zhao et al., 2024; Wang et al., 2024c;a). Yet video-only classifiers assume fixed tax- onomies, and video-text models require massive paired corpora but still struggle with fine-graing grained distinctions (Doughty et al., 2024; Thoker et al., 2025). We instead study category splitting in video- only classifiers, refining coarse labels into finer ones with minimal supervision while preserving original categories.
Compositionality in Vision. Compositionalityy has long been studied in computer vision, where complex categories built from simpler parts or attributes. Early work focused on part-based ob- ject recognition (Felzenszwalb et al., 2008; Fischler & Elschlager, 1973), evolving into composi- tional generation (Zhao al., 2018; Tan et al., 2019) and interpretable representations (Böhle al., 2022: Stone et al., 2017). Compositional reasoning has also been applied to human-objec inter- action detection (Hou et al., 2020; Kato et al., 2018) and compositional zero-shot learning (Misra et al., 2017; Nagarajan & Grauman, 2018), where models must recognize unseen combinations of known primitives. Lcrge-scale vision-language models have renewed interest, supporting tasks like compositional retrieval (Baldrati et al., 2023; Hsich et al., 2023) and visual editing (Ceylan et al., 2023; Kawar et al., 2023), with evidence that compositional structure emerges in their representa- tions (Lewis et al., 2024; Trager et al., 2023; Berasi et al., 2025). Yet studies (Hsieh et al., 2023; Thrush et al., 2022; Tong et al., 2024; Yuksekgonul et al., 2023) show persistent weaknesses in ob- ject-attribute binding, spatial relationships, and other compositional inputs. While most prior work emphasizes compositionality in vision-language spaces, we focus on video-only classifiers and study how we can leverage compositional structure in such models in category splitting.
8 DISCUSSION
Summary. We introduced the problem of category splitting: refining coarse categories into fine- grained subcategories with lightweight model edits. We showed that video classifiers already encode compositional knowledge that supports zero-shot editing of the classifier to create new categories without additional data. In the low-shot setting, fine-tuning only the extended head is effective with as little as one video per category, and performance improves further with our zero-shot ini- tialization. On our new category splitting benchmarks, SSv2-Split and FineGym-Split our method consistently outperforms VLM basclines, achieving higher generality while preserving locality.
Future Work. We show that video classifiers implicitly encode compositional structure: actions decompose into base concepts and modifiers that can be isolated and repurposed. Beyond category splitting, this compositionality enables interpretability, continual learning via incremental modifiers, and reasoning across contexts. Within category splitting, several avenues remain open. While re- stricting edits to the classification head preserves stability, it limits flexibility. Exploring edits deeper in the model could unlock richer adaptations, though maintaining locality will be challenging. An- other step is increasing expressivity beyond single text-based modifiers to multiple modifiers, hier- archical taxonomies or alternative representations. The geometry of model also warrants study, as hyperbolic or spherical classifiers may better capture compositionality. Finally, extending category splitting to images, audio, or multimodal recognition will test its generality and broaden its impact.
Acknowledgements. This work is supported by the Dutch Research Council (NWO) under a Veni grant (VI. Veni. Veni.222.260 260.1
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Dana Arad, Hadas Orgad, and Yonatan Belinkov. Refact: Updating text-to-image models by editing the text encoder. In Proceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), 2024.
Piyush Bagad and Andrew Zisserman. Chirality action: Time-aware video representation learning by latent straightening. arXiv preprint (5XXi:2500908502, 2025.
Piyush Bagad, Makarand Tapaswi, and Cees GM Snoek. Test of time: Instilling video-language models with sense of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Alberto Baldrati, Lorenzo Agnolucei, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
David Bau, Steven Liu, Tongzhou Wang, Lun-Yan Zhu, and Antonio Torralba. Rewriting a deep generative model. In European Conference on Computer Vision (ECCV), 2020.
Davide Berasi, Matteo Farina, Massimiliano Mancini, Elisa Ricci, and Nicola Strisciuglio. Not only text: Exploring compositionality of visual representations in vision-language models. In Proceedings of the IEEF/CVF Computer Vision and Puttern Recognition Conference (CVPR), 2025.
Moritz Böhle, Mario Fritz, and Bernt Schiele. B-cos networks: Alignment is all we need for in- terpretability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
James Burgess, Xiaohan Wang, Yuhui Zhang, Anita Rau, Alejandro Lozano, Lisa Dunlap, Trevor Darrell, and Serena Yeung-Levy. Video action differencing. In International Conference on Learning Representations (ICLR), 2025.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European Conference Computer Vision (ECCV), 2018
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations (ICLR), 2021.
Hazel Doughty, Dima Damen, and Walterio Muyol-Cuevas. Who's better? who's best? pairwise deep ranking for skill determination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
11

---

Published as a conference paper at ICLR 2026
Hazel Doughty, Ivan Laptev, Walterio from adverbs in instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
and Dima Damen. Action modifiers: Learning
Hazel Doughty, Fida Mohammad Thoker, and Cees GM Snoek. Locomotion: Learning motion- focused video-language representations. In Proceedings of the Asian Conference Computer Vision (ACCV), 2024.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma. Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. In International Conference Learning Representations (ICLR), 2025.
Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A discriminatively trained, multiscale, deformable part model. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2008.
Martin A Fischler and Robert A Elschlager. The representation and matching of pictorial structures. IEEE Transactions on Computers, 1973.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Hcuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Muuellr-FFreitgg et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Zhi Hou, Xiaojiang Peng. Yu Qiao, and Dacheng Tao. Visual compositional learning for human- object interaction detection. In European Conference on Computer Vision (ECCV), 2020.
Cheng- Yu Hsich, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision- languala compositionality. Advances in Neural Infor- mation Processing Systems (NeurIPS), 2023.
Huazhang Hu, Sixun Dong, Yiqun Zhao, Dongze Lian, Zhengxin Li, and Shenghua Gao. Transrac: Encoding multi-scale temporal correlation with transformers for repetitive action counting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong- Transformer-patcher One mistake worth neuron. In International Conference on Learning Representations (ICLR), 2023.
Keizo Kato, Yin Li, and Abhinav Gupta. Compositional learning for human object interaction. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Martha Lewis, Nihal Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H Bach, and Ellie Pavlick. Does clip bind concepts? probing compositionality in large image models. In Findings of the European Chapter of the Association for Computational Linguistics (EACL), 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
12

---

Published as a conference paper at ICLR 2026
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Kevin Meng, Amab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. In Findings of the Association for Compulational Linguistics (ACL), 2024.
Ishan Mísra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher Manning. Fast model editing scale. In International Conference Learning Representations (ICLR), 2022a.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based model editing scale. In International Conference on Machine Learning (ICML), 2022b.
Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, al. Never-ending learning. Com- munications of the ACM, 2018.
Davide Moltisanti, Frank Keller, Hakan Bilen, and Laura Sevilla-Lara. Learning action changes by measuring verb-adverb textual relationships. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Tushar Nagarajan and Kristen Grauman. Attributes as operators: factorizing unseen attribute-object compositions. In Proceedings of the European Conference on Computer Vision (ECCV), 2018
Tushar Nagarajan and Lorenzo Torresani. Step differences in instructional video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Paritosh Parmar and Brendan Tran Morris. Learning to score olympic events. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops (CVPRW), 2017.
Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Kumar Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A highly-detailed egocentric video dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Hamed Pirsiavash, Carl Vondrick, and Antonio Torralba. Assessing the quality of actions. In Euro- pean Conference Computer Vision (ECCV), 2014.
Will Price and Dima Damen. Retro-actions: Learning' close'by time-reversing oponn videos. Pro- ceedings of the IEEE/CVF International Conference Computer Vision Workshops (ICCVW), 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning (ICML), 2021.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap. and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
13

---

Published as a conference paper at ICLR 2026
Mohammadreza Salehi, Michael Dorkenwald, Fida Mohammad Thoker, Efstratios Gavves, Cees GM Snoek, and Yuki Asano. Sigma: Sinkhorn gonnndddddddd masked video modeling. In European Conference Computer Vision (ECCV), 2024.
Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Alek- sander Madry. Editing classifier by rewriting its prediction rules. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2021.
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: hierarchical video dataset for fine- grained action understanding, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Austin Stone, Huayan Wang, Michael Stark, Yi Liu, D Scott Phoenix, and Dileep George. Teaching compositionality to cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Xinyu Sun, Peihao Chen, Liangwei Chen, Changhao Li, Thomas H Li, Mingkui Tan, and Chuang Gan. Masked motion encoding for self-supervised video representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR). 2023.
Fuwen Tan, Song Feng, and Vicente Ordonez. Text2scene: Generating compositional scenes from textual descriptions. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few- shot class- -nasemnnreemetal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Fida Mohammad Thoker, Letian Jiang, Chen Zhao, Piyush Bagad, Hazel Doughty, Bernard Ghanem, and Cees GM Snoek. Severe++: Evaluating benchmark sensitivity in generalization of video representation learning. arXiv preprint arXiv:2504.05706, 2025.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2022.
Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for sepe-supe video pre-training. Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder Bhatia, and Ste- fano Soatto. Linear spaces of meanings: compositional structures in vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Jaapeng Wang. Chengyu Wang, Kunzhe Huang, Jun Huang, and Lianwen Jin. Videoclip-xl: Advanc- ing long description understanding for video clip models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024a.
Liyua Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362-538) 2024b.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self- supervised video representation learning. Proceedings of the IEEE/CVF Conference on Com- pufer Vision and Patfern Recognition (CVPR), 2023.
14

---

Published as a conference paper at ICLR 2026
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He. Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvide02: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision (ECCV), 2024c.
Xiang Xiang, Yuwen Tan, Qian Wan, Jing Ma. Alan Yuille, and Gregory D Hager. Coarse-to-fine incremental few-shot learning. In European Conference on Computer Vision (ECCV), 2022.
Rui Xiao, Sanghwan Kim, Mariana-luliana Georgescu, Zeynep Akata, and Stephan Alaniz. Flair: VIm with fine-grained language-infone informed image representations. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24884-24899 2025.
Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. Fg-clip: Fine-grained visual and textual alignment. arXiv preprint arXiv:2505.0507,, 2025.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Conference on Empirical Methods in Natural Language Process- ing (EMNLP), 2021.
Yunqiao Yang. Long-Kal Kai Huang, Shengzhuang Chen, Kede Ma, and Ying Wei. Learning where to edit vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron-indexed dynamic lora. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of- of-dsrss, and what to about it? International Conference on Learning Representations (ICLR), 2023.
Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021a.
Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Temporal query networks for fine-grained video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2021b.
Huaidong Zhang, Xuemiao Xu, Guoqiang Han, and Shengfeng He. Context-aware and seale- insensitive temporal repetition counting. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR), 2020.
Bo Zhao, Bo Chang, Zequn Jie, and Leonid Sigal. Modular generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Lingln Zhao, Jing Lu, Yunlu Xu, Zhanzhan Cheng, Dashan Guo, Yi Niu, and Xiangzhong Fang. Few-shot class-incremental learning via class-aware bilateral distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. In International Conference on Machine Learning (ICML), 2024.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang- Can we edit factual knowledge by in-context learning? In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
15

---

Published as a conference paper at ICLR 2026
A CATEGORY SPLITTING BENCHMARK DETAILS
A.1 COARSE CATEGORY CONSTRUCTION
We provide the complete list of constructed coarse categories along with the fine- fine-grained categories within each. Table 11 and Table 12 show our constructed category hierarchy for SSv2 and Fine- Gym288, respectively. Across both splits, it includes 174 fine-grained SSv2 categories, with 162 of them grouped into 54 coarse categories, while the rest remain ungrouped. For FineGym2888, since the original data annotation has some repeated text label, for avoiding ambiguity, we remove 3 fine- grained categories which has the same text label with other fine-grained categories, and then operate grouping on the remaining 285 categories, with 271 of them grouped into 42 coarse categories, while the rest remain ungrouped.
A.2 BENCHMARK SPLIT
For each benchmark split, we list the coarse categories merged in the original SSv2 dataset (Table 13, Table 14) and in FineGym (Table 15, Table 16).
B EXPERIMENT DETAILS
BENCHMARK IMPLEMENTATION DETAILS
For benchmarking, we used the official implementations of video-language models with default hyperparameters provided in the respective repositories. Specifically, for CLIP used the ViT- L/14 model via the clip Python package; for VideoCLIP-XL we used the official implementation provided by the authors; for VideoPrism Intern Video2 we used the Intern Video2-clip-68 implementation. CLIP and VideoCLIP XL experi- ments were on single A100 GPU, while VideoPrism and InternVideo2, experiments were on a single H100 GPU.
used the piroprrssmlv__publii..... large model; and for
B.2 ADDITIONAL EXPERIMENTS
We present additional ablation studies on video backbone size, an analysis of text quality robustness for the zero-shot modifier alignment method, and an ablation the modifier alignment method using different training pair compositions.
Backbone Size. Table 8 evaluates the effect of backbone capacity zero-shot category splitting. Results are consistent across ViT-Small, ViT-Based and ViT-Large with mean performance varying only slightly (71.3-72.4). This suggest that small models learn representations rich enough to support splitting categories into fine-grained subcategories, and additional capacity offers little benefit for this task.
Table 8: Backbone Size Ablation. Results are consistent across different backbone sizes.
Method
Generality
Locality
Mean
ViT-Small
98.9
72.6105
ViT-Base
43.6
99.1 +00
71.3
Vir-Largee
45.0
99.3+
72.1 +04
Robustness of Text Encoder. We assess the robustness of the text encoder used by our method in Table We selected one coarse category bending something and introduced three types pertur- bation to the modifier texts of its fine-grained categories: typos, multi-sentence descriptions, and multilingual labels (German and Chinese). Our method reasonably robust to typos, while multi- sentence definitions introduce some performance drop, likely due to noise introduced from longer, more complex text. Multilingual labels cause the largest drop as CLIP's text encoder optimized for English. Importantly, framework is modular. The text encoder can be replaced with stronger or multilingual alternatives, allowing the approach to better support non- non-English labels and further broaden its applicability.
16

---

Published as a conference paper at ICLR 2026
Table Robustness Text Encoder. Our method is reasonably robust typos but struggles with multi-sentence and mulli--lnguua text.
Text Perturbation
Generality
Locality
Mean
Clean
51.3
99.4
75.4
Typos
48.9
99.4
74.2
Multi-Sentence
41.1
99.6
70.4
Multi- Mungi-Lingual (DE ZH avg)
27.3
99.2
63.2
Ablation Training Pair Composition. Table 10 evaluates the effect of different training pair compositions on zero-shot modifier alignment. We compare training pair compositions with only modifier pairs (Dmod) to those that also include existing-category and pseudo-coarse category pairs (Dmod U Deat). The results consistently show that incorporating more diverse training pairs leads to improved performance, indicating that richer pair supervision helps the model better align modifiers and enhances zero-shot generalization.
Table 10: Ablation Training Pair Composition. The results consistently show that enriching the training signal by incorporating more diverse pairs leads to improved performance.
SSv2-Split
SineGyy--plii
Subset A
Subset
Subset A
Subset B
Training Pairs
Gen.
Loc.
Mean
Gen.
Loc.
Mean
Gen.
Loc.
Meun
Gen.
Loc.
Meun
Dmnnt
44.9
99.3
72.1
40.2
99.2
69.7
27.6
98.5
63.1
14.0
98.5
56.2
Dinod Doat
98.9
72.6
38.4
68.7
34.2
97.8
18.9
58.4
B.3 ANALYSIS SETTING
We provide details on how we group different subcategory for analysis. Table 18 is the grouping for different subcategory types and Table 17 the grouping for whether a subcategory has analogies.
C STATEMENTS
C.I REPRODUCIBILITY STATEMENT
The source code and instructions for running all experiments will be publicly released upon paper acceptance. All experimental details necessary for reproducing the results are included in the main paper and appendix.
C.2 THE USE OF LARGE LANGUAGE MODELS
We utilised large language model (LLM) to assist with language polishing and minor grammatical corrections, as well minor refinements to grouping of sub-category types in Figure 4. All scientific content, experimental design, and conclusions were solely developed by the authors.
Table 11: Constructed category hierarchy for SSv2.
Coarse categories Moving the camera and Turning the camera
Fine-grained categories
Approaching something with your camera Moving away from something with your camera Turning the camera downwards while filming something
Turning the camera left while filming something Turning the camera right while filming something Turning the camera upwards while filming some- thing
Attaching something something and Trying but failing to attach something to something because it doesn't stick
Attaching something something
Continued on next page
17

---

Published as a conference paper at ICLR 2026
Table 1I: Constructed category hierarchy for SSv2.
Coarse categories
Fine-grained categories
Trying but failing attach something to some-
thing because it doesn't 't stick
Bending something
Bending something so that deforms
Bending something until it breaks
Trying to bend something unbendable so nothing happens
Burying something in something and Digging something out of something
Burying something in something
Digging something out of something
Closing something and Opening something
Closing something
Opening something
Covering something with something and Uncovering something
Covering something with something
Uncovering something
Dropping something with spatial relation
Dropping something behind something
Dropping something in front of something
Dropping something into something Dropping something next to something Dropping something onto something Failing to put something into something because
Failing to put something into something because something does not fit Folding something and Unfolding something
something does not fit Folding something
Unfolding something
Hitting something with something Holding something
Hitting something with something
Holding something
Holding something with spatial relation
Holding something behind something
Holding something in front of something
Holding something next something
Holding something over something
Putting something on the table
Laying something on the table on its side, not up- right
Putting something similar to other things that are already the table
Putting something that cannot actually stand up- right upright on the table, falls on its side Putting something upright on the table Letting something roll along a flat surface Letting something roll down slanted surface Letting something roll up a slanted surface, so rolls back down
Letting something roll
Lifting a surface with something on it
Lifting surface with something on it but not enough for it to slide down
Lifting surface with something on it until it starts sliding down
Lifting something up completely
Lifting something up completely without letting it drop down
Lifting something up completely, then letting it drop down
Lifting something with something on it Lifting up end of something
Lifting something with something on it Lifting up end of something without letting drop down
Lifting up end of something, then letting it drop down
Moving part of something
Moving part something
Continued next page
18

---

Published as a conference paper at ICLR 2026
Table 1I: Constructed category hierarchy for SSv2.
Coarse categories
Fine-grained categories
Moving something across a surface
Moving something across surface until it falls down
Moving something across surface without falling down
Moving something and something
Moving something and something away from each other
Moving something and something closer to each other
Moving something and something so they collide with each other
Moving something and something so they pass each other
Moving something
Moving something away from something Moving something away from the camera
Moving something closer something
Moving something down
Moving something towards the camera Moving something up
Picking something up and Pretending to pick something up
Picking something up
Pretending to pick something up Piling something up
Piling something up and Stacking number of something
Stacking number of something Plugging something into something
Plugging something into something and Plugging something into something but pulling it right out as you remove your hand
Plugging something into something but pulling right out you remove your hand
Poking a hole
Poking a hole into some substance Poking hole into something soft Poking a stack of something so the stack collapses
Poking a stack of something
Poking a stack of something without the stack col-
lapsing
Poking something
Poking something so it slightly moves
Poking something lightly that it doesn't 't al- most doesn't 't move
Poking something that it falls
Poking something so that it spins around Pouring something into something
Pouring something
Pouring something into something until it over- flows
Pouring something onto something Pouring something out of something
Pretending to pour something out of something, but something is empty
Trying to pour something into something. but missing so it spills next to
Pretending or failing to wipe something off of something and Wiping something off of something
Pretending or failing to wipe something off of something
Wiping something off of something Pretending or trying and failing twist something
Twisting something, Twisting (wringing) something wet until water comes out and Pretending or trying and failing to twist something
Continued next page
19

---

Published as a conference paper at ICLR 2026
Table 11: Constructed category hierarchy for SSv2.
Coarse categories
Fine-grained categories
Twisting (wringing) something wet until water comes out
Twisting something
Tearing something
Pretending to be tearing something that is not tear-
able
Tearing something into two pieces
Tearing something just a little bit Pretending to close something without actually closing it
Pretending close something without actually closing it and Pretending to open something without actually opening it
Pretending to open something without actually opening it
Pretending to poke something Pretending to put something with spatial relation
Pretending to poke something
Pretending to put something behind something Pretending to put something into something Pretending to put something next to something Pretending to put something on surface Pretending to put something onto something Pretending to put something underneath some- thing Pretending scoop something up with something
Pretending scoop something up with something and Scooping something up with something
Scooping something up with something Pretending spread air onto something
Pretending spread air onto something and Spreading something onto something
Spreading something onto something Pretending to sprinkle air onto something
Pretending to sprinkle air onto something and Sprinkling something onto something
Sprinkling something onto something Pretending squeeze something
Pretending to squeeze something and Squeezing something
Squeezing something
Pretending to take something from somewhere and Taking something from somewhere
Pretending to take something from somewhere
Taking something from somewhere
Pretending to take something out of something and Taking something out of something
Pretending to take something out of something
Taking something out something Pretending throw something
Pretending throw something and Throwing something
Throwing something
Pretending to turn something upside down and Turning something upside down
Pretending to turn something upside down
Turning something upside down Pulling something from behind of something
Pulling something
Pulling something from left to right
Pulling something from right to left
Pulling something onto something
Pulling something out of something
Pulling two ends of something
Pulling two ends of something but nothing hap- pens
Pulling two ends of something so that it gets stretched
Pulling two ends of something so that it separates into two pieces
Continued on next page
20

---

Published as a conference paper at ICLR 2026
Table 11: Constructed category hierarchy for SSv2.
Coarse categories
Fine-grained categories
Pushing something
Pushing something from left right
Pushing something from right to left
Pushing something off of something
Pushing something onto something
Pushing something so spins
Pushing something so that it almost falls off but
doesn't
Pushing something so that it falls off the table
Pushing something so that it slightly moves Pushing something with something
Putting multiple things
Putting number of something onto something Putting something and something on the table Putting something, something and something on the table
Putting something with spatial relation
Putting something behind something Putting something in front of something Putting something into something Putting something next to something Putting something something
Putting something underneath something Putting something on flat surface without letting it roll
Putting something on surface or onto something
Putting something on a surface Putting something the edge of something it is not supported and falls down Putting something onto slanted surface but it doesn't 't glide down
Putting something onto something else that cannot support it so it falls down
Putting something that can't 't roll onto a slanted surface
Putting something that can't roll onto a slanted surface, it slides down
Putting something that can't roll onto a slanted surface, it stays where it is
Removing something, revealing something
Removing something, revealing something behind
behind
Rolling something on flat surface
Rolling something on flat surface Showing a photo of something to the camera
Showing something's properties
Showing that something is empty
Showing that something is inside something
Showing something with spatial relation
Showing something behind something
Showing something next to something
Showing something on top of something
Showing something to the camera Something colliding with something
Showing something to the camera
Something being deflected from something Something colliding with something and both are being deflected
Something colliding with something and both come to a halt
Something falling
Something falling like feather or paper Something falling like rock
Spilling something with spatial relation
Spilling something behind something Spilling something next to something Spilling something onto something Spinning something so continues spinning Spinning something that quickly stops spinning Continued on next page
Spinning something
21

---

Published as a conference paper at ICLR 2026
Table 11: Constructed category hierarchy for SSv2.
Coarse categories
Fine-grained categories
Stufting something into something
Stufting something into something
Taking one of many similar things on the table
Taking one of many similar things on the table
Throwing something toward something
Throwing something against something
Throwing something onto surface
Throwing something in the air
Throwing something in the air and catching it
Throwing something in the air and letting fall Tilting something with something on slightly so it doesn't 't fall down
Tilting something with something on it
Tilting something with something on it until it falls off
Tipping something over and Tipping something with something in it over, so something in it falls out
Tipping something over
Tipping something with something in it over, so something in it falls out Touching (without moving) part of something
Touching (without moving) part of something
Table 12: Constructed category hierarchy for FineGym.
Coarse categories
Fine-grained categories
(VT) round-off, flic-flac with 0.5 turn on, salto forward
(VT) round-off, flic-flae with 0.5 turn on, stretched salto forward with 1.5 turn off
(VT) round-off, flic-flae with 0.5 turn on,
stretched salto forward with 0.5 turn off
(VT) round-off, flic-flac with 0.5 turn on, stretched salto forward with I turn off (VT) round-off, flic-flac with 0.5 turn on, stretched salto forward with 2 turn off (VT) round-off, flic-flae with 0.5 turn on, piked salto forward with 0.5 turn off
(VT) round-off, flic-flac with 0.5 turn on, piked salto forward off
(VT) round-off, flic-flac with 0.5 turn on, tucked
salto forward with 0.5 turn off
(VT) round-off, flic-flac with 0.5 turn 0.5 turn to piked salto backward off (VT) round-off, flic-flac with turn on, salto backward
(VT) round-off, flic-flac with 0.5 turn on, 0.5 turn to piked salto backward off
(VT) round-off, flic-flac with I turn on, stretched
salto backward with I turn off
(VT) round-off, flic-flac with turn on, piked salto backward off
(VT) round-off, flie-flac on, salto backward
(VT) round-off, flic-flac on, stretched salto back- ward with 2 turn off
(VT) round-off, flic-flac on, stretched salto back- ward with I turn off (VT) round-off, flic-flac on, stretched salto back- ward with 1.5 turn off (VT) round-off, flic-flac on, stretched salto back- ward with 0.5 turn off (VT) round-off, flic-flac on, stretched salto back- ward with 2.5 turn off
(VT) round-off, flic-flac on, stretched salto back- ward off
(VT) round-off, flic-flac on, piked salto backward off
Continued on next page
22

---

Published as a conference paper at ICLR 2026
Table 12: Constructed category hierarchy for FineGym.
Coarse categories
Fine-grained categories
(VT) round-off, flic-flac on, tucked salto back-
ward off
(VT) tsukahara
(VT) tsukahara stretched with turn
(VT) tsukahara stretched with turn
(VT) tsukahara stretched with 1.5 turn
(VT) tsukahara stretched with 0.5 turn
(VT) tsukahara stretched salto
(VT) tsukahara stretched without salto
(VT) tsukahara piked
(VT) tsukahara tucked with turn
(VT) tsukahara tucked
(VT) handspring forward on, salto forward
(VT) handspring forward on, stretched salto for-
ward with 1.5 turn off
(VT) handspring forward on, stretched salto for-
ward with 0.5 turn off
(VT) handspring forward on, stretched salto for- ward with I turn off
(VT) handspring forward on, piked salto forward with 0.5 turn off
(VT) handspring forward on, piked salto forward with turn off
(VT) handspring forward on, piked salto forward
off
(VT) handspring forward on, tucked salto forward
with 0.5 turn off
(VT) handspring forward on, tucked salto forward with turn off
(VT) handspring forward on, tucked double salto
forward off
(VT) handspring forward on, tucked salto forward
off
(VT) handspring forward on
(VT) handspring forward on, 1.5 turn off
(VT) handspring forward on, 1 turn off
(VT) handspring forward on, no turn off
(FX) leap with turn
(FX) switch leap with 0.5 turn
(FX) switch leap with 1 turn
(FX) split leap with 0.5 turn
(XX) split leap with turn
(X) split leap with 1.5 turn more
(FX) johnson with additional 0.5 turn
(FX) switch leap to ring position with turn
(FX split leap with I turn or more to ring position
FX) cat lepp with turn
(FX) leap
(X) switch leap (leap forward with leg change
cross split)
(FX) leap forward
(FX) johnson (switch lapp with 0.25 turn to side
split or straddle pike position)
(FX) switch leap to ring position
(FX) split ring leap
(FX) stride leap forward with change of legs to
wolf position
(FX) cat leap (leap with alternate leg change)
(FX) jump or hop with turn
FX) juli jump with turn
(FX) split jump with 0.5 turn
Continued on next page
23

---

Published as a conference paper at ICLR 2026
Table 12: Constructed category hierarchy for FineGym.
Coarse categories
Fine-grained categories
(FX) jump with 1.5 turn
(FX) straddle pike or side split jump with turn
(FX) straddle pike or side split jump with 0.5 turn
(FX) split jump with I turn or more to ring posi-
tion
(FX) stag. jump with 0.5 turn
(FX) tuck hop or jump with turn
(PX) tuck hop or jump with turn
(FX) stretched hop or jump with turn
(FX) pike jump with turn
(FX) wolf hop or jump with I turn
(FX) hop with 0.5 turn free leg extended above horizontal throughout
(FX) hop with turn free leg extended above hor-
izontal throughout
(FX) jump or hop
(FX) sissone jump (leg separation 180 degree on the diagonal to the floor, take off two feet, land on one foot)
(FX) split jump (leg separation 180 degree parallel to the floor)
(FX) straddle pike jump side split jump
(FX) stag ring jump
(FX) ring jump (rear foot head height, body
arched and head dropped backward)
(FX) stag jump
(FX) sheep jump jump with upper back arch and
head release with feet almost touching head (FX) wolf hop or jump hop jump with one leg bent and the other extended straight forward above horizontal with knees together (FX) jump with legs separated, landing in front lying support
(FX) butterfly forward, torso parallel to floor, slightly arched, legs straddled and feet above hip height during flight
(FX) tr
(FX) illusion turn through standing split (FX) 1 turn with free leg held upward in 180 split position throughout turn
(FX) turn in tuck stand tional
leg, free leg op-
(FX) turn in back attitude, knee of free leg horizontal throughout turn
(FX) turn one leg, free leg optional below horizontal
(FX) turn with heel of free leg forward at hori- zontal throughout turn
(FX) 3 turn
(FX) 3 turn with free leg held upward in 180 split position throughout turn
(FX) 3 turn in tuck stand on one leg, free leg straight throughout turn
(FX) turn on one leg, free leg optional below horizontal
(FX) 2 turn or more
(FX) 2 turn with free leg held upward in 180 split position throughout turn
(FX) 2 turn in tuck stand on one leg, free leg
straight throughout turn
Continued next page
24

---

Published as a conference paper at ICLR 2026
Table 12: Constructed category hierarchy for FineGym.
Coarse categories
Fine-grained categories
(FX) turn in back attitude, knee of free leg at horizontal throughout turn
(FX) turn on one leg, free leg optional below horizontal
(FX) turn or more with heel of free leg forward at horizontal throughout turn
(FX) spin or less on back in kip position, hip and leg closed
(FX) 1 spin or less on back in kip position, hip and
leg are closed
(FX) 4 turn on one leg, free leg optional below horizontal
(FX) 4 turn on one leg, free leg optional below horizontal
(FX) take-off forward from or both legs, salto
(FX) take-off forward from or both legs, salto
sideward tucked and (FX) acrial cartwheel
sideward tucked
(FX) aerial cartwheel
(FX) arabian double salto
(FX) arabian double salto tucked
(FX) arabian double salto piked
(FX) salto forward
(FX) double salto forward tucked with 0.5 twist
(FX) double salto forward tucked
(FX) salto forward tucked
(FX) double salto forward piked
(FX) salto forward piked
FX) salto forward stretched with 2 twist
FX) salto forward stretched with 1 twist
FX) salto forward stretched with 1.5 twist
FX) salto forward stretched with 0.5 twist
(FX) salto forward stretched, feet land succes-
sively
(FX) salto forward stretched, feet land together
(FX) aerial walkover forward
(FX) aerial walkover forward
(FX) salto backward
FX) double salto backward stretched with twist
FX) double salto backward stretched with twist
(FX) double salto backward stretched with 0.5
twist
(FX) double salto backward stretched
(FX) salto backward stretched with 3 twist
(FX) salto backward stretched with 2 twist
(FX) salto backward stretched with I twist
(FX) salto backward stretched
(FX) salto backward stretched with 3.5 twist
FXX salto backward stretched with 2.5 twist
FX) salto backward stretched with 1.5 twist
FX) salto backward stretched with 0.5 twist
FX) double salto backward tucked with 2 twist
FX) double salto backward tucked with I twist
FX) double salto backward tucked
FX) salto backward tucked
FXX) double salto backward piked with twist
FX) double salto backward piked
FXX whip salto backward
(BB) ring and arch jump
(BB) yang-bo (jump to cross over split with body
arched and head dropped backward)
(BB) stag-ring jump
(BB) ring jump (rear foot at head height, body
arched and head dropped backward, 180 separa- tion of legs)
(BB) split ring jump (ring jump with front leg hor- izontal to the floor)
Continued on next page
25

---

Published as a conference paper at ICLR 2026
Table 12: Constructed category hierarchy for FineGymm
Coarse categories
Fine-grained categories
(BB) split ring leap
(BB) sheep jump (jump with upper back arch and
head release with feet head height/closed Ring)
(BB) split and straddle jump
(BB) sissone (leg separation 180 degree on the di- agonal to the floor, take off two feet, land on one foot)
(BB) jump with 0.5 turn in side position
(BB) split jump with 0.5 turn
(BB) split, jump with turn
(BB) split jump
(BB) straddle pike jump with 0.5 turn in side po- sition
(BB) straddle pike jump with 0.5 turn
(BB) straddle pike jump with turn
(BB) straddle pike jump or side split jump in side position
(BB) straddle pike jump or side split jump
(BB) johnson with additional 0.5 turn
(BB) johnson (leap forward with leg change and
0.25 turn to side split or straddle pike position)
(BB) split and switch leap
(BB) switch leap with 0.5 turn
(BB) switch leap with turn
(BB) split leap with turn
(BB) switch leap (leap forward with leg change)
(BB) stag split leap forward
(BB) split leap forward
(BB) switch leap to ring position
(BB) tuck hop or jump, pick jump and stretched
(BB) tuck hop or jump with 1 turn
hop or jump
(BB) tuck hop or jump with 0.5 turn
(BB) pike jump from cross position
(BB) stretched jump/hop with turn
(BB) wolf hop or jump and cat leap
(BB) wolf hop or jump with I turn
(BB) wolf hop or jump with 0.5 turn
(BB) wolf hop or jump (hip angle at 45, knees to- gether)
(BB) cat leap (knees above horizontal alternately)
(BB) turn with legs in 180 split
(BB) I illusion turn through standing split, 180
legs separation
(BB) 1.5 turn with free leg held upward in 180 split position throughout turn
(BB) turn with free leg held upward in 180 split position throughout turn
(BB) turns with free leg at horizontal and turns with free leg optional below horizontal
(BB) 1.5 turn with heel of free leg forward at hor- izontal throughout turn
(BB) 2 turn with heel of free leg forward at hori- zontal throughout turn
(BB) turn with heel of free leg forward at hori- zontal throughout turn
(BB) 2 turn on one leg, free leg optional below horizontal
(BB) 1.5 turn
leg, free leg optional below
horizontal
(BB) I turn on one leg, free leg optional below horizontal
Continued on next page
26

---

Published as a conference paper at ICLR 2026
Table 12: Constructed category hierarchy for FineGym.
Coarse categories
Fine-grained categories
(BB) turn on one leg, thigh of free leg at hori- zontal, backward upward throughout turn
(BB) turn in tuck
(BB) 2.5 turn in tuck stand on one leg, free leg optional
(BB) 1.5 turn in tuck stand on one leg, free leg optional
(BB) turn in tuck stand on one leg, free leg op- tional
(BB) turn in tuck stand on one leg, free leg op- tional (BB) turn in tuck stand on one leg, free leg op- tional
(BB) salto backward
(BB) jump forward with 0.5 twist and salto back- ward tucked
(BB) salto backward piked
(BB) gainer salto backward stretched-step out
(feet land successively)
(BB) salto backward stretched-step out (feet land successively)
(BB) salto backward stretched with legs together
(BB) salto sideward tucked
(BB) salto sideward tucked with 0.5 turn, take off from one leg to side stand
(BB) salto sideward tucked, take off from one leg
side stand
(BB) free aerial
(BB) free aerial cartwheel landing in side position
(BB) free aerial round-off
(BB) free acrial cartwheel landing in cross posi- tion
(BB) free aerial walkover forward, landing on one or both feet
(BB) arabian salto tucked (take-off backward
(BB) arabian salto tucked (take-off backward with
with 0.5 twist, salto forward)
0.5 twist, salto forward)
(BB) salto forward tucked
(BB) salto forward tucked to cross stand
(BB) salto forward tucked (take-off from one leg
to stand on one or two feet)
(BB) salto forward piked to cross stand
(BB) salto forward piked to cross stand
(BB) flic-flac
(BB) flic-flac with twist, swing down to cross
straddle sit
(BB) flic-flac, swing down to cross straddle sit
(BB) gainer fic-flac, also with support on one arm
(BB) flic-flac with 0.75 twist to side handstand, keep 2 seconds, lower to optional end position
(BB) flic-flac with minimum 0.75 twist before hand support
(BB) jump backward, flic-flac take-off with 0.5 twist through handstand to walkover forward, also with support on arm
(BB) flic-flac to land on both feet
(BB) flic-flac with step-out, also with support on one arm
(BB) round-off
(BB) round-off
(BB) handspring forward with flight to land on
(BB) handspring forward with flight to land on
one or both legs, also with support on one arm
one or both legs, also with support on one arm
(BB) salto tucked
(BB) arabian double salto forward tucked
(BB) salto forward tucked with 1 twist Continued on next page
27

---

Published as a conference paper at ICLR 2026
Table 12: Constructed category hierarchy for FineGym.
Coarse categories
Fine-grained categories
(BB) salto forward tucked
(BB) double salto backward tucked with 1 twist
(BB) double salto backward tucked
(BB) salto backward tucked with twist
(BB) salto backward tucked
(BB) salto backward tucked with 1.5 twist
(BB) salto forward piked
(BB) salto forward piked
(BB) salto stretched
(BB) salto forward stretched with 1.5 twist
(BB) salto forward stretched with 1 twist
(BB) salto forward stretched
(BB) salto backward stretched with twist
(BB) salto backward stretched with twist
(BB) salto backward stretched with twist
(BB) salto backward stretched
(BB) salto backward stretched with 2.5 twist
(BB) salto backward stretched with 1.5 twist
(BB) salto backward stretched with 0.5 twist
(BB) double salto backward piked
BB) double salto backward piked
(BB) gainer salto
(BB) gainer salto backward stretched with twist
side of beam
(BB) gainer salto tucked at end beam
(BB) gainer salto piked at end of beam
(BB) gainer salto stretched with 1 twist at end of beam
(BB) gainer salto stretched with legs together at end of the beam
(UB) circle backward with turn
(UB) pike sole circle backward with 1.5 turn to handstand
(UB) pike sole circle backward with I turn to handstand
(UB) pike sole circle backward with 0.5 turn to handstand
(UB) giant circle backward with 1.5 turn to hand- stand
(UB) giant circle backward with hop I turn to handstand
(UB) giant circle backward with turn to hand- stand
(UB) giant circle backward with 0.5 turn to hand- stand
(UB) clear hip circle backward with turn to handstand
(UB) clear hip circle backward with 0.5 turn to handstand
(UB) clear pike circle hackward with 1 turn to handstand
(UB) clear pike circle backward with 0.5 turn to
handstand
(UB) stalder backward with turn to handstand
(UB) stalder backward with 0.5 turn to handstand
(UB) circle backward without turn
(UB) pike sole circle backward to handstand
(UB) giant circle backward
(UB) clear hip eircle backward to handstand
(UB) clear pike circle backward to handstand
(UB) stalder backward handstand
Continued on next page
28

---

Published as a conference paper at ICLR 2026
Table 12: Constructed category hierarchy for FineGym.
Coarse categories (UB) circle forward with turn
Fine-grained categories
(UB) pike sole circle forward with 0.5 turn to handstand
(UB) giant circle forward with I turn on one arm
before handstand phase
(UB) giant circle forward with turn to handstand
(UB) giant circle forward with 1.5 turn to hand- stand
(UB) giant circle forward with 0.5 turn to hand- stand
(UB) clear hip circle forward with 0.5 turn to handstand
(UB) stalder forward with 0.5 turn to handstand
(UB) circle forward without turn
(UB) giant circle forward
(UB) clear hip circle forward to handstand
(UB) clear pike circle forward to handstand
(UB) stalder forward to handstand
(UB) over high bar
(UB) (swing backward) with 0.5 turn and straddle
flight backward over high bar to catch high bar
(UB) counter straddle over high bar with 0.5 turn to hang
(UB) counter straddle over high bar to hang
(UB) counter piked over high bar to hang
(UB) (pike sole circle backward) counter stretched
(reverse hecht) in layout position over high bar to hang
(UB) hang high bar
(UB) (swing backward or front support) salto for- ward straddled to hang on high bar (UB) (swing backward) salto forward piked to hang on high bar
(UB) (swing forward or hip circle backward) salto backward with 0.5 turn piked to hang on high bar (UB) (swing backward) salto forward stretched to hang on high bar
(UB) (swing forward) salto backward stretched with 0.5 turn to hang high bar
(UB) transition flight
(UB) transition flight from high bar to low bar
(UB) transition flight from low bar to high bar
(UB) (swing forward) double salto backward
(UB) (swing forward) double salto backward tucked with 1.5 turn
(UB) (swing forward) double salto backward tucked with 2 turn (UB) (swing forward) double salto backward tucked with I turn (UB) (swing forward) double salto backward tucked
(UB) (swing forward) double salto backward
piked
(UB) (swing forward) double salto backward
stretched with 2 turn (UB) (swing forward) double salto backward stretched with I turn (UB) (swing forward) double salto backward
stretched
(UB) (swing forward) salto with 0.5 turn into
(UB) (swing forward) salto with 0.5 turn into salto
salto forward tucked
forward tucked
Continued on next page
29

---

Published as a conference paper at ICLR 2026
Table 13: Merge Setting for SSv2 Split A.
Merged Coarse Category
Attaching something to something and Trying but failing to attach something to something
because doesn't stick
Bending something
Burying something in something and Digging something out of something
Closing something and Opening something
Dropping something with spatial relation
Letting something roll
Lifting a surface with something on it
Lifting something up completely
Moving something and something
Picking something up and Pretending to pick something up
Poking a stack of something
Twisting something, Twisting (wringing) something wet until water comes out and Pre- tending trying and failing twist something
Pretending to put something with spatial relation
Pretending to spread air onto something and Spreading something onto something Pretending to squeeze something and Squeezing something
Pushing something
Putting multiple things
Putting something on the table
Putting something on surface or onto something
Showing something with spatial relation
Something falling
Spinning something
Pretending to take something from somewhere and Taking something from somewhere
Tearing something
Throwing something in the air
Tilting something with something on it Something colliding with something
Table 12: Constructed category hierarchy for FineGym.
Coarse categories
Fine-grained categories
(UB) salto forward tucked
(UB) (swing backward) double salto forward tucked
(UB) (swing backward) double salto forward
tucked with 0.5 turn
(UB) (under-swing or clear under-swing) salto
forward tucked with 0.5 turn
(UB) (swing backward) salto forward with 0.5 turn into salto backward tucked
(UB) (swing backward) salto forward with 0.5 turn into salto backward tucked
(UB) (swing forward) salto backward stretched with turn
(UB) (swing forward) salto backward stretched
with turn
(UB) (swing forward) salto backward stretched
(UB) (swing forward) salto backward stretched
Table 17: Grouping Detail of whether a subcategory has Analogies
W or w/o Analogies
Subcategory
with
Trying bend something unbendable so nothing happens Dropping something behind something
Dropping something front of something
Dropping something into something
Continued on next page
30

---

Published as a conference paper at ICLR 2026
Table 17: Grouping Detail of whether a subcategory has Analogies
W or w/o Analogies
Subcategory
Dropping something next something
Dropping something onto something
Lifting something up completely without letting drop down Lifting something up completely, then letting it drop down Pretending to put something behind something Pretending to put something into something Pretending to put something next to something Pretending to put something onto something Pretending to put something underneath something Pushing something from left to right Pushing something from right to left Pushing something onto something Pushing something so that slightly moves Showing something behind something Showing something next to something Pretending to be tearing something that not tearable
without
Attaching something to something
Trying but failing to attach something to something because it doesn't t stick
Bending something so that it deforms
Bending something until breaks
Burying something in something
Digging something of something
Closing something
Opening something
Letting something roll along a flat surface
Letting something roll down a slanted surface Letting something roll up slanted surface, so it rolls back down Lifting a surface with something on but not enough for it to slide down Lifting surface with something on it until it starts sliding down Moving something and something away from each other Moving something and something closer each other Moving something and something so they collide with each other Moving something and something so they pass each other Picking something up
Pretending to pick something up Poking stack of something so the stack collapses Poking stack of something without the stack collapsing Pretending or trying and failing to twist something Twisting (wringing) something wet until water comes out Twisting something
Pretending to put something on a surface Pretending to spread air onto something Spreading something onto something Pretending to squeeze something Squeezing something
Pushing something off of something Pushing something so that almost falls off but doesn't 't Pushing something so that falls off the table Pushing something with something Putting number of something onto something Putting something and something the table Putting something, something and something on the table Laying something the table its side, not upright Putting something similar to other things that are already on the table
Continued on next page
31

---

Published as a conference paper at ICLR 2026
Table 17: Grouping Detail of whether a subcategory has Analogies
W or w/o Analogies
Subcategory
Putting something that cannot actually stand upright upright the ble, so it falls on its side
Putting something upright on the table
Putting something on a flat surface without letting it roll
Putting something on a surface
Putting something on the edge of something so it is not supported and falls down
Putting something onto slanted surface but it doesn't 't glide down
Putting something onto something else that cannot support it so it falls down
Showing something on top of something Something falling like feather paper
Something falling like a rock Spinning something so it continues spinning Spinning something that quickly stops spinning Pretending to take something from somewhere Taking something from somewhere Tearing something into two pieces Tearing something just little bit Throwing something in the air and catching it Throwing something in the air and letting it fall Tilting something with something on it slightly so it doesn't 't fall down Tilting something with something it until it falls off Something being deflected from something Something colliding with something and both being deflected Something colliding with something and both come to halt Pushing something it spins
Table 18: Grouping Detail within Sub-Category Type
Type
Subcategory
Count
Putting number of something onto something Putting something and something the table
Putting something, something and something on the table
Spatiality
Moving something and something away from each other
Moving something and something closer to each other Dropping something next to something
Showing something next to something
Pretending to put something next to something Dropping something behind something Dropping something front of something Showing something behind something Pretending to put something behind something
Dropping something onto something Showing something on top of something Pretending to put something onto something Pretending to put something underneath something Pushing something onto something Pretending to put something on surface Putting something on a surface Dropping something into something Pretending to put something into something Pushing something from left to right Pushing something from right to left
Direction
Continued on next page
32

---

Published as a conference paper at ICLR 2026
Table 18: Grouping Detail within Sub-Category Type
Type Success
Subcategory
Trying but failing to attach something something because it doesn't stick
Attaching something to something
Pretending to pick something up
Picking something up
Pretending or trying and failing to twist something Twisting something
Pretending to spread air onto something
Spreading something onto something Pretending to squeeze something Squeezing something
Pretending to take something from somewhere Taking something from somewhere
Pretending to tearing something that tearable
Trying to bend something unbendable so nothing happens Poking stack of something without the stack collapsing Bending something so that it deforms Bending something until breaks Tearing something into two pieces Tearing something just a little bit Twisting (wringing) something wet until water comes out Burying something in something
State
Digging something of something
Closing something
Opening something
Poking stack of something so the stack collapses Throwing something in the air and catching it Throwing something in the air and letting it fall Lifting a surface with something on but not not enough for it to slide down Lifting surface with something Lifting up one end of something without letting it drop down Lifting up one end of something, then letting drop down Pushing something so that almost falls off but doesn't Pushing something so that falls off the table Tilting something with something on it slightly so it doesn't 't fall down Tilting something with something on it until falls off Letting something roll down a slanted surface
Motion
it until it starts sliding down
Letting something roll up a slanted surface, so it rolls back down Letting something roll along flat surface Putting something on a flat surface without letting it roll Pushing something it spins
Pushing something so that slightly moves Spinning something so it continues spinning Spinning something that quickly stops spinning Moving something and something so they collide with each other Moving something and something so they pass each other
Interaction
Something being deflected from something
Something colliding with something and both are being deflected Something colliding with something and both come to a halt Pushing something off of something Pushing something with something
Laying something the table its side, upright Putting something similar to other things that are already on the table Putting something that cannot actually stand upright upright on the ta- ble, so it falls on its side Putting something upright on the table
Continued on next page
33

---

Published as a conference paper at ICLR 2026
Table 18: Grouping Detail within Sub-Category Type
Type
Subcategory
Putting something on the edge of something so it is not supported and falls down
Putting something onto slanted surface but it doesn't 't glide down
Putting something onto something else that cannot support it so it falls
down
Something falling like a feather or paper
Something falling like a rock
34

---

Published as a conference paper at ICLR 2026
Table 14: Merge Setting for SSv2 Split B.
Merged Coarse Category
Moving the camera and Turning the camera
Covering something with something and Uncovering something
Holding something with spatial relation
Folding something and Unfolding something
Lifting up end of something
Moving something across surface
Moving something
Piling something up and Stacking number of something
Plugging something into something and Plugging something into something but pulling it right out as you remove your hand
Poking a hole
Pouring something
Pretending or failing to wipe something off of something and Wiping something off of
something
Pretending to close something without actually closing it and Pretending to open some- thing without actually opening Pretending to scoop something up with something and Scooping something up with some- thing
Pretending sprinkle air onto something and Sprinkling something onto something
Pretending to turn something upside down and Turning something upside down
Pulling something
Pulling two ends of something
Putting something with spatial relation
Putting something that can't 't roll onto slanted surface
Showing something's 's properties
Spilling something with spatial relation
Pretending to take something out of something and Taking something out of something
Pretending to throw something and Throwing something
Throwing something toward something Tipping something over and Tipping something with something in it over, something in it falls out
Poking something
35

---

Published as a conference paper at ICLR 2026
Table 15: Merge Setting for FineGyy2288 Split A.
Merged Coarse Category
(VT) round-off, flic-flac with 0,5 turn on, salto forward
(VT) round-off, flic-flac with turn on, salto backward
(VT) handspring forward on, salto forward
(FX) leap
(FX) jump or hop
(FX) 2 turn more
(FX) 3 turn
(FX) take-off forward from one or both legs, salto sideward tucked and (FX) aerial
cartwheel
(FX) salto forward
(FX) arabian double salto
(BB) ring and arch jump
(BB) wolf hop or jump and cat leap
(BB) turns with free leg horizontal and turns with free leg optional below horizontal
(BB) salto backward
(BB) free aerial
(BB) salto stretched
(UB) circle backward without turn
(UB) circle forward with turn
(UB) hang on high bar
(UB) transition flight
(BB) salto forward tucked
(BB) gainer salto
(UB) (swing forward) double salto backward
Table 16: Merge Setting for FineGym288 Split B.
Merged Coarse Category
(VT) round-off, flic-flac on, salto backward
(VT) tsukahara
(VT) handspring forward on
(FX) leap with turn
(FX) jump hop with turn
(FX) 1 turn
(FX) salto backward
(BB) split and straddle jump
(BB) split and switch leap
(BB) tuck hop or jump, pick jump and stretched hop or jump
(BB) turn with legs in 180 split
(BB) turn in tuck
(BB) salto sideward tucked
(BB) flic-flac
(BB) salto tucked
(UB) circle backward with turn
(UB) circle forward without turn
(UB) over high bar
(UB) salto forward tucked
36


Reviews:
[{"review_id": "ohSOKtviiX", "reviewer": "Reviewer_Lq1x", "summary": "This paper presents a method for zero-shot adaptation of video-language models to perform fine-grained classification. This is tackled by \"splitting\" existing categories to sub-categories. The method improves over baselines. The paradigm is also extended to a few-shot setting where similar gains are shown.", "questions": "None", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The method shows good improvement over baselines. The authors also provide many ablations such as choice of encoder, pretraining among others. The dataset generated has also been provided fully aiding in transparency. The concept of zero shot adaptation to finer granularity levels is interesting and warrants more attention.", "weaknesses": "The paper uses the term \"video model\" very generally to refer specifically to a kind of video-language model. This is misleading as \"video model\" can refer to other concepts such as video generation models.\n\nThe method is very hard to follow as the authors do not provide any preliminary information of the architecture that they are based upon. Eg. it is hard to follow which are the weight vectors that are being referred to as additive.\n\nSubjective: The title of the paper does not convey the problem being tackled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ykxcWdT0BC", "reviewer": "Reviewer_XTPn", "summary": "This work introduces the task of category splitting for video recognition: starting from an already trained video classifier with a fixed label set, you later realize a coarse label like “poking something” or “dropping something” actually needs to be broken into several fine-grained subactions, but you don’t want to retrain the whole model or disturb performance on the other, unrelated classes. The paper’s key insight is that existing video models already contain reusable, compositional structure in their classification head: across related actions, the difference between labels often looks like a consistent “modifier” (e.g. direction, manner, target), so they build a modifier dictionary from existing labels and then synthesize new subcategory weights by adding an appropriate modifier vector to the original coarse class weight, producing new labels in a fully zero-shot way; to go beyond seen modifiers, they train a small alignment module that maps modifier text to modifier vectors, and if a few labeled clips for the new sublabels are available, they fine-tune only those new head weights to improve accuracy while keeping the rest of the model fixed, thus preserving locality. Evaluated on split versions of video datasets (like SSv2-Split and FineGym-Split), the method improves recognition of the new, finer labels without degrading performance on untouched classes, showing that you can “edit” a video model’s label space after training by exploiting the structured differences already present in its classifier.", "questions": "- You show that adding a retrieved/aligned modifier vector to the coarse class weight works, but how often is the best modifier actually coming from the same base action vs. being borrowed from a semantically different action? I would like to see some statistical analysis of this.\n\n- How robust is the text encoder you use to messy, real-world label names (typos, multi-sentence definitions, multilingual labels etc) ? This choice should ideally be ablated", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Proposes and explores a novel problem of clear practical relevance\n- The work explores different forms of class splitting, both where the modifiers are seen or unseen\n- The proposed solution is tidy and lightweight: they edit only the classifier head, reuse structure already present in the model by building a modifier dictionary from existing labels\n- The authors construct 2 datasets for this problem from SSv2 and FineGym, which pose challenging test cases", "weaknesses": "- Method depends significantly on the assumption that the original label space already contains enough compositional variation to learn good modifier vectors\n\n- Because the edit happens at the classifier head, it also assumes the backbone already captures the visual distinctions the new sublabels require; if the new split introduces visual novelty rather than just semantic refinement, a head-only edit will struggle, and the paper doesn’t really explore that failure mode\n\n- There is some amount of over-claiming going on in this paper, e.g. the title claims \"YOUR VIDEO MODEL CAN BE EDITED\", but this method is fairly limited to a very specific set of situations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "auS5Zj1uxj", "reviewer": "Reviewer_duDd", "summary": "The paper proposes a method for zero-shot fine-grained classification by editing classifier weight matrices without retraining backbones. They extract \"modifier vectors\" by averaging coarse category weights from existing fine-grained examples and subtracting to isolate semantic differences, then use these to generate new subcategory weights via w_subcategory = w_coarse + v_modifier. The method is evaluated on SSV2-Split and FineGym-Split benchmarks, expanding classifier matrices from ~100 to ~150 categories while demonstrating improved performance over standard CLIP and VideoCLIP baselines on fine-grained classification tasks.", "questions": "I wonder, isn't it worth testing your model against fine-grained approaches (e.g., compositional approaches or methods that improve CLIP with additional losses for fine-grained evaluations)? There are many such methods, and while I'm not sure which one would be the best fit, I still raise this question. The current evaluation seems limited to basic vision-language baselines.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "-- Practical & Elegant Solution: Addresses real problem of fine-grained classification without expensive retraining - just intelligent matrix manipulation\n\n== Compositional Approach: The weight arithmetic (w_subcategory = w_coarse + v_modifier) is intuitive and enables systematic fine-grained category generation", "weaknesses": "-- Scalability Questions: Method requires existing fine-grained examples to extract modifiers, and matrix growth (100→150 categories) may not scale to truly large taxonomies\n\n-- Clarity and Organization Issues: Paper was slightly difficult to follow on the main contribution and method - would benefit from restructuring for better readability (more intuitive figures perhaps?)\n\n-- Insufficient Baseline Comparisons: Only compares against basic vision-language models (CLIP, VideoCLIP) rather than established fine-grained classification methods (e.g., other compositional approaches)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ACc4HBCmHg", "reviewer": "Reviewer_8rDB", "summary": "This paper introduces a new task named \"category splitting\", which aims to refine a coarse category into more detailed subcategories through classifier edits. They propose a zero-shot editing method that leverages the latent compositional structure within video models to create these new distinctions without requiring additional data. Besides, the study shows that low-shot fine-tuning is highly effective, and its performance is further enhanced when initialized with the edited weights. Experimental results on new benchmarks demonstrate that this approach significantly outperforms baselines, improving performance on the newly split categories while preserving performance on the others.", "questions": "* Why not choose the image encoder from a VLM as the base model, as it may possess better latent compositional structure during the pre-training.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "* This task introduces a new setting called \"category splitting\" and creates new benchmarks by reorganizing existing ones (SSv2, FineGym).\n* The proposed method is reasonable and performs well on new splits of category, outperforming existing prevalent Vision-Language Models.\n* The analysis section provides a sound analysis of the method's effectiveness, offering readers a deeper understanding of the method.", "weaknesses": "* The task setting proposed in this paper is akin to zero-shot classification or continual learning, but it is less challenging since the distributions of the fine-grained subcategories and the original coarse category are relatively close. Besides, a major limitation of this task setting is the requirement that the coarse and fine-grained categories must share the same base name, limiting the task's significance and practicality.\n* The method section is hard to follow. There exist undefined or unclaimed symbols, and some statements are confusing.\n* As stated in line 139, why use the mean of the associated fine-grained weight vectors as the pseudo vector of the coarse category? What if using the real text vector of coarse categories?\n* According to Table 2, despite achieving sound results on the new categories, it still causes a drop on the other categories, whereas VLMs (e.g. CLIP) can avoid the problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)