Title: Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening
Venue: NeurIPS 2025
Accepted: yes
Keywords: Time-sensitive video representations

Time chirality

Action recognition
Abstract: Our objective is to develop compact video representations that are sensitive to visual change over time. To measure such time-sensitivity, we introduce a new task: chiral action recognition, where one needs to distinguish between a pair of temporally opposite actions, such as “opening vs. closing a door", “approaching vs. moving away from something", “folding vs. unfolding paper", etc. Such actions (i) occur frequently in everyday life, (ii) require understanding of simple visual change over time (in object state, size, spatial position, count . . . ), and (iii) are known to be poorly represented by many video embeddings. Our goal is to build time aware video representations which offer linear separability between these chiral pairs. To that end, we propose a self-supervised adaptation recipe to inject time-sensitivity into a sequence of frozen image features. Our model is based on an auto-encoder with a latent space with inductive bias inspired by perceptual straightening. We show that this results in a compact but time-sensitive video representation for the proposed task across three datasets: Something-Something, EPIC-Kitchens, and Charade. Our method (i) outperforms much larger video models pre-trained on large-scale video datasets, and (ii) leads to an improvement in classification performance on standard benchmarks when combined with these existing models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on learning time-sensitive video representations. To achieve this, it (1) introduces a new task, chiral action recognition, where the model distinguishes between temporally opposite actions, and (2) proposes Linearized Feature Trajectories (LiFT), a method that injects time-sensitivity into frozen DINOv2 features. LiFT enforces linearity in the latent space while ensuring orthogonality between static and dynamic vectors. Experimental results demonstrate significant improvements in the proposed chiral action recognition task, with reasonable gains in standard action recognition tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Interesting Hypothesis:** The paper suggests an interesting idea for building video representations—per-frame features form a smooth trajectory in latent space, making the representation time-sensitive. This concept is inspired by human perception theory and is worth exploring.\n2. **Originality:** Many have tried to separate static and dynamic information, but this paper offers a fresh approach to solving that problem.\n3. **Clear Presentation:** The paper is well-written, easy to follow, and clearly explains the concepts, making them simple to understand.\n\n### Weaknesses\n\nOverall, I like the concept and direction of this paper. It presents a refreshing approach. However, there are a few concerns that need to be addressed before we can conclude that the proposed method effectively learns good video features:\n\n1. **Latent Space Assumption:** As stated in Equation 4, this method assumes a constant difference between consecutive frames in the learned latent space. However, real-world actions do not typically occur at a perfectly constant speed. Forcing a constant step size in the latent space could potentially limit the model's ability to accurately represent actions with highly variable speeds or sudden changes. Given that the paper primarily compares performance on short, relatively uniform videos, it would be valuable to see experimental results with more dynamic video benchmarks, such as long video recognition benchmark, to address this concern.\n2. **Potential Degradation on Spatial Features:** The linear step size in the learned latent space is a strong constraint, and I'm concerned that the improvements in handling temporal variations might come at the cost of losing spatial details. For instance, Table 4 shows that the method performs well for several attributes but struggles with the \"Spatial position ↔\" change. To investigate this, it would be helpful for the authors to include the \"DINOv2 ViT-S/14 Concat.\" baseline in Table 4. This would clarify whether LiFT's lower performance on this type of change is due to a degradation or an inherent limitation of the base features.\n3. **Missing Relevant Dataset:** The “action antonym” tasks in the TVBench dataset serve the same purpose as the CiA dataset—distinguishing between two exactly opposite actions. It would be helpful to include a discussion of this overlap in the related work and highlight the differences between the datasets.[TVBench] Cores, Daniel, et al. \"TVBench: Redesigning Video-Language Evaluation.\" *arXiv preprint arXiv:2410.07752* (2024).\n\nQUESTIONS:\nPlease see Weaknesses section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces chiral action recognition, a novel task that evaluates a video representation's sensitivity to time by distinguishing between temporally opposite actions (e.g., opening vs. closing). To address this, the authors propose LiFT (Linearized Feature Trajectories)—a self-supervised method that transforms per-frame DINOv2 features into smooth linear, time aware trajectories in latent space. LiFT outperforms much larger video models on three chiral datasets and enhances standard action recognition benchmarks when combined with existing models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors constructed a chiral action classification benchmark, which are useful for a broader community.\n- LiFT + video models have a great boost in action classification performance.\n\nWeaknesses:\n- An ideal video representation for action classification should work for both chiral actions and spatial information dominant actions. From Table 2 it’s unclear whether LiFT gets better chiral classification at the cost of other action types.\n- In Table 2, comparing with video models + average pooling is not fair, because if an action is reverse played (open VS close door) then these models would get almost the same outputs after average pooling. Why not concatenate them as well?\n- Fundamentally I’m curious why the authors chose an image-based model for this task. Theoretically, video-level embedding with 3D convs or transformers should always outperform image models since they can always be reduced to an image model. Since this is an application paper, why do we spend time focusing on the model that’s not designed for the task (action classification)? This model seems to suggest that motion information should be processed (by the small Transformer encoder) after frame-wise features are fully extracted (by DINO), which is hard to be true.\n- The paper mentioned “supplement” multiple times but I didn’t seem to find it?\n\nQUESTIONS:\n- Have you considered training the image (or video) encoder and LiFT together? Would that improve performance even further?\n- In Table 5, the benefits brought by LiFT seem to diminish with the attention probe. Do you have any intuition on the reason?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent video foundation models’ representations still get confused between actions which look similar but are time-sensitive, such as opening a door vs closing a door, folding a paper vs unfolding a paper, moving from left to right and right to left. These actions defined by the authors are called Chiral Action Pairs. This paper introduces Linearized Feature Trajectory (LiFT) which aims to distinguish such chiral actions, which are time-sensitive, such as opening a door and closing a door, etc. The authors achieve this by constructing time-aware representation that allows linear separability of chiral pairs by identifying two summary tokens z_s (static) and z_d (dynamic). The authors benchmark the efficacy of their algorithm on the Chirality in Action (CiA) dataset, developed in this research by identifying and clubbing different chiral groups in three datasets, such as SSV2, EPIC and Charades. Benchmark results show that their method surpasses existing video foundation models on such chiral action recognition and improves existing benchmarks as well when combined with existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n•\tThe paper introduces a fresh test chiral action recognitions that really checks whether a model understands the direction of an action instead of just spotting static background clues. Building a meta-dataset from three well-known video sets makes the benchmark easy for others to adopt.\n\n•\tLiFT is basically an “adapter” that could be plugged on top of the popular DINO-v2 image encoder. After self-supervised training, it turns every video into just two compact vectors z_s and z_d, that clearly separate “what stays” from “what moves”, respectively.\n\n•\tDespite being tiny, LiFT beats much larger video transformers, models ten times bigger and trained on six times more clips by over 7% on the new chiral benchmark. That shows the design targets the right signal (temporal change).\n\n•\tLiFT is complementary rather than redundant as concatenating LiFT’s descriptors with those from standard models improves the performance, as shown in Table 5. \n\nWeaknesses\n\n•\tThe evaluation reduces each inverse-action pair to an isolated, binary, linear separability test, it rewards embeddings that encode a single shortcut signal while overlooking multi-step dynamics. As a result, a model can excel on the benchmark proposed in the paper and yet fail to generalize to realistic tasks that require long-range temporal reasoning, compositional actions, or non-linear decision boundaries.\n\n•\tLiFT collapses each frame into a global feature and then forces the whole video to follow a straight-line “time path,” which tends to keep track of what changes but washes out where things are. As a result, it struggles to tell “object on the left” from “object on the right,” scoring far below competing models on left-versus-right tasks, as also evident in Table 4, where compared to VMAE and VJEPA, LiFT performance is greatly degraded.\n\nQUESTIONS:\n- Could you please diagnose where the spatial bias of LiFT is coming from, as shown in Table 4? Is it really from the global pooling or some augmentation techniques used? For example, you could disable flips in your experiments. The drop looks significant, and I’d request authors to provide appropriate solution to prevent such degradation.\n\n- The evaluation protocol is still simple. Could you please report results on at least one task that needs multi-step reasoning?\n\nLIMITATIONS:\nWhile the authors mention the limitation at the end of the paper, they do not mention the societal impacts in the paper. Even if the authors feel that there is no negative societal impact (as mentioned in the NeurIPS checklist), they ignore positive societal impact. It would be better to write about societal impacts in general.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach to time-aware video representation learning through a method called LiFT (Linearized Feature Trajectories). The central idea is that per-frame features of a video lie on smooth trajectories that encode temporal change, and these can be mapped into a latent space where they become approximately linear. This is inspired by a human vision study on Perceptual Straightening Hypothesis. The authors propose a lightweight, self-supervised model that adapts frozen image features (from DINOv2) using a Transformer encoder and a latent linearity constraint.\nTo evaluate time-sensitivity, the authors introduce a new benchmark task—chiral action recognition—and construct a meta-dataset (CiA) from existing video datasets (SSv2, EPIC-Kitchens, Charades), focusing on pairs of temporally opposite actions (e.g., opening vs. closing). The evaluation protocol uses linear probes to assess the separability of these chiral pairs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-- Novel and elegant architecture: The idea of linearizing feature trajectories in a latent space is both conceptually appealing and practically effective. The model is simple, efficient (8.7M trainable parameters), and avoids the need for large-scale video model training.\n\n-- Strong empirical results: LiFT outperforms much larger and more complex video models (e.g., VideoMAE, InternVideo 2.5) on the chiral action recognition task across all three datasets. It also shows promise as a plug-in descriptor for general video understanding.\n\n-- Valuable dataset contribution: The CiA dataset is a well-constructed benchmark for evaluating temporal sensitivity in video models. It is diverse, includes both egocentric and exocentric views, and is built from widely used datasets. If released, it would be a valuable resource for the community.\n\nWeaknesses\n\n-- Dataset availability: The CiA dataset is a strong contribution, but its impact depends on public release. The authors should be encouraged to publish the dataset and code for reproducibility.\n\n-- Linearity assumption: While the latent straightening idea is compelling, it would be helpful to see more discussion or visualization of when this assumption holds or breaks down.\n\nQUESTIONS:\n--\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}