Paxion: Patching Action Knowledge in Video-Language Foundation Models
Authors: Zhenhailong Wang
Ansel Blume
Sha Li
Genglin Liu
Jaemin Cho
Zineng Tang
Mohit Bansal
Heng Ji
Venue: NeurIPS 2023 spotlight (2023)
Keywords: video-language model
action knowledge benchmarking
action understanding
temporal understanding
TL;DR: Benchmarking and enhancing video-language foundation models with action knowledge

Abstract: Action knowledge involves the understanding of textual, visual, and temporal aspects of actions. We introduce the **Action Dynamics Benchmark (ActionBench)** containing two carefully designed probing tasks: Action Antonym and Video Reversal, which targets multimodal alignment capabilities and temporal understanding skills of the model, respectively. Despite recent video-language models’ (VidLM) impressive performance on various benchmark tasks, our diagnostic tasks reveal their surprising deficiency (near-random performance) in action knowledge, suggesting that current models rely on object recognition abilities as a shortcut for action understanding. To remedy this, we propose a novel framework, **Paxion**, along with a new **Discriminative Video Dynamics Modeling (DVDM)** objective. The Paxion framework utilizes a **Knowledge Patcher** network to encode new action knowledge and a **Knowledge Fuser** component to integrate the Patcher into frozen VidLMs without compromising their existing capabilities. Due to limitations of the widely-used Video-Text Contrastive (VTC) loss for learning action knowledge, we introduce the DVDM objective to train the Knowledge Patcher. DVDM forces the model to encode the correlation between the action text and the correct ordering of video frames. Our extensive analyses show that Paxion and DVDM together effectively fill the gap in action knowledge understanding (~50% → 80%), while maintaining or improving performance on a wide spectrum of both object- and action-centric downstream tasks.

PAXION: Patching Action Knowledge in Video-Language Foundation Models
Zhenhailong Wang', Ansel Blume', Sha Li', Genglin Liu', Jaemin Cho?, Zineng Tang', Mohit Bansal', Heng Ji'
'UIUC PNC
Abstract
Action knowledge involves the understanding of textual, visual, and temporal as- pects of actions. We introduce the Action Dynamics Benchmark (ActionBench) containing two carefully designed probing tasks: Action Antonym and Video Rever- sal, which targets multimodal alignment capabilities and temporal understanding skills of the model, respectively. Despite recent video-language models' (VidLM) impressive performance various benchmark tasks, diagnostic tasks reveal their surprising deficiency (near-random performance) in action knowledge, sug- gesting that current models rely object recognition abilities shortcut for tion understanding. To remedy this, we propose a novel framework, PAXION, along with Discriminative Video Dynamics Modeling (DVDM) objective. The PAXION framework utilizes a Knowledge Patcher network to encode new action knowledge and Knowledge Fuser component integrate the Patcher into frozen VidLMs without compromising their existing capabilities. Due to limitations of the widely-used Video- Contrastive (VTC) loss for learning action knowledge, we introduce the DVDM objective to train the Knowledge Patcher. DVDM forces the model encode the correlation between the action text and the correct ordering of video frames. Our extensive analyses show that PAXION and DVDM together effectively fill the gap action knowledge understanding (-50% 80%), while maintaining or improving performance on wide spectrum of both object- and action-centro downstream tasks. The code and data will be made publicly available for research purposes at https: //g thub. com /wt.///eeaaaaaa/aaaaaaae git.
1 Introduction
Recent video-language models (VidLMs) [30, 25, 55, 35, 57, 52] have shown impressive performance on wide range of video-language tasks. However, such multimodal models are not without deficiencies: [24] points out that many popular video-language benchmarks [56, 3, 16] can be solved by looking at single frame, and [59] shows that vision-language models struggle to understand compositional and order relations in images, treating images as bags of objects. Such limitations suggest that models' understanding of actions, which may require several frames and comprehension of object relationships, may be lacking.
To test this hypothesis, we first define action knowledge as understanding of the cause and effect of actions in textual, visual, and temporal dimensions. To quantify a model's action knowledge, we introduce the Action Dynamics Benchmark (ActionBench). ActionBench contains two probing tasks: distinguishing between (1) video's s caption and the caption with its action verbs replaced by their antonyms; (2) the original and reversed videos. The benchmark also includes baseline task for controlling the undesired impact from domain mismatch and investigating potential bias towards objects. The baseline task requires the model differentiate between the original video captions
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

ActionBench
Action-centric DVDIA Loss
Probing Tasks
Downstream Tasks
Data
Frozen
action-centric representation DD0-
VidLM
Knowledge Patcher
Knowledge Fuser
backbone representation
Learning action knowledge
Integrating with backbone
Figure 1: Overview of the PAXION framework. The goal is to patch frozen VidLMs with action knowledge without compromising their general vision-language capabilities. The Knowledge Patcher (KP) aims to learn an action-centrie representation by leveraging ActionBench data ($ 2) and our newly proposed Discriminative Video Dynamics Modeling (DVDM) training objectives ($ 3.1). The Knowledge Fuser (KF) aims to obtain a balanced representation for general downstream tasks by fusing the KP with the backbone.
and altered versions with randomly replaced objects. We find that state-of-the-arr video-language foundation models [52, 57, 24] exhibit near-random performance on our action-oriented probing tasks while excelling on the object-oriented baseline task (Figure 2). This shows that VidLMs lack action knowledge and suggests that their impressive performance on other benchmarks may be attributed to their object recognition ability instead action understanding.
To address this shortcoming, we propose a novel framework, PAXION (Patching Actions), to patch ex- isting VidLMs with action knowledge without compromising their general vision-language (VL) capa- bilities. PAXION comprises two main components, the Knowledge Patcher and the Knowledge Fuser. The Knowledge Patcher (KP) is a Perceiver-based [21, 20] lightweight module attached to a frozen VidLM backhone used to augment the VidLM with action-aware representations. Through our preliminary experiments, one major challenge for patching action knowledge is that the widely-used Video-Text Contrastive (VTC) objective [43, 55, 29, 26, 28] is insufficient, which echoes the findings of related work [7, 24, 59]. Hence, inspired by dynamics modeling in robotics and reinforcement learning [1. 4, 15, 23, 391, we introduce the Discriminative Video Dynamics Modeling (DVDM) objective that forces the model to learn the correlation between an action's 's textual signifier, the action text (e.g. the word "falling"), and the action's 's visual depiction (e.g. clip of falling book). DVDM includes two new losses, Video-Action Contrastive (VAC) and Action aectionTemporrl Matching (ATM), which are compatible with VTC without additional parameters. Specifically, we formulate discriminative tasks using action antonyms and reversed videos, with special emphasis on learn- ing from data instances with salient state changes. We demonstrate that the synergy between the Knowledge Patcher and DVDM leads to a dramatic improvement on our ActionBench tasks.
Next, we investigate whether our Knowledge Patcher, which is specialized for action understanding. can be integrated into existing VidLMs for downstream tasks that require both action and object knowledge. To this end, we introduce the Knowledge Fuser (KF) component of PAXION which fuses the action-centric representation from the Knowledge Patcher with the object-centric representation from the frozen backbonn using cross- -oss-attennion. We show that the fused representation from PAXION improves both object and action understanding on a wide spectrum of tasks, including Video- Text Retrieval (SSv2-label [13, 24]), Video-to-Action Retrieval (SSv2-template [24], Temporal [45]), and Causal-Temporal Video Question Answering (NExT-QA [54]). Moreover, our analysis shows that the Knowledge Fuser is essential maintain a balance between the models' object-related understanding and improving performance on downtream action and temporal-orienter tasks.
We also test the robustness of PAXION by considering zero-shot cross-domain transfer setting on the Moments-in-Tinn -ime [38] and Kinetics [22] datasets. We find that the Knowledge Fuser is critical for increasing robustness to domain shifts and that positive transfer to unseen domains can be achieved by further ensembling PAXION with the backbone model.
To the best of our knowledge, this is the first work to systematically evaluate action knowledge and patch it into video-language foundation models. Our main contributions are threefold:
1. We introduce the Action Dynamics Benchmark (& 2), which probes action understanding capabilities in video-language models. We evaluate three state-of-the-an video-language foundation models and conclude that they lack basic grasp of action knowledge.
2. We propose a novel learning framework called PAXION, which patches the missing action knowledge into frozen video-language foundation models without hurting their gen-

---

Probing Task: Action Antonym (AA)
GT
VidLM
Result
"Book falling like rock" Original Action Text
23.2%
VidLM
Original Video
"oook rising like rock Action Antonym Text
X
76.8%
Probing Task: Video Reversal (VR)
GT
VidLM Result
49.9%
Original Video
"Book falling like rock" Original Action Text
VidLM
X
50.1%
Reversed Video
Baseline Task: Object Replacement (OR)
GT
VidLM
"Book falling like a rock
Result
77.9%
Original Action Text
VidLM
"Cellphone falling like a rock"
Original
X
22.1%
Object Replaced Text
Video
ActionBench-Egood
ActionBench-SSV2
94.2
94.0
52.4
80
76.0
76.3
80
(96)
8
Accurarcy (96)
70.2
60-
46.6-mmmm
30.1
51.7
50.5
60
51
40.3
53.6
57.0
50.5
R0-
Random Guess
Random Guess
Action Antonym
Antonym
20
20
Video Reversal
Video Reversal
Object Replacement
Object Replacement
InternVideo
CLIP-VIP
Singularity
Avg
InternVideo
CLIP-VIP
Singularity
Illustration of the probing tasks and baseline task in our proposed ActionBench. The bounding boxes in the video frames are purely for visualization. The numbers on the right show the ranking scores from state-of-the-art VidLM, Intern Video [52]. The model struggles to determine whether a book is "falling" or "rising," but can confidently identify the object to be a "book" instead of "cellphone". Bottom: ActionBench results of three recent VidLMs [52, 57, 24J. The column "Avg" indicates averaged results on each task across all three models. Existing VidLMs achieve near-random results on the probing tasks (AA and VR) while excelling the baseline task (OR). This demonstrates that existing VidLMs lack fanddamnnennt action knowledge and exhibit strong bias to object understanding.
Figure 2: Top:
eral vasion-language capabilities. The key components of PAXION include a Perceiver-basse Knowledge Patcher ($ 3) and cross-attention-bass. Knowledge Fuser ($ 4).
3. We propose the DVDM objective ($ 3.1), an improvement over the widely-used VTC loss, which forces the model to encode the correlation between the action and the correct ordering of video frames. Extensive experiments show that PAXION with DVDM improves the joint understanding of objects and actions while being robust domain shift.
2 Action Dynamies Benchmark (ActionBench): Do Video-Language Foundation Models Understand Action Knowledge?
To investigate the presence of action knowledge in state-of-the video-language foundation models, we propose the Action Dynamics Benchmark (ActionBench). ActionBench comprises the Action Antonym (AA) and Video Reversal (VR) probing tasks, along with the Object Replacement (OR) baseline task. The probing tasks evaluate the multimodal and temporal correlations between an action text and a video. The baseline task controls for the potential impact of domain mismatch.

---

We construct this benchmark by leveraging two existing open-domain video-language datasets, Ego4D [14] and Someethnng v2 (SSv2) [13], which provide fine-grained action annotations for each video clip. Compared to a previous verb understanding probing benchmark [42] based on MSRVTT [56] and LSMDC [44], ActionBench
more action-oriented, larger in scale, and contains both ego-centric and third-person videos. Detailed statistics can be found in Appendix B. An illustration of each ActionBench task can be found in Figure 2.
Probing Task: Action Antonym (AA). The Action Antonym task probes the multimodal alignment of the action text and the video representation. We formulate AA as a binary classification task that involves distinguishing the original text annotation from its altered version with the action replaced by its antonym, given the corresponding video clip. For example, if the original text is "Book falling like rock',, the action antonym text would be "Book rising like rock', We leverage the WordNet [36] database and manually constructed mappings to automatically construct the antonym texts (details in Appendix B).
Video Reversal (VR). The Video Reversal task probes the temporal understanding of actions. We formulate VR as a binary classification task, where given a video-text pair with at least one action and a reversed version of the video, the goal is to distinguish the original video from the reversed one. Achieving non-trivial performance on the Video Reversal task requires the model to understand the temporal sequence implied by the action. The Video Reversal task also evaluates VLMs' abilities to identify violations of physical knowledge, as some clips defy expectation when reversed (e.g. falling book becomes one which rises without any disceriible cause).
Probing Task:
Object Replacement (OR). Object Replacement is a binary classification task that requires the model to distinguish between the original text annotation and an altered version with objects tokens randomly replaced by other object tokens in the dataset. The Object Replacement task allows us understand: (1) whether current VidLMs rely on object recognition as a "shortcut" for video-text matching (i.e., if they have an bbject-bbased representation), and (2) whether poor performance on Action Antonym can be attributed to domain mismatch (i.e., not being trained on Ego4D or SSv2) instead of lack of action knowledge.
Baseline Task:
2.1 Evaluating Video Video-Language Models on ActionBench
We evaluate three recent video-language foundation models, Intern [52], CLIP-ViP [57] and Singularity-eempraal [24]', on ActionBench. Despite their impressive improvements on video- language benchmarks, these models struggle to achieve non-trivial performance on Action Antonym and Video Reversal, as depicted in Figure 2. The fact that they achieve significantly better performance on the Object Replacement task indicates a strong bias towards objects over actions, and affirms that the poor performance on AA is not solely result of domain mismatch. The near-random performance on the VR task indicates lack of basic temporal reasoning and physical knowledge.
These observations align with previous approaches [17,42, 59, 37] which show similar limitations models [43, 28] and earlier video-language models [11, 35]. We find that high performance on ridco-language benchmarks does not necessarily equate to stronger understanding of action knowledge.
in
3 Patching Action Knowledge in Frozen Video-Language Models
In 2, we showed that current VidLMs exhibit limitations in their understanding of action knowledge, a crucial component for developing a comprehensive understanding of the external world. This raises the important question: Can we enhance existing VidLMs with this missing knowledge without hurting their general video-language capabilities?
To this end, propose novel learning framework, PAXION, which comprises two main components: the Knowledge Patcher (KP) (S 3) and the Knowledge Fuser (KF) ($ 4). An overview of the PAXION framework can be found in Figure 1. Analogous to releasing patches to fix bugs in published software, the Knowledge Patcher is Perceiver-based [21, 20] lightweight module attached to a frozen VidLM for steering the VidLM towards action-centric representations. As the widely used video- language contrastive (VTC) objective is insufficient for learning action knowledge, we introduce
For simplicity, we "Singularity" to represent temporr""tyytempral" in our figures and tables.

---

Actionieecc-SSV [bax Bbockbnee
"Book falling ...!
original lext
"Book rising antorym text
Aligning original text and video
96.2
16.0
23.3
13
22
2
10
$30
Video-Text Contrastive (VTC)
80
111
2/2
Adding action antonym text as hard negs
original
40
videos
3/3
Video-Action Contrastive (VAC)
...
Random Guess
Action Antonym
Via
Discriminative Vidleo
Video
Object Replacement
2/2
Adding
negs
Dynamic Modeling (DVDM)
Intersvidee
CLIPMPP
Singularity
reversed
videos
V3a
Action-Temport Matching (ATM)
Knowledge
trained with VTC-DVDM
Patcher
Losses
100
ActionBench-SSu (with Patcher)
334
Data
VTC
ATM
66.3
22.4
backbone text feeture
... falling ... OR Te. rising .... original antonym
patched features
Frozen VidLM
Perceiver
OR
backbone video
Arguery
Action Antonym
Bandom Guess
original
reversed videos
C
DDM
@
learnalee
Video Reversal
videos
latents
Object Resfecernent
Paxion: Knowledge Patcher
Interviideo
CLIP-PP
Singularity
Figure 3: Illustration of the Knowledge Patcher component (bottom left) of PAXION and the training objectives (upper left). On the right, show the comparison of performance ActionBench before and after adding the Knowledge Patcher.
Table 1: ActionBench results (in accuracy %). KP_* refers Knowledge Patcher. AA and VR indicate the Action Antonym task and the Video Reversal task. vrc and DVDM stands for Video- Text Contrastive loss and newly proposed Diseriminative Video Dynamics Modeling losses detailed in 3.1. Trainable Param# indicates the size of the trainable parameters compared to the backbone.
Action Dynamics Benchmark (ActionBench) Results
Trainable Param#
AA
AA
VR (SSv2)
Backbone
Method [Patcher Training
Avg
(Ego4d)
(Ego4d)
(SSv2)
Backbone
58.8
46.2
51.8
48.3
51.3
KP- Transformer [VTC]
8.4M (1.8%)
68.2
62.8
65.5
60.6
64.3
Intern Vidco
KP- Perceeevve [VTC]
4.2M (0.9%)
66.5
63.6
69.8
71.0
67.7
KP-Perceiver [VTC+DVDM]
4.2M (0.9%)
90.1
75.5
90.7
87.4
85.9
Backbone
49.3
55.0
70.2
53.6
57.0
KP- Transformer (VTC)
3.9M (2.6%)
61.9
53.4
72.2
54.3
60.5
Clip-ViP
KP-Perceiver [VTC]
2.4M (1.6%)
61.9
54.6
71.5
48.8
59.2
KP- Perceiver [VIC+DVDM]
2.4M (1.6%)
89.3
56.9
89.3
66.0
75.4
Backhone
47.0
50.1
48.9
49.6
48.9
IP-Transformer [VTC]
3.9M (1.8%)
61.9
48.2
63.8
49.5
55.9
Singularity
KP-Perceiver [VTC]
1.3M (0.6%)
60.3
46.1
63.3
51.5
55.3
KP-Perceiver (VTC+DVDM)
1.3M (0.6%)
83.8
58.9
82.4
68.8
73.5
Human
92.0
78.0
96.0
90.0
89.0
Discriminative Video Dynamics Modeling (DVDM) ($ 3.1) objectives that force the model to encode the correlation between the actual action text (e.g., "falling") and the correct sequence of visual state-changes (i.e., video frames).
Knowledge Patching with Perceivers. Inspired by recent work [2, 27] leveraging Perceivers [21, 20] to extract language-related visual features, use Perceivers to extract knowledge-spectee features. As shown in Figure Knowledge Patcher,
lightweight Perceiver which per- forms cross-attention between a sequence lower-dimensional, learnable latents and the higher- dimensional visual embedding V* from frozen, pretrained VidLM backbone. To further investigate the viability of Perceivers as an alternative to Transformers [49], we include another variant of the KP where replace the Perceiver with standard Transformer Encoder. Table I shows that the

---

Perceiver-based KP achieves competitive or better performance compared to the Transformer variant while being 2-3 times smaller in scale. Architecture details of the KPss can be found in Appendix D.I.
Video-Text Contrastive (VTC) is insufficient for learning action knowledge. We initially train both variants of the Knowledge Patcher on the training set of ActionBench with only the Video-Text Contrastive (VTC) loss. VTC loss aligns the visual representation V from the KP with the pooled textual representation ** from the frozen backbone. Results in Table show that training with the VTC loss alone provides marginal to no improvements on Action Antonym (AA) and Video Reversal (VR), particularly on smaller backbone models. This suggests the need for new training objectives (S 3.1) for learning action knowledge.
3.1 Learning Action Knowledge with Discriminative Video Dynamics Modeling
To address the limitation of the VTC loss in learning action knowledge, we propose two new losses that draw inspiration from dynamics modeling in Robotics and Reinforcement Learning [1, 4, 15, 23, 39). Specifically, in a typical Markov Decision Process (MDP) setup, aims to predict the next world state 1 given the current world state 2y and the action ue Inverse dynamics modeling aims to predict the current action ú, given the current and next world state representation of the world states, existing work usually formulates
dynamics modeling
Tt, 2t+1- Given video frame forward dynamics modeling as a generative task [1, 39, 15], directly reconstructing the pixels or the latent embedding of the next frame. For inverse dynamics modeling. the action class is usually predicted using dedicated classification layer [4, 23, 11. However, our preliminary experiments show that the existing formulation cannot be directly applied in our setting due to the following unique challenges: (1) Real world videos are much more complex than videos in lab setting, with constantly changing backgrounds and moving camera angles, causing a large portion of visual features to be unrelated to the main objects and actions. Furthermore, without additional annotation, it is difficult to identify the frames corresponding to the "current" and "next" states, as actions may be continuous (e.g., "walking") or repetitive (e.g., "doing push-ups") within a video. Thus, the training signal from a regression loss becomes extremely noisy. (2) Unlike previous work that has a small fixed number of action classes, model actions natural language phrases, making direct classification inapplicable.
To address these unique challenges, we propose a novel "relaxed" formulation of dynamics mod- eling, dubbed Discriminative Video Dynamics Modeling (DVDM), which contains two losses: Video-Action Contrastive (VAC) and Action-Temporal Matching (ATM). Both VAC and ATM can be directly incorporated into the Video-Text Contrastive (VTC) loss without any additional parameters. As illustrated in Figure 3 Losses the VAC loss aims to encourage the model to learn the correlation between the visual observations and the actual actions. We formulate the VAC loss as adding action antonym texts as hard negatives. The ATM loss encourages the model to consider the temporal ordering of the visual observations (video frames). Instead of directly generating the next state frames, we formulate ATM as a discriminative task similar to Video Reversal in $ 2, where the model distinguishes reversed videos from the original videos, alleviating the need for explicit state annotations. In order to make sure that the reversed videos are indeed distinguishable from the original ones, we further introduce method (Appendix C) for identifying videos with salient state-changes by leveraging image-language foundation models [28]. The idea is to measure the frame-text and trame-frame similarity between the first and second half of video. We compute ATM loss only on the videos that have salient state-changes between frames. Experimental results, as shown in Table I and Figure 3, indicate that adding the DVDM objectives significantly improves the performance on both probing tasks, suggesting that the resulting representation from the Knowledge Patcher demonstrates a stronger understanding of action knowledge.
4 Leveraging Patched Action Knowledge for Downstream Tasks
In we showed that the Knowledge Patcher (KP) and DVDM objectives together effectively learn action knowledge-ppecific representations. However, these representations are highly specialized to action understanding, which may not be optimal for general downstream tasks that require both object and action understanding. Thus, the remaining challenge is to retain the general VL capabilities of the backbone while leveraging the newly learned action knowledge.

---

One naive idea is to simply use the backbone embeddings whenever the task is less action-centric However, difficult to decide when to use the backbone without prior knowledge of given task. Further, using the backbone embeddings alone gives up the patched action knowledge that can be essential for certain downstream tasks, such as action recognition. In this section, we demonstrate that we can get the best of both worlds by fusing the action-centric representation from the Knowledge Patcher with the object-centric representation from the frozen backbone.
VTC
backbone text feature
fused video feature
Cross-Attn
Knowledge Patchar
key vvalla
pooled backbone video feature
V
D00m
features
Paxion:
Knowledge Fuser
Figure 4:
Illustration of the
For this we introduce the second component of PAXION, the Knowledge Fuser (KF), illustrated in Figure 4 The KF takes the pooled visual feature (o*) from the frozen VL backbone as the input query, and performs cross-attention with the extracted visual tokens (V) from the Knowledge Patcher.
Knowledge Fuser component.
4.1 Experimental Setup
To evaluate the model's ability retain general visual-linguistic capabilities while leveraging newly learned action knowledge, we consider a spectrum video-language tasks with different emphases on object and action understanding. Specifically, we consider Video-Text Retrieval (SSv2-label [24]), which is object-centric and biased towards static appearances [59, 24]; Causal- Temporal VQA (NExT-QA [54]), which requires joint understanding of static objects and dynamic events; and Video- to-Action Retrieval (SSv2-template [24], Temporal-SSy [45]). which is highly action-centric and temporal-intensiin A task is considered to be temporal-intensive if it cannot be solved without correct temporal information [45], e.g., reversed or shuffled frames. For example, as illustrated in Figure 5, the Video-to- Aco-Action Retrieval task obscures object names in the text, making it impossible to align text with a video based solely on objects. Moreover, it impossible to distinguish "approaching" and "moving away" without considering the temporal ordering of the frames.
For PAXION, we finetune the Knowledge Fuser jointly with the Knowledge Patcher on downstream tasks using VTC loss. By default, the KP in PAXION is trained with VTC and DVDM losses ($ 3.1). We include the baselines KP-Transformer FT [VIC) and KP-Perceiver FT (vrc), which are both obtained by continuing to finetune the VTC-only KPs from Table 1 on downstream tasks. Additionally, we compare PAXION with Side-Tunine Tuning [61], J. a Creameter-Efficienn Finetuning (PEFT) method that could serve using the same Knowledge Patcher as PAXION and do alpha blending with the frozen backbone. Implementation and configuration details for each method and task can be found in Appendix D. The results are shown in Tables 2 and 3.
alternative to the KF. For the Side-Tuning variant, initialize the
Table Video-Texx Retrieval and Video-to-Action Retrieval results. RI and R5 represent Recall @ and Recall @5 (in %) respectively. Subscripts wt2 and :2n represent video-to-text and text-to-video, respectively.
Video-Tee Retrieval SSv2-label
Video-to-Action Retrieval
Method [Patcher Training Loss]
SSv2-template
Temporal-SSv2
Rl
R5.2
Rl
R5
R1
R5
R1
R5
Intern Video Backbone
18.8
39.9
19.9
40.0
5.6
15.9
11.2
35.8
KP-Transformer FT (VrC]
24.1
50.0
21.7
46.0
21.1
55.9
41.1
88.9
KP-Perceiver FT [VTC]
27.0
57.4
27.1
56.8
24.8
59.7
42.5
91.3
Side-Tuning [61] (VTC+DVDM]
30.9
59.2
26.6
53.1
22.2
55.1
50.2
90.9
PAXION IVCC+DVDMI
32.3
26.9
51.2
91.9
Analysis
PAXION improves joint understanding of objects and actions. Tables 2 and show that PAXION outperforms both the Backbone and the VTC-only baselines (KP-*). This indicates that PAXION not only retains the original VI. capabilities of the backbone, but also fills in the gap of the missing action knowledge by fusing the original representations with the patched ones. We corroborate this finding by observing more significant improvements action-centio and temporal-intensive tasks,

---

Table 3: Causal-Tempar VQA (NEXT-QA) results (in accuracy %) on the validation set. We consider both the original and ATP-hard [7] split. We report accuracy for all questions or specific types of questions, including causal (C), temporal (7), and descriptive (D) questions.
NEXT-QA
Method [Patcher Training Lows]
Original
ATP-hard [7]
all
C
T
all
Intern InternVidco Backbone
43.3
38.6
52.5
43.2
27.0
27.3
27.1
KP- Transffffrrerr FT [VTC]
46.1
45.0
61.3
48.1
32.5
33.6
33.0
KP-Perceiver FT [VTC]
46.0
46.0
58.9
48.0
30.1
31.6
30.7
Side-Tuning [61] [VTC+DVDM]
54.9
52.0
69.8
56.3
37.4
36.0
36.8
PAXION [VTC+DVDM]
56.0
53.0
68.5
57.0
38.8
38.1
38.5
Video-to-Action Retrieval (Temporal-SSv2)
Ranking Scores
"Approaching something with your camera"
45.1%
60.0%
X
"Moving away from something with your camera"
46.8%
23.5%
VTC-Fineture
Paxion
Causal-Temporal VQA (NEXT-QA)
Ranking Scores
Question:
V "raised his
camera"
18.8%%
29.9%
"what did the
X B. "bored"
baby X C. "tur back the toy"
28.2%
20.2%
approached near
X
legs"
21.7%
16. 2%
the camera? ?"
X "suck his thumb"
10.4%
VTC-Finetune
Paxion
Figure 5: Qualitative examples on Temporal- -Sv2 [45] and NEXT-QA [54]. VTC-Finetune and PAXION refer to methods in row 3 and row 5 in Tables 2 and 3.
such Temporal-SSv2 (+20% @R1), compared to object-centric tasks, such SSv2-label (+1 (+11% @R1)'. The decomposed results on NExT-QA (Table 3) show that PAXION helps more on the Causal (C) and Temporal (7) types of questions, and on the harder subset (ATP-hard) where the temporal and action knowledge is emphasized.
PAXTON also outperforms Side-Tuning highlighting the effectiveness of cross-attention for deep fusion. Specifically, the Knowledge Fuser allows us to attend to all extracted visual tokens from the Knowledge Patcher instead of only blending with pooled representations as in Side-Tuning.
Qualitative analysis. Figure shows two qualitative examples on Temporal -Sv2 and NEXT-QA. For the Temporal-SSh example, we find that the finetuned Knowledge Patcher trained with only VTC fails to distinguish "Moving away" from "Approaching." while PAXTON trained with DVDM successfully correlates the seemingly expanding object with the action "Approaching" For the NExT- QA example, the question asks the model to identify what happens after the action "approached near the camera" " The VTC baseline incorrectly selects the action "turn back to the toy," which happens before approaching the camera. On the other hand, PAXION successfully chooses "raised his hand to take the camera". This indicates a stronger understanding of both action dynamics words such as "approach" " and the temporal ordering implied by words such "after". Additional qualitative examples and analysis of failure cases can be found in Appendix E.
Disentangling the impact of Knowledge Patching and Fusing. We further investigate the dis- entangled impact of the Knowledge Fuser (KF) and the Knowledge Patcher (KP) with two ablation settings: (1) KP+Finetune, where instead of adding the KF, we directly finetune the KP trained with DVDM on downstream tasks; (2) KPIVTC]+KF, where we train the KP without DVDM and then add the KF upon it. The results are shown Figure 6, where the A score represents the relative difference of downstream task performance between our original PAXION (Row 5 in Tables 2 and 3) and the two ablated settings. The key observations are as follows: (1) The Knowledge Fuser contributes more to object understanding. From Figure 6 Left, find that the KF helps most when the tasks are more object-centric, c.g., label. On highly action-centriy tasks, c.g.. Temporal-SSv2, directly using the action- www.--oowowwoweeeeeeeeee
representation is preferable to fusing with the backbone representation.
"The scores are calculated between PAXION and CP-Perceiver FT (VTC). The improvement @R1 for SSv2- label is averaged across Rleat and Rl.20.
8

---

object-
SSv2-L RIv
+41.7
SSv2-L R1v 4.4
centric
SSv2-L Rit
+26.1
SSv2-L R1t
+2.2
NEXT-QA test
NEXT-QA test
+0.0
55v2-T
SSv2-T
+10.7
action- centric
Temp-55v2
Temp-55v2
+18.5
20
30
-5
5
10
15
20
Scorre (%%
Score (%)
Finetune v.S. Fuse
VTC V.S. VTC+DVDM
Figure 6: Left: Impact of the Knowledge Fuser. Comparinggfmeetunnng fusing the same Knowledge Patcher trained with VTC+DVDM losses. Right: Impact of action knowledge patching (DVDM) on downstream tasks. Comparing fusing with the Knowledge Patcher trained with VTC loss only or VTC+DVDM losses. The A score indicates the relative difference in terms of downstream task accuracy between our original PAXION and the ablated settings (detailed in $4.2).
(2) Patching with action knowledge contributes more to action-centri understanding. From Figure 6 Right, we find that patching with action knowledge, i.e., training with DVDM objectives, contributes to better performance on downstream tasks that are more action-centrre Importantly. this result also indicates that the improvements observed in Tables and do not come solely from adding the KF. However, if the task is more object-centric, such as SSv2-label, VTC training alone is sufficient.
Robustness to domain shift. Learned action knowledge should be generalizable to unseen tasks and domains. However, this goal is difficult to realize with only Nomain-specitf datasets like SSv2[13] which contains only 174 actions. Therefore, in Appendix A we conduct experiments on zero-shot cross-domain transfer which demonstrate that the Knowledge Fuser in PAXION increases robustness to domain shift and can introduce positive transfer during zero-shot inference.
5
Related Work
Limitations of vision-language contrastive pretraining. Since CLIP [43], multimodal contrastive losses have been the major pretraining objective for almost all recent image-language [43, 28, 27, 53, 51, and video-language models [55, 35, 10, 60, 12, 50, 57, 52]. Previous work (17, 24, 5] has revealed the limitation contrastive pretraining on fine-grained compositional understanding, verb understanding, and temporal reasoning. Concurrent work [37, 9] proposed mining hard negatives by rule-based heuristics or large-language models [8] to improve understanding of structured vision- language concepts and verbs. In this work, focus on general action knowledge which includes verb understanding as well as action temporal understanding. Instead of directly tuning the entire backbone in [59, 37]. PAXION enables fast action knowledge patching while also achieving improved performance on both object-centric and action-centric downstream tasks. It is worth noting that the hard negative mining method proposed by [37] can be easily incorporated with our VAC loss and could potentially result in stronger results.
Parameter-efficient fine-tuning (PEFT). The recent surge in the size of large language models [6, 41, 62, 40, 48] has spurred research on parameter-effiiient fine-tuning [18, 31, 61, 46, 19, 33). Although current video-language models are smaller in scale, we aim to develop PAXION to be applicable larger models that we anticipate will emerge in the near future. The most similar PEFT-related work to ours is Side-Tuning [61]. which we compare against in $ 4. At high-level, unlike existing PEFT methods that optimize for specific downstream tasks, PAXION designed to learn specific type of knowledge that can benefit various downstream tasks ($ 4.2). Furthermore, it unclear how to aggregate the task-specific parameters, such as those adapters [18] or low-rank layers [19], to perform multiple tasks. The versatility of PAXION allows for its use in learning various types of knowledge, each with its own Knowledge Patcher. Subsequently, the patched knowledge- specific representations can be fused together using one Knowledge Fuser. This work serves proof-of-concept where we focus on action knowledge. We leave the exploration of other types of knowledge and more comprchensive comparison with PEFT methods as future work.

---

6 Conclusions and Future Work
In this work we propose the ActionBench benchmark for evaluating models' understanding of action knowledge, and reveal a major deficiency in state-of-the-art video-language foundation models in this area. We then propose PAXION patch in such action knowledge without compromising models' existing capabilities. We show that PAXION significantly improves the model's action understanding while achieving competitive or superior performance on downstream tasks. One limitation of this work is that we only experimented with patching one type of knowledge. We intend address this in future work, where we plan expand PAXION to patch broader aspects of physical knowledge such as object affordances and mental simulation, and to explore fusion with multiple learned Knowledge Patchers.
Acknowledgements
This research is based upon work supported by U.S. DARPA ECOLE Program No. #HR001 12390060 and U.S. DARPA KAIROS Program No. FA8750-19-2-100 1004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
[1] Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016. 2, 6
[2] lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. ArXiv preprini, abs/2204.14198 2022. 5
[3] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivie, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812, 2017.
[4] Adrià Puigdomênech Badia, Pablo Sprechmann, Alex Vitvitskyi. Daniel Guo, Bilal Piot, Steven Kaptur- owski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, al. Never give up: Learning directed exploration strategies. arXiv preprins MXXi::000200603: 2020. 2,
[5] Piyush Bagad, Makarand Tapaswi, and Cees G. M. Snoek. Test of time: Instilling video- video-language models with sense of time. ArXiv, abs/2301. 020744 2023. 9
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are fow-shot leamers. Advances neural information processing systems, 33:1877- 1901, 2020. 9
[7] Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the" vidco" in video-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2917-2927, 2022. 2,8
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Scbastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233 2022. 9
[9] Sivan Doveh Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2657-2668, 2023.
[10] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106 1197, 2021. 9
[11] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid, Multi-moda transformer for video retrieval. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 28, 2020, Proceedings, Part IV 16, pages 214-229, Springer, 2020.
10

---

[12] Zijian Gao, Jingyu Liu, Sheng Chen. Dedan Chang, Hao Zhang, and Jinwei Yuan. Clip2tv: An empirical study transformer-based methods for video-text retrieval. arXiv preprint arX002211..0610, 1(2)6, 2021. 9
[13] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitage et The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference computer vision, pages 5842-5850, 2017. 2.4, 4, 9, 14
[14] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022, 4, 14
[15] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lec, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2560 PMLR, 2019. 2,
[16] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large- scale video benchmark for human activity understanding. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 961-970. IEEE Computer Society, 2015.
[17] Lisa Anne Hendricks and Aida Nematzadeh, Probing image-language transformers for verb understanding. arXiv preprint arXiv:2106. 09141, 2021. 4,9
[18] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Auariyan, and Sylvain Gelly. Parameter-efficient transfer learning for alp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019. 9
[19] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106.0066..... 2021. 9
[20] Andrew Jaegle, Sebastian Borgeaud, lean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107. 14795,, 2021. 2,
[21] Andrew Jaegle. Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. International conference on machine learning, pages 4651- 4664. PMLR, 2021. 2,4,
[22] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:/703 06950, 2017. 2
[23] Ashish Kumar, Saurabh Gupta, and Jitendra Malik. Learning navigation subroutines from egocentric videos. In Conference Robot Learning, pages 617-626. PMLR, 2020. 2, 6
[24] Jie Lei, Tamara L Berg, and Mohit Bansal, Revealing single frame bias for video-and iido-and-languuge learning. arXiv preprint arXiv:2206.0342 2022. 2, 7, 14, 17, 18
[25] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In CVPR, 2021. 1
[26] Dongxu Li, Junnan Li, Hongdong Li. Juan Carlos Niebles, and Steven C.H. Hoi. Align and prompt: Aideo-and-languee pre-training with entity prompts. In arxiv, 2021.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2:
Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023. 5, 9, 16
[28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ArXiv preprint, abs/2201. 2022. 2, 4, 6, 9, 15
[29] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 970659 2021. 2

---

[30] Linjie Li. Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. HERO: Hierarchical encoder for Video=Language onne-represnntationn pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2046-2065, Online, 2020. Association for Computational Linguisties.
[31] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint or 2iv::iii..00... 00000, 2021. 9
[32] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z. XU, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video- languaae pretraining. Advances in Neural Information Processing Systems, 35:7575- 7586, 2022. 15
[33] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter- -fffcicette fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:19500119900 2022. 9
[34] Iyy Loshchilov and Frank Hutter.
Decoupled
weight decay regularization. arXiv preprint
urXiv:1711001001, 2017. 17
[35] Huaishao Luo, Ji, Ming Zhong, Yang Chen, Wen Lei. Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputiin, 508:293-304,
2022. 1, 1,4,9
[36] George A Miller. WordNet: An electronic lexical database. MIT press, 1998. 4, 15
[37] Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding video-language models. arXiv preprint arXiv:2304.06708, 2023. 4,
[38] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million vidcos for event understanding. IEEE transactions on pattern analysis and machine intelligence, 42(2):502-508,
2019. 2, 14, 17, 18
[39] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. Advances in neural information processing systems, 28, 2015. 2,6 6
[40] OpenAL. Gpt-4 technical report. http:://arziaorr//////2200000::0 2023. 9
[41] Long Ouyang, Jeff Wu, Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. ArXiv preprint, abs/2203.02155, 2022. 9
[42] Jae Sung Park, Sheng Shen, Ali Farhadi, Trevor Darrell, Yejin Choi, and Anna Rohrbach. Exposing the limits of vidco-text models through contrast sets. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3574-3586, 2022. 4
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMI.R 2021. 2, 4, 9, 14
[44] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bertt Schiele. Movie description. International Journal of Computer Vision, 123:94 1230
2017. 4
[45] Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time tell: Discovering temporal data for temporal modeling. In Proceedings the IEEE/CVF Winter Conference Applications of Computer Vision, pages 535-544, 2021. 2, 14, 17, 18
[46] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient
transfer learning. arXiv preprini arXiv:2206.06:3 06552, 2022. 9
[47] Dídac Surís. Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning, arXiv preprint arXiv:2303.0812 08128, 2023. 18
12

---

[48] Hugo Touvron. Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems. 30, 2017.
[50] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for langggl and video-language tasks, arXiv preprint arXiv:2209,07533 2022. 9
[51] Wenhui Wang. Hangbo Bao. Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as foreign language: Beit pretraining for all vision and vision-language tasks, arXiv preprint arXiv:2208. 10442, 2022. 9
[52] Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu. Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint 12.00191,,, 2022. 1, 3, 4,9, 9, 16
[53] Zirui Wang. Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. ArXiv preprint, abs/2108. 10904, 2021."
[54] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9777-9786, 2021. 2, 7, 8, 17
guestion-answering
[55] Hu Xu, Gargi Ghosh, Po-Yao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCIIP: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 202/ Conference on Empirical Methods in Natural Longuage Processing, pages 6787-6800, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. 2.9
[56] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In 2016 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 5288 -299. IEEE Computer Society, 2016. 1, 4
[57] Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip- vip: Adapting pre-trained image-text model to video-language representation alignment. arXiv preprint Stti::20090043310 2022. 1, 2, 3, 4,9
[58] Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao. Houdong Hu, Xuedong Huang, Boxin Li. Chunyuan Li, et al. Florence: A new foundation model for computer vision. ArXiv preprint, abs/21 11. 1432, 2021. 9
[59] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-worde words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023. 4.
[60] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34, 2021. 9
[61] Jeffrey O Zhang, Alexander Sax. Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 28, 2020, Proceedings, Part III 16, pages 698-714. Springer, 2020.
7, 8, 9, 14, 17
[62] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diah, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. ArXiv preprint, abs/2205. 01 01068, 2022. 9
13

---

A Robustness to Domain Shift: Zero-shot Cross-Domain Transfer
Table 4: Evaluating robustness to domain shift. We train the models on SSv2-label and perform zero-shot action classification on out-of-domain datasets, i.e., Moments-In In-Time [38] and Temporal- Kinetic [45]. A indicates the relative increase/decrease compared to the backbone.
Zero-shot Cross-domain Transfer
Method [Patcher Training Loss]
Moments-In-Time Temporal-Kingg Kinnttc
Val (Acc)
A(%)
Val (Acc)
A(%
Interv Backbone
23.3
57.7
KP-Transformer FT [VTC]
16.5
-29%
44.7
-23%
KP-Perceiver FT [VTC]
9.9
-58%
24.7
-57%
Side-Tuning [61] [VTC+DVDM)
21.2
-10%
54.5
-6%
PAXION [VIC+DVDM]
21.6
-7%
49.7
-14%
w/o Knowledge Fuser w/ Backbone Ensemble
4.3
-82%
16.3
-72%
23.9
+3%
58.1
+1%
Humans acquire action knowledge through multisensory interactions, and have the remarkable ability to generalize to new objects and scenarios. Similarly, our ultimate goal is to learn the underlying rules of action knowledge that is generalizable to unseen domains. However, it is highly challenging when we are given only domain-specific datasets. For instance, the SSv2 dataset [13] only has 174 action classes, which is insufficient to capture the full range of open-world actions. The Ego4d dataset is limited to ego-centric videos, making it difficult to generalize to other types of videos. Training on such Nomain-speciffe data can easily lead to overfitting to spurious features and introduce catastrophic forgetting of tasks from other domains. In this section, we further explore whether PAXION is robust to domain shift and whether the learned action knowledge can bring positive transfer action-centric tasks unseen domains.
We consider a zero-shot cross doos-n transfer setting where we directly apply the models trained on SSv2-label [24] to unseen domains. We consider two zero-shot action classification tasks based on Moments-Ine Tn-e [38] and Temporal-Kinetic [45]. Moments-In- In-me contains 305 action classes with diverse types of videos that are distinct from SSv2, including movie clips, stock footages, and cartoons. Temporal- Kinetii contains 32 manually selected action classes from Kinetic-400, with special focus temporal reasoning. We directly use the action labels (c.g.. "bouncing" " and "kicking") "), as the text candidates for the zero-shot classification [43], which introduces additional domain shifts in terms of text distribution compared with the annotations in SSv2-label (e.g.. "book falling like a rock").
Fusing with the backbone improves robustness to domain shift. Table 4 shows the zero-shot action classification accuracy and the relative difference A(%) compared with the frozen backbone. We find that adding the Knowledge Fuser effectively increases robustness to domain shift, reflected by a smaller negative A. The Side-tuning also demonstrate similar benefit via alpha blending between the Knowledge Patcher and the backbone.
Positive transfer can be achieved by ensembling the Knowledge Fuser (KF) with the backbone. We further propose a simple inference trick, Backbone Ensemble, which combines the output
probability from the KF and the backbone model through addition. Specifically, the final prediction of
the action class index E 0, 1..., C is computed as = arg max (pa = c) Pa(2 = c)). where the number of classes, Pa and p6 are the predicted probability distribution from the KF and the backbone respectively. We obtain the final prediction by ranking the combined probability of the action text candidates. Our experiments show that this simple inference technique can effectively enhance zero-shot performance and achieve positive transfer on unseen domains.
B Details of Action Dynamics Benchmark (ActionBench)
We construct ActionBench based on two existing video-language datasets with fine-graines action text annotation, Ego4d [14] and SSv2 [13]. To automatically generate the antonym text for the Action
We subsample 2k instances for doing this evaluation.
14

---

Table 5: ActionBench Statistics
Dataset
#Train
#Eval
Video Type
ActionBench--gooke
274,946
34,369
first-person
ActionBench-SSy2
162,475
23,807
first-person, third-person
Antonym task, we leverage WordNet [36]* to find antonyms for verb text tokens. Additionally, construct an additional verb-to-antonym mapping by leveraging ChatGPT and manual curation, since the WordNet database does not cover all verbs in the action taxonomy of the dataset. Furthermore, to ensure that the action antonym indeed forms a negative video-text pair with the original video, we exclude verbs that do not have a semantically reasonable antonym, such "use" and "look". For Ego4d, we consider a subset of EgoClip [32] annotations, for SSv2 we consider the entire dataset. The final statistics of the training and evaluation splits be found in Table 5. For SSv2, since the test set does not provide label annotation, i.e., annotation with filled object names, we report scores on the validation set. For Ego4d, we evaluate the test For results in Table 1, train the Knowledge Patcher variants for one epoch on the training sets and report the accuracy on the evaluation sets. We downsampled the videos into 224x224 in scale with frame rate of 8 fps for both training and evaluation. For human evaluation, we randomly sample 50 instances for the Action Antonym and the Object Replacement task, and another 50 instances for the Video Reversal task. The human evaluation is done by the authors.
C Identifying State-change Salient Videos for Action-Temporal Matching (ATM)
As detailed in $ 3. we formulate the Action Temporal Matching (ATM) loss as distinguishing reversed video from the original one given action text. ATM requires the model to learn the correlation between the correct temporal ordering of the visual observations and the corresponding actions. However, some actions, such "wiping" and "holding". are repetitive continuous and may not result in visible state-changes across the frames in the video clip. This can introduce additional noise for the ATM loss when the reversed video is indistinguishable from the original one. To address this issue, we propose two metrics to identify state-change salient videos by leveraging image- language foundation models. We use pretrained BLIP [28] to compute (1) frame-text semantic change 6et, which indicates how the frame-text alignment changes across the first half and second half of the video; (2) frame-frame similarity O.n. which indicates how different the frames from the first half and second half the video are.
d
1
Σ
S(vi,t)- Σ S(vj, t)
N/2
iE(O,N/2)
JE(N/2.N)
EiEjo,N/2
ΣjENN/2N) VJ
0v =
(2)
N/2
N/2
where N is the total number of sampled f and t are the frame image embedding and the text embedding from pretrained BLIP encoders, S denotes cosine similarity.
Intuitively, if we observe large frame-text semantic change (6.z) and a small frame-frame similarity (8vv), could expect We empirically set a threshold for 8,, and 8 During training, we only compute ATM loss on videos that satisfy d,t 0.003 and 0.p 0.95. The metrics are computed off-line thus do not bring computational overhead during training. Figure shows an example of the videos that are kept and skipped based on the computed metrics.
salient state-changes between the first half and the second half frames.
"We use the WordNet Interface from NLTK https: htttps:/www.nlkk.. wwwp:/k...gghhooooooohheeeo "httpa://opent" wwtpccog///////cccccccooo chhttpt.. We use N 8 in experiments.
httpA/////////////aaatttt//o///a html.
15

---

"spreading margarine onto bread"
6ut 0.032
pe = 0.936
Keep
"holding bulb"
8
0.004
0,r 0.979
X Skip
Figure 7: Example of identifying state-change saliency in videos for forward dynamics modeling. &ut and 6e indicates frame- text semantic change and trame-frame similarity metrics.
Paxion (Ours)
VTC
Baseline
Baseline
VAC
Projection Pooling
Perceiver
Transformer
FFN
FFN
FFN
Cross-Atin
Crass-Attn
Salf-Alth
Frozen VidLM
Perceiver
witan
V
a. Knowledge Putcher (Perceiver)
Knowledge Fuser
Knowledge Patcher (Transformer)
Side-Tuning
Figure 8: Detailed architecture of Knowledge Patcher (Perceiver), Knowledge Patcher (Transformer), Knowledge Fuser and Side-Tuning fuser.
D Implementation Details
D.1 Architecture Details.
Figure 8 shows detailed architecture of the Knowledge Patcher and Knowledge Fuser in our PAXION framework, as well as the baseline variants being compared in Tables and
Knowledge Patcher (Perceiver). The Perceiver-based Knowledge Patcher contains single cross- attention layer and a two-layer feedforward network. The Perceiver module performs cross-attention between sequence of learnable latent queries 2 RRd and the visual embeddings V P.D from the frozen backbone, where P denotes the visual token length and D represents the hidden dimension of the visual backbone. Since the user-defined sequence length / and hidden dimension d of the learnable latent queries are typically much smaller than and / from the backbone, the Perceiver information bottleneck that extracts knowledge-specific features from the raw
module serves visual features. For instance, in the case of Intern Video [52] backbone, we set = 16, = 768 which much smaller than P 1576, D 1024 for each video clip with 8 sampled frames. Similar to BLIP-2 [27], when computing the similarity between the visual tokens v/ gt.d from the Knowledge Patcher and the single textual feature vector E Rd we first compute the pairwise similarity between each visual token and the text feature vector, and then take a maximum across all visual tokens as the final video-text similarity. The results Table demonstrate the Perceiver- based Knowledge Patcher achieves competitive or better performance compared to the Transformer variant while being 2-3 times smaller. Additionally, we measure the computation overhead of the two variants, and find that the Perceiver variant requires 10 times fewer multiply-add operations than the Transformer variant. This further demonstrate that Perceivers can serve effective and efficient extractors for knowledge-specee features.
16

---

Table 6: Detailed configurations for methods in Tables and 3, and Figure
has Knowledge Fuscr?
Trainable Paramă
Patching Objectives
Fusing/Finetuning
Method
Objectives
KP- Transfoorrrr FT KP- KPPPerceiver FT
X
8,4M (1.8%)
VTC
VTC
X
4.2M (0.9%)
VTC
VTC
Side-Tuning
X
4.2M (0.9%)
VTC DVDM
VTC
PaXION
8.2M (1.7%)
VTC DVDM
VTC
KP+Finetune KP[VTC]*KF
X
4.2M (0.9%)
VTC+ DVDM
VTC
/
8.2M (1.7%)
VTC
VTC
Table 7: Detailed training configurations for tasks in Tables 2, and 4.
Downstream Task
Patching
Patching #Epochs
Musing/Finetuningg
Fusing/Finetuning
Dataset
Dataset
#Epochs
SSv2-label [24]
SSv2
SSv2 SSv2-template
SSv2-template [24]
SSv2
2
Temporal-SSwe [45]
SSv2
I
SS+2-template
2
NExT-QA [54]
NEXT-QA
1
NEXT-QA
4
Moments-In -ineetts---Timee/ [38]
SSv2
1
SSv2
1
Temporal-King -Knnttic [45]
SSv2
SSv2
1
Knowledge Patcher (Transformer). The Transformer variant of the Knowledge Patcher a stan- dard Transformer Encoder which contains self-attention layer and feedforward layer. The Transformer Encoder performs self-attention on the raw visual embeddings V E P.D.D from the frozen backbone and output an updated visual emhedding V IR P.D.D. To obtain video-text similarity, we first project the visual embeddings into the same dimension as the textual feature vector I* E IRd and then do mean pooling before computing dot product.
Knowledge Fuser. The Knowledge Fuser has the same architecture the Knowledge Patcher which contains single cross-attention layer and a two-layer feedforward network. In this case, we the pooled visual feature from the backbone o* E gd to provide query and the Knowledge Patcher output IR P.D to provide key and value for the cross-attention. The intuition to obtain a balanced representation for general downstream tasks by fusing the action-centric KP representation (V) with the object-centric backbone representation.
Side-Tum MadenTuning. As an alternative to the Knowledge Fuser, we consider Side-Tuning [61] for further integrating the Knowledge Patcher with the backbone. Side-Tuning contains a base-model and side-model, where the base-model is pretrained and frozen and the side-model is trainable. In our setting, we treat the backbone as the base-moded and initialize the side-model using the trained Knowledge Patcher. We then side-tune the Knowledge Patcher along with the backbone using alpha blending. Specifically, the final fused visual feature Py is obtained by 9, a(e") + (1 a)o, where o* is the mean-pooled backbone visual feature, and the is the mean-pooled Knowledge Patcher feature. And Sigmoid(a) E [0, 1], where a learnable scalar.
D.2 Knowledge Patcher Training.
We use two Nvidia Tesla V100 (16GB) GPUs for all experiments. For the Knowledge Patcher variants in Table 1, we train them on the training set of the datasets in the ActionBench for one epoch with either VTC loss only or VTC DVDM (VAC + ATM) loss. We use Adam [34] optimizer with learning rate of le-5 and weight decay of 0.05. For the transformer variant, we use a batch size per GPU. For the Perceiver variant, we are able increase the batch to per GPU due to the reduced computation complexity.
17

---

D.3 Downstream Task Training.
Tables 6 and 7 shows detailed configurations for downstream task training with methods described in Tables 2 and 3, and Figure 6.
As shown in Table 7, the finetuning dataset for SSv2-label is identical to the SSv2 action knowledge patching dataset where the annotations are filled templates, such as "Book falling like a rock". The SSv2-template dataset, the other hand, contains the object-obscured version of the original SSv2 annotations such as "Something falling like a rock". For the Video-to-Action Retrieval tasks, we consider two different subsets from the SSv2 validation set with the object-obfuscated annotations: SSv2-template [24] and Temporal-SSv2 [45]. SSV2-template contains all 174 action classes while Temporal-SSv2 contains 18 manually selected action classes that require more temporally-demann distinctions, and cannot be distinguished using shuffled frames, such as "Approaching" and "Moving away". In order investigate the impact of the action knowledge patching, we do not finetune dedicated model for the 18 action classes for Temporal-SSv2, but instead use the model trained on SSv2-template to directly evaluate on Temporal-SSv2. Therefore, when observed larger improvements on Temporal-SSv2, we can draw the conclusion that patching with action knowledge contributes more to action-centric tasks ($ 4.2).
The hyperparameters, such as the learning rate, are identical to those used during Knowledge Patching training. For Video-Text Retrieval (SSv2-label) and Video-to-Action Retrieval (SSv2-template, Temporal-SSv2). the DVDM ($ 3.1) objective includes VAC and ATM, while for Causal-Temports VQA (NEXT-QA), we only use VAC. This is because the training instances NExT- QA are not formatted as video-text pairs but instead are in the format of multiple choice QA, making it not suitable for the ATM loss. Each video corresponds to one question and five candidate answers. We apply VAC to NExT-QA by adding action antonym text for each question as hard negative candidate answers.
For the downstream tasks (in Appendix A) for zero-shot cross-domain transfer (Moments-In Time [38] and Cemporal-Kinettc [45]), we use the model trained on SSv2 to perform zero-shot evaluation.
E Additional Qualitative Analysis
Figures and 10 show additional qualitative examples on downstream tasks. The examples in demonstrate that PAXION improves understanding of challenging actions that require fine fine-graine grained temporal reasoning on the frames. For example, whether is "pretending" to do something or actually doing that, and whether an object is moving "towards" or "awayy" from the camera.
In Figure 11, show failure cases of PAXION diseuss remaining challenges. We find that PAXION still struggle to understand negation and spatial attributes. For example, both VTC- Finetune baseline and PAXION fail distinguish 'without letting it drop down" from "then letting it drop down". For questions that require fine-grained spatial information of objects such as "how many goats can be spotted", PAXION cannot perform well. Potential solutions including incorporating the patched VidLM with code language model to disentangle perception and reasoning similar to ViperGPT [47]. By leveraging the strong logical reasoning ability of a code language model, we can easily solve the negation and counting problems by creating code scripts with booleans and loops, and then use the VidLMs as "API calls".
18

---

Dataset
GT
Text Candidates
Score Rank
Score Rank
Temporal-
/
"Moving something towards the camera"
28. 9%
42.6%
S8v2
X
"Moving something away from the camera"
34.0%
1
21.5%
2
VTC-Finetune
Paxion
MAVAYaVSIMM "Pretending to turn something upside down" "Tumnnng something upside down"
Video-to-Action Retrieval
Temporal-
[43.2%
2
43.1%
1
SSv2
X
(46.2%
24.9%
2
VTC-Finetune
Paxion
"Putting something, something and something the table"
4.0%
5
21.
template
X
"Something colliding with something and both are being deflected"
15.9%
2.8%
5
VTC-Finetune
Paxion
SSv2- template
"Something falling like rock" "Sprinkling something something"
8.6%
3
9.2%
X
12
VTC-Finetune
Paxion
Video- Toxt Retrieval (V2T)
SSv2-
"pretending to pick heart up" "pushing tomato it spins"
0.6%
25
label
X
3.1%
0.7%
14
VTC-Finetune
Paxion
SSv2-
"turning the right while filming old mobile phone"
6.2%
20.5%
label
"approaching smart phone with your camera"
22.6%
7.6%
3
VTC-Finetune
Figure Additional qualitative examples (Retrieval).
19

---

Question
GT
Answer Candidates
Score
Rank
Score
Rank
"wants to play with girl"
X
it"
27.1%
25 .5%
"why did the baby hold the ball and moving forward?"
"kick off the ground"
21.8%
2
24 2%
3
X
D. "give ball to lady'
19.2%
3
15.1%
4
Causal-Temporal VQA (NEXT-QA)
X
E. "person inside is walking"
13.7%
5
9.4%
5
VTC-Finetune
Paxion
/
A. "pet its back"
24.3%
2
63.1%
"what does man do dog stood him?"
X
B. "bends down to hug dog"
20.4%
16 2%
2
"resting"
24.8%
11.7%
3
X
D. "jump dog"
X
E. "walk towards the cameraman"
2.6%
VTC-Finetune
Paxion
"took goggles"
2%
26.4
"how did the man on the most right reacted after man in red showed hand gesture?"
X
"performed"
27. 77
19 9%
2
X
C.
"adjust the rein"
18.8%
4
19.1%
3
X
"excited"
21.3%
3
17 77%
4
"grab his shoulders"
21.4%
17 17.3%
5
VTC-Finctune
Paxion
Figure 10: Additional qualitative examples (VQA).
20

---

Video-Text Retrieval Video-to-Action Retrieval Failure Examples
Dataset
GT
Candidates
Score
Rank
Score
Rank
Temporal-
"Lifting something up completely without letting drop down"
25.4%
26.4%
SSv2
X
"Lifting one end of something. then letting it drop down"
47.2%
39.0%
VTC-Finetune
Paxion
SSv2-
/
"bending tube so that deforms" "holding soaps over paste"
0.02%
286
0.05%
213
label
X
13.9%
15.4 4%
VTC-Finetune
Paxion
NEXT-QA Failure Example
Question
GT
Candidates
Score
Rank
Score
Rank
V
A. "eight"
8.5%
17.3%
5
X
B. "one"
23.3%
17.5%
4
"how many goats can be spotted?"
X
C. "two"
27.2%
20.2%
3
X
D. "three"
24.8%
24.3%
X
E. "four"
16.2%
20.6%
2
Figure 11: Failure examples.
21


Reviews:
[{"review_id": "9mmpaGxRTn", "reviewer": "Reviewer_ou3W", "summary": "This paper addresses the task of improving video-language understanding models, which a particular emphasis on their ability to align described actions to video segments and also their ability to model temporal dynamics. The authors first propose ActionBench, which is a modification of two datasets (Ego4D, SS-v2) for probing the degree to which action antonyms and reversed videos impact the models, and show that frozen VidLMs do not perform these probing tasks well. The authors then propose Paxion + a DVDM training objective: Paxion has a knowledge patcher network (better action encoding) and a knowledge fuser (incorporating into frozen VidLMs), and DVDM extends the standard contrastive VTC loss to better correlate action text with the specific ordering of the video frames. The authors then benchmark their approach on downstream datasets and tasks (SS-v2, NExT-QA, etc.)", "questions": "Please see the weaknesses section above for questions / areas to address in the rebuttal. Because the preliminary rating is borderline, it will be very helpful to finalize/potentially improve the rating.", "limitations": "The authors provide a brief discussion of the limitation on focusing on one type of knowledge patch in the conclusion + dataset cards in the supplement.\n\n---\n\n---\n\n**Post-rebuttal update:** \n\nThe authors partially address the weaknesses/comments described in the initial review:\n\n`+` The additional analysis on the NExT-QA results helps to better support their method/claims (compared to the original result, which showed minor differences), and we can see a larger gap on some settings for temporal/causal/etc where it is helping now.\n\n`+` The additional discussion with related work provides helpful context of how these works can be viewed as complementary to one another.\n\n`-` The limitations around the impact to downstream tasks, however, still remain. The authors make the argument that there are not a lot of temporal datasets, which is understandable. However:\n1. Given one of the highlighted contributions of the work is proposing a new *benchmark* for probing (ActionBench), it would have been much stronger to make a connection to at least a couple other datasets that represent a spectrum of temporal/action understanding. Does ActionBench performance help to differentiate downstream settings where temporal/action understanding correlates well? Why or why not? (This is something that is illustrated by the proxy tasks in [5] well, in Table 5 of their work; it would have been interesting to see if this benchmark helps to complement/refine that other analysis).\n2. Furthermore, the fact that many datasets are not temporal does not mean there aren't *any* additional datasets (or subsets of datasets) for event reasoning that could have been interesting to consider (e.g., [5] considers AGQAv2; there are others like STAR as well; [36] identifies an action-focused split of Kinetics). \n3. Finally, it's worth noting the current work considers *two* evaluation datasets (SSv2 and NExT-QA), and SSv2 is also the basis for the original probing task (so that leaves only one fully new dataset to make a connection to). In contrast, related and concurrent work consider *many more* to help establish the broad efficacy of their proposed learning objectives, techniques, and analysis tasks (e.g., [5] considers 5+ downstream, [36] considers at least 4). In general, I don't think it is strictly necessary for papers to benchmark on tons of datasets, but given that a core contribution of the work is centered around a proposed probing task (and corresponding remedy), establishing this connection with at least one other dataset would have significantly improved the contribution of the work.\n\n`-` The author response that \"the way we construct the probing datasets...is fully automatic, meaning that it is not confined to a particular dataset such as SSv2\" is not necessarily persuasive, since this pipeline is tied to the specific (templated) language distribution of SSv2, and  more importantly, some of the tasks may not make sense on other video distributions besides SSv2 (e.g., reversing the frame order may only make sense on SSv2's short videos of gestures/object interactions, where the reversed video can map to a realistic new video (\"push object\" $\\leftrightarrow$ \"pull object\"), whereas if you show a model a video of people walking backwards or \"uncooking\" a pancake, it isn't quite as effective as a probing task since that's not a realistic new action).\n\n---\n\n**Overall (post-rebuttal):** I think the rebuttal response did help to address some key concerns, and I do think there is value in considering this work in the broader context of other related efforts [5, 36], so I am increasing my rating leaning towards acceptance. I hope the points mentioned above regarding continued areas of improvement will be helpful for future revisions and/or later work.\n\n---", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "`+` Action/temporal understanding in large-scale pretrained video-language models is an important topic.\n\n`+` ActionBench contains probing tasks that seem like they may be interesting for future investigations, and the authors plan to release data and code for reproducibility.\n\n`+` The proposed Paxion/DVDM improvements seems to make a difference for the probing tasks relative to the base frozen VidLM, and there are ablations to show this.\n\n`+` Downstream evaluations on both SS-v2 and NExT-QA datasets to characterize the proposed model.", "weaknesses": "`-` There seems to be a limitation of how much the proposed techniques/tasks make a difference in the final downstream setting (e.g., side-tuning seems to ~match Paxion, within 0.3 points). This calls into question how over-specific the design of the proposed techniques and objectives are to the particular definitions of the proposed probing tasks in ActionBench (and by extension, how useful those formulations of the probing tasks are in downstream settings beyond SS-v2). (e.g., in [36] there is stronger impact on a larger range of downstream  video distributions/tasks)\n\n`-` The analysis for NExT-QA also seems incomplete. Prior/concurrent work (e.g., [5, 36]) report more detailed breakdowns of the accuracy into the specific causal/temporal splits of NExT-QA (are the improvements coming in the right categories?). Further, these other works report accuracy on a harder subset (ATP-hard) that makes the improvements from their verb/temporal augmentation techniques clearer. There is significant space in the current paper (Table 3) for these numbers (and they do not require additional compute, since they are all subsets), so it is unclear why they are not reported especially since they may help to bolster the core claims of the work.\n\n`-` The probing tasks seem closely related to ones proposed in prior work [5], but there doesn't seem to be a full discussion / comparison in this work of what these tasks add to those other ones (there is a brief citation in related work, but no discussion of this kind). Having a clearer sense of the potential complementarity would help to better clarify the contribution of this work (relatedly, how text/visual biases are controlled for in the probing tasks proposed here).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ou3W for the detailed and constructive comments. We appreciate your acknowledgment of the value of our proposed probing tasks and the DVDM objective. We will address your comments and questions in the following paragraphs.\n\n### Limited impact on downstream tasks\n1. First, we would like to clarify that the Side-Tuning row is not entirely a standalone baseline since it also incorporates our proposed VTC+DVDM objectives. The motivation for comparing with Side-Tuning is to demonstrate that the Patcher-Fuser framework in Paxion can perform competitively, or even better, than current PEFT frameworks. When comparing Paxion with the baselines that use only the VTC objective in Table 2 and 3, we do observe a significant improvement in Paxion's performance on downstream tasks, highlighting the impact of the DVDM objective. *(Please also see the updated Table 3 below.)*\n2. As detailed in Appendix B, the way we construct the probing datasets (i.e., the action antonym, video reversal and object replacement) is fully automatic, meaning that it is not confined to a particular dataset such as SSv2.\n3. We totally agree that it is ideal to show impact on more real-world downstream tasks. Unfortunately, as mentioned in the Introduction, many popular video-language downstream tasks still suffer from strong single-frame bias, underscoring the need for better Vid-L benchmarks. Our ActionBench serves as an initial step in this direction, and we will further explore establishing stronger connections to real-world downstream applications.\n\n### Detailed analysis of the Next-QA results\nWe totally agree with the reviewer's suggestion, as it can bolster the claims made in the paper. We already updated Table 3 to include a comprehensive breakdown of the results (including three types: Causal(C), Temporal(T), and Descriptive(D)), and the results on the ATP-hard split. \nSince the policy does not allow submitting revision, we include the **updated Table 3** in markdown below: \n\n| Method | NExT-QA | | | | | | | |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| | **Original** | | | | | **ATP-hard**|\n| | C | T | D | all | | C | T | all |\n| InternVideo Backbone | 43.3 | 38.6 | 52.5 | 43.2 | | 27.0 | 27.3 | 27.1 |\n| KP-Transformer FT [VTC] | 46.1 | 45.0 | 61.3 | 48.1 | | 32.5 | 33.6 | 33.0 |\n| KP-Perceiver FT [VTC] | 46.0 | 46.0 | 58.9 | 48.0 | | 30.1 | 31.6 | 30.7 |\n| Side-Tuning [VTC+DVDM] | 54.9 | 52.0 | **69.8** | 56.3 | | 37.4 | 36.0 | 36.8 |\n| **Paxion** [VTC+DVDM] | **56.0** | **53.0** | 68.5 | **57.0** | | **38.8** | **38.1** | **38.5** |\n\n*Table 3: Causal-Temporal VQA (NExT-QA) results (in accuracy %) on the validation set. We consider both the original and ATP-hard~[1] split. We report accuracy for 'all' questions or specific types of questions, including causal ('C'), temporal ('T'), and descriptive ('D') questions.*\n\nAs the reviewer expected, the decomposed results show that Paxion helps more on the Causal (‘C’) and Temporal (‘T’) types of questions, and achieves a more significant improvement on the harder subset (ATP-hard) where the temporal and action knowledge is emphasized.\n\n\n### Detailed comparison with the prior work “Test of Time” [5]\nWe identify the following key differences/contribution of our work compared with [5]. We will enrich line 281-283 to make these comparisons more explicit.\n\n1. While [5] primarily focuses on understanding the temporal ordering between events, our work focuses on general action knowledge, encompassing both causal and temporal understanding of actions.\n2. We propose very different probing tasks. [5] did not investigate verb or object replacement. The time-order reversal in [5] is reversing the appearing order of two video segments while the Video Reversal task in our ActionBench is reversing the frames of a single video.\n3. Our proposed Paxion framework enables fast action knowledge patching on frozen VL backbones, while [5] requires post-pretraining the backbone model.\n\n\n\n\n"}, {"review_id": "RlLL4yAL5C", "reviewer": "Reviewer_SP6c", "summary": "The manuscript presents an Action Dynamics Benchmark (ActionBench) with three new evaluation metrics for existing video-language models. Through ActionBench, the authors find that existing video-language models essentially rely on recognizing objects to recognize actions. Hence, a parameter-efficient component named Knowledge Patcher is connected in parallel to the video-language model and is trained with Discriminative Video Dynamics Modeling to improve the action understanding ability of the model. Finally, the authors present a knowledge fuser to infuse the knowledge learned by the Knowledge Patcher into the video-language model for downstream tasks. Empirical results show that training a knowledge patcher for use is effective in improving the video-text retrieval and temporal VQA tasks. \n", "questions": "- To help object-centric understanding, why don't you use the pre-trained representations from the backbone as well (e.g., concatenate or add them to the KP during fine-tuning)?", "limitations": "See weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Although it is proposed before that current video models may rely on spatial information to recognize actions in videos, the manuscript is the first to present such a benchmark with proper evaluation metrics to show the problem. This can inspire further work to solve the problem in video understanding. \n\n- The Discriminative Video Dynamics Modeling is straightforward and effective in training the video-language model to be aware of the action and the temporal direction of the videos. \n\n- The approach that the manuscript presents is similar to post-pre-training, and it is shown in Table 2 and Table 3 that such a post-pre-training strategy can benefit downstream retrieval tasks. \n\n- The writing is good and the organization of the manuscript is clear. ", "weaknesses": "- The evaluation in the downstream task is limited to video-text retrieval on SSV2 and VQA on NExT-QA, which makes it difficult to compare the presented approach with existing methods. Since the evaluation is based on InternVideo backbone, it is possible to provide some other comparisons where InternVideo has some published results. \n\n- Further on the above note, I am not quite familiar with VQA but on SSV2, the performance seems oddly low. Especially for video-to-text retrieval, which is essentially a text-driven classification for SSv2 videos, the performance has reached 61% for EVL-B/16 [1] when 8 frames are used, which also connect a side network to the pre-trained encoder. I didn't find details in both the manuscript and the supplemental material why this is the case. Could you provide an explanation for this? \n\n[1] Frozen CLIP Models are Efficient Video Learners\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer SP6c for the constructive comments. We are glad that you find our benchmark to be inspiring and our paper to be well-written. In the following paragraphs, we will address your comments and questions.\n\n### Limited downstream tasks for evaluation\nWe completely agree that evaluating more downstream tasks would be ideal. However, as mentioned in the Introduction, many popular video-language downstream tasks still suffer from strong single-frame bias, which does not faithfully reflect a model's understanding of action knowledge. This is precisely why we propose ActionBench; it serves as an initial step in building better Vid-L benchmarks that require true **video** understanding. We will investigate extending the probing tasks to connect with more real-world downstream applications in future work.\n\n### Low performance on SSv2\nWe thank the reviewer for pointing out this interesting observation. Here are several key differences between our setting and previous work. We will make these points clearer and do more investigating in the next version:\n1. In Paxion, as discussed in Sec 3 and Table 1, both the Knowledge Patcher and the Fuser are much more lightweight (one Perceiver layer; consisting of only 0.9% parameters) compared to the side module in [1]. This may result in limited expressiveness. Nevertheless, we acknowledge that further work on scaling would be beneficial.\n2. One of the fine-tuning objectives (DVDM) in Paxion emphasizes action understanding, while the video-to-text retrieval (SSv2-label) is more object-centric. Figure 6 shows that DVDM significantly benefits more action-centric and temporal-heavy tasks, such as video-to-action retrieval tasks.\n3. The backbone model might favor certain datasets. Although InternVideo demonstrates generally better zero-shot video retrieval performance than CLIP on many datasets (e.g., MSR-VTT, MSVD, etc.), there is no result on zero-shot SSv2 in the paper. This makes me wonder if CLIP is favored on SSv2 compared to InternVideo. We thank the reviewer for the insight and will include more controlled experiments to investigate this.\n4. Currently, we only fine-tuned on SSv2-label for one epoch for efficiency, as our main goal is to demonstrate the impact of the proposed DVDM objective versus VTC. We will investigate fully fine-tuning until convergence in the next revision.\n\n\n"}, {"review_id": "He8GdrQRk0", "reviewer": "Reviewer_CJDm", "summary": "In this paper, the authors introduce an interesting ActionBench, aiming to handle action antonym and video reversal problems. To remedy the problem in well-trained VidLM, the authors propose the DVDM objective, along with knowledge patcher and fuser. Extensive experiments demonstrate the effectiveness of the novel objective and modules.", "questions": "See weakness", "limitations": "See weakness", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel and internting ActionBench.\n- Simple modified contrastive objective to enhance the VidLM abilities for recognizing action antonym and video reversal problems.\n- Extensive ablation studies show the effectiveness.\n\nOverall, I appreciate the paper's idea to handle the interesting ActionBench. Though it has been somehow proposed in SthSth, there isn't any work trying to handle the problem. The authors propose the smart fine-tuning method with the perceiver as a knowledge patcher and fuser. And they design a modified contrastive loss for fine-tuning VidLM.", "weaknesses": "In fact, the VAC and ATM losses are modified contrastive losses with different positive and negative examples. However, the authors give a complicated motivation in Section 3.1 for their DVDM, which makes the paper harder to read. I suggest to present their idea in more easy-to-follow way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer CJDm for the constructive comments. We appreciate that you find our ActionBench to be novel and interesting. We will address your comments in the following paragraph.\n\n### Complicated motivation for the DVDM objectives\nWe appreciate the reviewer's suggestion, and will revise section 3.1 to provide a clearer presentation of this motivation.\n\nThe reason we motivate DVDM from the Markov Decision Process (MDP) is to give a more **generalized formulation** of the new **video dynamic modeling objective**. Due to the unique challenges mentioned in line 171, we leveraged a “relaxed” formulation of the dynamic modeling and integrated it into a contrastive learning framework as VAC and ATM losses. However,  in scenarios with cleaner training data or within a fully controlled environment (e.g., simulator), it is definitely feasible to explore a generative version of the video dynamic modeling objective in future research. Our current \"discriminative\" version serves as a straightforward yet effective initial step towards fine-grained modeling of visual changes in videos.  \n\n\n"}, {"review_id": "iq0NqFHNZi", "reviewer": "Reviewer_BUR1", "summary": "The paper tackles the known issue of CLIP-like models acting similar to bag-of-words models, in which structured information (e.g. relations between objects) is not really captured, so for action recognition the information leveraged is mostly object and scene (e.g. a person besides a guitar vs a person playing a guitar). The authors propose to use some extra augmentations related to action recognition information (both on the visual side and the language side), and also add an adaptor (a perceiver/Q-former) that becomes trainable using the extra augmentations. The underlying L&V model is kept frozen, so the training is efficient and the underlying model is not degraded for other domains.", "questions": "The main question is how this method compares against CLIP + ST-adapter or a similar strategy to adapting CLIP to video through lightweight adapters.\n\nOpen to hearing the author's opinion about the first point (the setting) ", "limitations": "No. \nAuthors could comment on the setting (need to have downstream training data), or the standard issues with vision + language pre-training for example. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- clear issue, sensible yet innovative solution, good results, all in a well-written paper.", "weaknesses": "\n- Right now there is little information in the main text that points to the \"patching data\", only the appendix has this info. I think this should be made very clear in the text and in the tables. One imagines that there's domain-specific fine-tuning, but it is not a must, it could be that the calibration set comes from a mixture of sources so the model has general applicability off-the-shelf. That the setting is the former is important. Does this make sense?\n\n- The other implication is: this assumes some CLIP-like model for video, and then uses the \"knowledge patching\" trained on downstream on-domain data. But one could equally decide to go the other way and start with the image CLIP, and train an adaptor to make it work on video using downstream data. This is the approach for example of ST-adapter and AIM. These comparisons are not included in the paper.\n\n(Not important but maybe interesting) The paper (text augmentations) reminded me of the following paper: \"CVPR'23, Teaching Structured Vision & Language Concepts to Vision & Language Models\", which is not discussed. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer BUR1 for the constructive comments. We are glad that you find our paper to be innovative and well-written. We will address your comments and questions in the following paragraphs.\n\n### Little information about the “patching data”\nWe appreciate the reviewer’s suggestion to include more details about the setting and the construction of the patching data. In the next version, we will clarify the following points in the main text:\n1. The construction of the patching dataset (action antonyms and reversed videos, detailed in Appendix B) is fully automatic, meaning that the users can create their own patching datasets easily.\n2. As shown in Table 7, the patching dataset and the fine-tuning dataset are not necessarily the same (i.e., patching with SSv2-label but fine-tuning on SSv2-template). Hence, as mentioned by the reviewer, it is possible to create a mixture of patching datasets for the model to learn general action knowledge and then apply it to a new task in a zero-shot manner. The zero-shot cross-domain transfer result in Appendix A demonstrates some initial promising results regarding this generalization ability, but we still have a large room for improvements.\n\n### Comparison with CLIP-adapted models\nWe agree with the reviewer that a promising line of work for solving video-language tasks is to initialize and adapt from strong image-language models, such as CLIP. In fact, we also included a recent representative model along this line as one of our backbone, CLIP-ViP. We want to reemphasize that the primary motivation of Paxion is to patch missing action knowledge into existing frozen foundation models, including those CLIP-adapted models. As depicted in Table 1, the original CLIP-ViP still demonstrates nearly random performance on our ActionBench, while Paxion significantly further enhances its action understanding. We thank the reviewer for the suggestion for adding comparison with CLIP+ST-adapter setup and will investigate that in the next version.\n\n### Comparison with the \"CVPR'23\" paper\nWe appreciate the reviewer's recommendation to compare our work with the paper \"CVPR'23, Teaching Structured Vision & Language Concepts to Vision & Language Models.\" In the next version, we will incorporate the following discussion in the related work section. \n\nAlthough the high-level idea is related, especially concerning negative instance generation, we identify the following major differences:\n1. They focus on object attributes and relations in static images, whereas we focus on actions and state-changes in dynamic videos.\n2. They adopt a LoRA-based PEFT method, while we propose a novel Perceiver-based patch-and-fuse framework, enabling the backbone to be fully frozen. \n"}]

Author rebuttal:

Meta-review: The paper received primarily positive reviews from 4 reviewers. Several questions were raised during the review period, including about performance analysis, missing comparisons with relevant prior work, and choice and generalization to downstream tasks made in the paper. The rebuttal seems to have addressed many of these concerns effectively. The authors are strongly encouraged to incorporate the information from the rebuttal and subsequent discussion into the final version for completeness.
Decision: Accept (spotlight)