SlowGuess/NeurIPS_2023_OCR huggingface dataset:
PAXION: Patching Action Knowledge in Video-Language Foundation Models
Zhenhailong Wang $^{1}$ , Ansel Blume $^{1}$ , Sha Li $^{1}$ , Genglin Liu $^{1}$ , Jaemin Cho $^{2}$ , Zineng Tang $^{2}$ , Mohit Bansal $^{2}$ , Heng Ji $^{1}$ $^{1}$ UIUC  $^{2}$ UNC  
{wangz3,hengji}@illinois.edu
Action knowledge involves the understanding of textual, visual, and temporal aspects of actions. We introduce the Action Dynamics Benchmark (ActionBench) containing two carefully designed probing tasks: Action Antonym and Video Reversal, which targets multimodal alignment capabilities and temporal understanding skills of the model, respectively. Despite recent video-language models' (VidLM) impressive performance on various benchmark tasks, our diagnostic tasks reveal their surprising deficiency (near-random performance) in action knowledge, suggesting that current models rely on object recognition abilities as a shortcut for action understanding. To remedy this, we propose a novel framework, PAXION, along with a new Discriminative Video Dynamics Modeling (DVDM) objective. The PAXION framework utilizes a Knowledge Patcher network to encode new action knowledge and a Knowledge Fuser component to integrate the Patcher into frozen VidLMs without compromising their existing capabilities. Due to limitations of the widely-used Video-Text Contrastive (VTC) loss for learning action knowledge, we introduce the DVDM objective to train the Knowledge Patcher. DVDM forces the model to encode the correlation between the action text and the correct ordering of video frames. Our extensive analyses show that PAXION and DVDM together effectively fill the gap in action knowledge understanding ( $\sim 50\% \rightarrow 80\%$ ), while maintaining or improving performance on a wide spectrum of both object- and action-centric downstream tasks. The code and data will be made publicly available for research purposes at https://github.com/MikeWangWZHL/Paxion.git.
Recent video-language models (VidLMs) [30, 25, 55, 35, 57, 52] have shown impressive performance on a wide range of video-language tasks. However, such multimodal models are not without deficiencies: [24] points out that many popular video-language benchmarks [56, 3, 16] can be solved by looking at a single frame, and [59] shows that vision-language models struggle to understand compositional and order relations in images, treating images as bags of objects. Such limitations suggest that models' understanding of actions, which may require several frames and comprehension of object relationships, may be lacking.
To test this hypothesis, we first define action knowledge as an understanding of the cause and effect of actions in textual, visual, and temporal dimensions. To quantify a model's action knowledge, we introduce the Action Dynamics Benchmark (ActionBench). ActionBench contains two probing tasks: distinguishing between (1) a video's caption and the caption with its action verbs replaced by their antonyms; (2) the original and reversed videos. The benchmark also includes a baseline task for controlling the undesired impact from domain mismatch and investigating potential bias towards objects. The baseline task requires the model to differentiate between the original video captions
Figure 1: Overview of the PAXION framework. The goal is to patch frozen VidLMs with action knowledge without compromising their general vision-language capabilities. The Knowledge Patcher (KP) aims to learn an action-centric representation by leveraging ActionBench data ( $\S 2$ ) and our newly proposed Discriminative Video Dynamics Modeling (DVDM) training objectives ( $\S 3.1$ ). The Knowledge Fuser (KF) aims to obtain a balanced representation for general downstream tasks by fusing the KP with the backbone.
and altered versions with randomly replaced objects. We find that state-of-the-art video-language foundation models [52, 57, 24] exhibit near-random performance on our action-oriented probing tasks while excelling on the object-oriented baseline task (Figure 2). This shows that VidLMs lack action knowledge and suggests that their impressive performance on other benchmarks may be attributed to their object recognition ability instead of action understanding.
To address this shortcoming, we propose a novel framework, PAXION (Patching Actions), to patch existing VidLMs with action knowledge without compromising their general vision-language (VL) capabilities. PAXION comprises two main components, the Knowledge Patcher and the Knowledge Fuser. The Knowledge Patcher (KP) is a Perceiver-based [21, 20] lightweight module attached to a frozen VidLM backbone used to augment the VidLM with action-aware representations. Through our preliminary experiments, one major challenge for patching action knowledge is that the widely-used Video-Text Contrastive (VTC) objective [43, 55, 29, 26, 28] is insufficient, which echoes the findings of related work [7, 24, 59]. Hence, inspired by dynamics modeling in robotics and reinforcement learning [1, 4, 15, 23, 39], we introduce the Discriminative Video Dynamics Modeling (DVDM) objective that forces the model to learn the correlation between an action's textual signifier, the action text (e.g. the word "falling"), and the action's visual depiction (e.g. a clip of a falling book). DVDM includes two new losses, Video-Action Contrastive (VAC) and Action-Temporal Matching (ATM), which are compatible with VTC without additional parameters. Specifically, we formulate discriminative tasks using action antonyms and reversed videos, with special emphasis on learning from data instances with salient state changes. We demonstrate that the synergy between the Knowledge Patcher and DVDM leads to a dramatic improvement on our ActionBench tasks.
Next, we investigate whether our Knowledge Patcher, which is specialized for action understanding, can be integrated into existing VidLMs for downstream tasks that require both action and object knowledge. To this end, we introduce the Knowledge Fuser (KF) component of PAXION which fuses the action-centric representation from the Knowledge Patcher with the object-centric representation from the frozen backbone using cross-attention. We show that the fused representation from PAXION improves both object and action understanding on a wide spectrum of tasks, including Video-Text Retrieval (SSv2-label [13, 24]), Video-to-ACTION Retrieval (SSv2-template [24], Temporal [45]), and Causal-Temporal Video Question Answering (NExT-QA [54]). Moreover, our analysis shows that the Knowledge Fuser is essential to maintain a balance between the models' object-related understanding and improving performance on downstream action and temporal-oriented tasks.
We also test the robustness of PAXION by considering a zero-shot cross-domain transfer setting on the Moments-in-Time [38] and Kinetics [22] datasets. We find that the Knowledge Fuser is critical for increasing robustness to domain shifts and that positive transfer to unseen domains can be achieved by further ensembling PAXION with the backbone model.
To the best of our knowledge, this is the first work to systematically evaluate action knowledge and patch it into video-language foundation models. Our main contributions are threefold:
1. We introduce the Action Dynamics Benchmark (§ 2), which probes action understanding capabilities in video-language models. We evaluate three state-of-the-art video-language foundation models and conclude that they lack a basic grasp of action knowledge.
2. We propose a novel learning framework called PAXION, which patches the missing action knowledge into frozen video-language foundation models without hurting their gen
Probing Task: Action Antonym (AA)
GT VidLM Result  $\boxed{77.9\%}$
Probing Task: Video Reversal (VR)
GT VidLM Result  $\boxed{49.9\%}$
Baseline Task: Object Replacement (OR)
GT VidLM Result  $\sqrt{77.9\%}$
Figure 2: Top: Illustration of the probing tasks and baseline task in our proposed ActionBench. The bounding boxes in the video frames are purely for visualization. The numbers on the right show the ranking scores from a state-of-the-art VidLM, InternVideo [52]. The model struggles to determine whether a book is "falling" or "rising," but can confidently identify the object to be a "book" instead of a "cellphone". Bottom: ActionBench results of three recent VidLMs [52, 57, 24]. The column "Avg" indicates averaged results on each task across all three models. Existing VidLMs achieve near-random results on the probing tasks (AA and VR) while excelling on the baseline task (OR). This demonstrates that existing VidLMs lack fundamental action knowledge and exhibit strong bias to object understanding.
eral vision-language capabilities. The key components of PAXION include a Perceiver-based Knowledge Patcher (§ 3) and a cross-attention-based Knowledge Fuser (§ 4).
3. We propose the DVDM objective (§ 3.1), an improvement over the widely-used VTC loss, which forces the model to encode the correlation between the action text and the correct ordering of video frames. Extensive experiments show that PAXION with DVDM improves the joint understanding of objects and actions while being robust to domain shift.
2 Action Dynamics Benchmark (ActionBench): Do Video-Language Foundation Models Understand Action Knowledge?
To investigate the presence of action knowledge in state-of-the-art video-language foundation models, we propose the Action Dynamics Benchmark (ActionBench). ActionBench comprises the Action Antonym (AA) and Video Reversal (VR) probing tasks, along with the Object Replacement (OR) baseline task. The probing tasks evaluate the multimodal and temporal correlations between an action text and a video. The baseline task controls for the potential impact of domain mismatch.
We construct this benchmark by leveraging two existing open-domain video-language datasets, Ego4D [14] and Something-Something v2 (SSv2) [13], which provide fine-grained action annotations for each video clip. Compared to a previous verb understanding probing benchmark [42] based on MSRVTT [56] and LSMDC [44], ActionBench is more action-oriented, larger in scale, and contains both ego-centric and third-person videos. Detailed statistics can be found in Appendix B. An illustration of each ActionBench task can be found in Figure 2.
Probing Task: Action Antonym (AA). The Action Antonym task probes the multimodal alignment of the action text and the video representation. We formulate AA as a binary classification task that involves distinguishing the original text annotation from its altered version with the action replaced by its antonym, given the corresponding video clip. For example, if the original text is "Book falling like a rock", the action antonym text would be "Book rising like a rock". We leverage the WordNet [36] database and manually constructed mappings to automatically construct the antonym texts (details in Appendix B).
Probing Task: Video Reversal (VR). The Video Reversal task probes the temporal understanding of actions. We formulate VR as a binary classification task, where given a video-text pair with at least one action and a reversed version of the video, the goal is to distinguish the original video from the reversed one. Achieving non-trivial performance on the Video Reversal task requires the model to understand the temporal sequence implied by the action. The Video Reversal task also evaluates VLMs' abilities to identify violations of physical knowledge, as some clips defy expectation when reversed (e.g. a falling book becomes one which rises without any discernible cause).
Baseline Task: Object Replacement (OR). Object Replacement is a binary classification task that requires the model to distinguish between the original text annotation and an altered version with objects tokens randomly replaced by other object tokens in the dataset. The Object Replacement task allows us to understand: (1) whether current VidLMs rely on object recognition as a "shortcut" for video-text matching (i.e., if they have an object-biased representation), and (2) whether poor performance on Action Antonym can be attributed to domain mismatch (i.e., not being trained on Ego4D or SSv2) instead of a lack of action knowledge.
2.1 Evaluating Video-Language Models on ActionBench
We evaluate three recent video-language foundation models, InternVideo [52], CLIP-ViP [57] and Singularity-temporal  $[24]^1$ , on ActionBench. Despite their impressive improvements on video-language benchmarks, these models struggle to achieve non-trivial performance on Action Antonym and Video Reversal, as depicted in Figure 2. The fact that they achieve significantly better performance on the Object Replacement task indicates a strong bias towards objects over actions, and affirms that the poor performance on AA is not solely a result of domain mismatch. The near-random performance on the VR task indicates a lack of basic temporal reasoning and physical knowledge.
These observations align with previous approaches [17, 42, 59, 37] which show similar limitations in image-language models [43, 28] and earlier video-language models [11, 35]. We find that high performance on video-language benchmarks does not necessarily equate to a stronger understanding of action knowledge.
3 Patching Action Knowledge in Frozen Video-Language Models
In § 2, we showed that current VidLMs exhibit limitations in their understanding of action knowledge, a crucial component for developing a comprehensive understanding of the external world. This raises the important question: Can we enhance existing VidLMs with this missing knowledge without hurting their general video-language capabilities?
To this end, we propose a novel learning framework, PAXION, which comprises two main components: the Knowledge Patcher (KP) ( $\S 3$ ) and the Knowledge Fuser (KF) ( $\S 4$ ). An overview of the PAXION framework can be found in Figure 1. Analogous to releasing patches to fix bugs in published software, the Knowledge Patcher is a Perceiver-based [21, 20] lightweight module attached to a frozen VidLM for steering the VidLM towards action-centric representations. As the widely used video-language contrastive (VTC) objective is insufficient for learning action knowledge, we introduce
For simplicity, we use "Singularity" to represent "Singularity-temporal" in our figures and tables.
Figure 3: Illustration of the Knowledge Patcher component (bottom left) of PAXION and the training objectives (upper left). On the right, we show the comparison of performance on ActionBench before and after adding the Knowledge Patcher.
Table 1: ActionBench results (in accuracy %).  $KP - *$  refers to Knowledge Patcher. AA and VR indicate the Action Antonym task and the Video Reversal task.  $VTC$  and  $DVDM$  stands for Video-Text Contrastive loss and our newly proposed Discriminative Video Dynamics Modeling losses detailed in § 3.1. Trainable Param# indicates the size of the trainable parameters compared to the backbone.
Action Dynamics Benchmark (ActionBench) Results
Backbone	Method [Patcher Training Loss]	Trainable
Param#	AA
(Ego4d)	VR
(Ego4d)	AA
(SSv2)	VR
(SSv2)	Avg
InternVideo	Backbone	-	58.8	46.2	51.8	48.3	51.3
KP-Transformer [VTC]	8.4M (1.8%)	68.2	62.8	65.5	60.6	64.3
KP-Perceiver [VTC]	4.2M (0.9%)	66.5	63.6	69.8	71.0	67.7
KP-Perceiver [VTC+DVDM]	4.2M (0.9%)	90.1	75.5	90.7	87.4	85.9
Clip-ViP	Backbone	-	49.3	55.0	70.2	53.6	57.0
KP-Transformer [VTC]	3.9M (2.6%)	61.9	53.4	72.2	54.3	60.5
KP-Perceiver [VTC]	2.4M (1.6%)	61.9	54.6	71.5	48.8	59.2
KP-Perceiver [VTC+DVDM]	2.4M (1.6%)	89.3	56.9	89.3	66.0	75.4
Singularity	Backbone	-	47.0	50.1	48.9	49.6	48.9
KP-Transformer [VTC]	3.9M (1.8%)	61.9	48.2	63.8	49.5	55.9
KP-Perceiver [VTC]	1.3M (0.6%)	60.3	46.1	63.3	51.5	55.3
KP-Perceiver [VTC+DVDM]	1.3M (0.6%)	83.8	58.9	82.4	68.8	73.5
Human	92.0	78.0	96.0	90.0	89.0
Discriminative Video Dynamics Modeling (DVDM) ( $\S 3.1$ ) objectives that force the model to encode the correlation between the actual action text (e.g., "falling") and the correct sequence of visual state-changes (i.e., video frames).
Knowledge Patching with Perceivers. Inspired by recent work [2, 27] leveraging Perceivers [21, 20] to extract language-related visual features, we use Perceivers to extract knowledge-specific features. As shown in Figure 3 Knowledge Patcher, we use a lightweight Perceiver which performs cross-attention between a sequence of lower-dimensional, learnable latents  $\mathcal{Q}$  and the higher-dimensional visual embedding  $\nu^{*}$  from a frozen, pretrained VidLM backbone. To further investigate the viability of Perceivers as an alternative to Transformers [49], we include another variant of the KP where we replace the Perceiver with a standard Transformer Encoder. Table 1 shows that the
Perceiver-based KP achieves competitive or better performance compared to the Transformer variant while being 2-3 times smaller in scale. Architecture details of the KPs can be found in Appendix D.1.
Video-Text Contrastive (VTC) is insufficient for learning action knowledge. We initially train both variants of the Knowledge Patcher on the training set of ActionBench with only the Video-Text Contrastive (VTC) loss. VTC loss aligns the visual representation  $\mathcal{V}$  from the KP with the pooled textual representation  $t^*$  from the frozen backbone. Results in Table 1 show that training with the VTC loss alone provides marginal to no improvements on Action Antonym (AA) and Video Reversal (VR), particularly on smaller backbone models. This suggests the need for new training objectives ( $\S$  3.1) for learning action knowledge.
3.1 Learning Action Knowledge with Discriminative Video Dynamics Modeling
To address the limitation of the VTC loss in learning action knowledge, we propose two new losses that draw inspiration from dynamics modeling in Robotics and Reinforcement Learning [1, 4, 15, 23, 39]. Specifically, in a typical Markov Decision Process (MDP) setup, forward dynamics modeling aims to predict the next world state  $\hat{x}_{t + 1}$  given the current world state  $x_{t}$  and the action  $u_{t}$ . Inverse dynamics modeling aims to predict the current action  $\hat{u}_t$  given the current and next world state  $x_{t}, x_{t + 1}$ . Given video frame as a representation of the world states, existing work usually formulates forward dynamics modeling as a generative task [1, 39, 15], directly reconstructing the pixels or the latent embedding of the next frame. For inverse dynamics modeling, the action class is usually predicted using a dedicated classification layer [4, 23, 1]. However, our preliminary experiments show that the existing formulation cannot be directly applied in our setting due to the following unique challenges: (1) Real world videos are much more complex than videos in a lab setting, with constantly changing backgrounds and moving camera angles, causing a large portion of visual features to be unrelated to the main objects and actions. Furthermore, without additional annotation, it is difficult to identify the frames corresponding to the "current" and "next" states, as actions may be continuous (e.g., "walking") or repetitive (e.g., "doing push-ups") within a video. Thus, the training signal from a regression loss becomes extremely noisy. (2) Unlike previous work that has a small fixed number of action classes, we model actions as natural language phrases, making direct classification inapplicable.
To address these unique challenges, we propose a novel "relaxed" formulation of dynamics modeling, dubbed Discriminative Video Dynamics Modeling (DVDM), which contains two losses: Video-Action Contrastive (VAC) and Action-Temporal Matching (ATM). Both VAC and ATM can be directly incorporated into the Video-Text Contrastive (VTC) loss without any additional parameters. As illustrated in Figure 3 Losses, the VAC loss aims to encourage the model to learn the correlation between the visual observations and the actual actions. We formulate the VAC loss as adding action antonym texts as hard negatives. The ATM loss encourages the model to consider the temporal ordering of the visual observations (video frames). Instead of directly generating the next state frames, we formulate ATM as a discriminative task similar to Video Reversal in § 2, where the model distinguishes reversed videos from the original videos, alleviating the need for explicit state annotations. In order to make sure that the reversed videos are indeed distinguishable from the original ones, we further introduce a method (Appendix C) for identifying videos with salient state-changes by leveraging image-language foundation models [28]. The idea is to measure the frame-text and frame-frame similarity between the first and second half of a video. We compute ATM loss only on the videos that have salient state-changes between frames. Experimental results, as shown in Table 1 and Figure 3, indicate that adding the DVDM objectives significantly improves the performance on both probing tasks, suggesting that the resulting representation from the Knowledge Patcher demonstrates a stronger understanding of action knowledge.
4 Leveraging Patched Action Knowledge for Downstream Tasks
In § 3, we showed that the Knowledge Patcher (KP) and DVDM objectives together effectively learn action knowledge-specific representations. However, these representations are highly specialized to action understanding, which may not be optimal for general downstream tasks that require both object and action understanding. Thus, the remaining challenge is to retain the general VL capabilities of the backbone while leveraging the newly learned action knowledge.
One naive idea is to simply use the backbone embeddings whenever the task is less action-centric. However, it is difficult to decide when to use the backbone without prior knowledge of a given task. Further, using the backbone embeddings alone gives up the patched action knowledge that can be essential for certain downstream tasks, such as action recognition. In this section, we demonstrate that we can get the best of both worlds by fusing the action-centric representation from the Knowledge Patcher with the object-centric representation from the frozen backbone.
For this we introduce the second component of PAXION, the Knowledge Fuser (KF), illustrated in Figure 4. The KF
Figure 4: Illustration of the Knowledge Fuser component.
takes the pooled visual feature  $(\mathfrak{o}^{*})$  from the frozen VL backbone as the input query, and performs cross-attention with the extracted visual tokens  $(\mathcal{V})$  from the Knowledge Patcher.
To evaluate the model's ability to retain general visual-linguistic capabilities while leveraging newly learned action knowledge, we consider a spectrum of video-language tasks with different emphases on object and action understanding. Specifically, we consider Video-Text Retrieval (SSv2-label [24]), which is object-centric and biased towards static appearances [59, 24]; Causal-Temporal VQA (NExT-QA [54]), which requires joint understanding of static objects and dynamic events; and Video-to-Action Retrieval (SSv2-template [24], Temporal-SSv2 [45]), which is highly action-centric and temporal-intensive. A task is considered to be temporal-intensive if it cannot be solved without correct temporal information [45], e.g., reversed or shuffled frames. For example, as illustrated in Figure 5, the Video-to-Action Retrieval task obscures object names in the text, making it impossible to align text with a video based solely on objects. Moreover, it is impossible to distinguish "approaching" and "moving away" without considering the temporal ordering of the frames.