SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For PAXION, we finetune the Knowledge Fuser jointly with the Knowledge Patcher on downstream tasks using VTC loss. By default, the KP in PAXION is trained with VTC and DVDM losses (§ 3.1). We include the baselines KP-Transformer FT [vTc] and KP-Perceiver FT [vTc], which are both obtained by continuing to finetune the VTC-only KPs from Table 1 on downstream tasks. Additionally, we compare PAXION with Side-Tuning [61], a Parameter-Efficient Finetuning (PEFT) method that could serve as an alternative to the KF. For the Side-Tuning variant, we initialize the "side-model" using the same Knowledge Patcher as in PAXION and do alpha blending with the frozen backbone. Implementation and configuration details for each method and task can be found in Appendix D. The results are shown in Tables 2 and 3.
Table 2: Video-Text Retrieval and Video-to-Action Retrieval results. R1 and R5 represent Recall@1 and Recall@5 (in %) respectively. Subscripts  $vt2$  and  $t_{2v}$  represent video-to-text and text-to-video, respectively.
Method [Patcher Training Loss]	Video-Text Retrieval	Video-to-Action Retrieval
SSv2-label	SSv2-template	Temporal-SSv2
R1v2t	R5v2t	R1t2v	R5t2v	R1	R5	R1	R5
InternVideo Backbone	18.8	39.9	19.9	40.0	5.6	15.9	11.2	35.8
KP-Transformer FT [VTC]	24.1	50.0	21.7	46.0	21.1	55.9	41.1	88.9
KP-Perceiver FT [VTC]	27.0	57.4	27.1	56.8	24.8	59.7	42.5	91.3
Side-Tuning [61] [VTC+DVDM]	30.9	59.2	26.6	53.1	22.2	55.1	50.2	90.9
PAXION [VTC+DVDM]	32.3	61.2	28.0	54.3	26.9	61.5	51.2	91.9
PAXION improves joint understanding of objects and actions. Tables 2 and 3 show that PAXION outperforms both the Backbone and the VTC-only baselines  $(KP - *)$ . This indicates that PAXION not only retains the original VL capabilities of the backbone, but also fills in the gap of the missing action knowledge by fusing the original representations with the patched ones. We corroborate this finding by observing more significant improvements on action-centric and temporal-intensive tasks,
Table 3: Causal-Temporal VQA (NExT-QA) results (in accuracy %) on the validation set. We consider both the original and ATP-hard [7] split. We report accuracy for all questions or specific types of questions, including causal  $(C)$ , temporal  $(T)$ , and descriptive  $(D)$  questions.
Method [Patcher Training Loss]	NExT-QA
Original	ATP-hard [7]
C	T	D	all	C	T	all
InternVideo Backbone	43.3	38.6	52.5	43.2	27.0	27.3	27.1
KP-Transformer FT [VTC]	46.1	45.0	61.3	48.1	32.5	33.6	33.0
KP-Perceiver FT [VTC]	46.0	46.0	58.9	48.0	30.1	31.6	30.7
Side-Tuning [61] [VTC+DVDM]	54.9	52.0	69.8	56.3	37.4	36.0	36.8
PAXION [VTC+DVDM]	56.0	53.0	68.5	57.0	38.8	38.1	38.5
Figure 5: Qualitative examples on Temporal-SSv2 [45] and NExT-QA [54]. VTC-Finetune and PAXION refer to methods in row 3 and row 5 in Tables 2 and 3.
such as Temporal-SSv2 (+20% @R1), compared to object-centric tasks, such as SSv2-label (+11% @R1) $^2$ . The decomposed results on NExT-QA (Table 3) show that PAXION helps more on the Causal  $(C)$  and Temporal  $(T)$  types of questions, and on the harder subset (ATP-hard) where the temporal and action knowledge is emphasized.
PAXION also outperforms Side-Tuning, highlighting the effectiveness of cross-attention for deep fusion. Specifically, the Knowledge Fuser allows us to attend to all extracted visual tokens from the Knowledge Patcher instead of only blending with pooled representations as in Side-Tuning.
Qualitative analysis. Figure 5 shows two qualitative examples on Temporal-SSv2 and NExT-QA. For the Temporal-SSv2 example, we find that the finetuned Knowledge Patcher trained with only VTC fails to distinguish "Moving away" from "Approaching," while PAXION trained with DVDM successfully correlates the seemingly expanding object with the action "Approaching". For the NExT-QA example, the question asks the model to identify what happens after the action "approached near the camera". The VTC baseline incorrectly selects the action "turn back to the toy," which happens before approaching the camera. On the other hand, PAXION successfully chooses "raised his hand to take the camera". This indicates a stronger understanding of both action dynamics in words such as "approach" and the temporal ordering implied by words such as "after". Additional qualitative examples and analysis of failure cases can be found in Appendix E.
Disentangling the impact of Knowledge Patching and Fusing. We further investigate the disentangled impact of the Knowledge Fuser (KF) and the Knowledge Patcher (KP) with two ablation settings: (1) KP+Finetune, where instead of adding the KF, we directly finetune the KP trained with DVDM on downstream tasks; (2) KP[VTC]+KF, where we train the KP without DVDM and then add the KF upon it. The results are shown in Figure 6, where the  $\Delta$  score represents the relative difference of downstream task performance between our original PAXION (Row 5 in Tables 2 and 3) and the two ablated settings. The key observations are as follows: (1) The Knowledge Fuser contributes more to object understanding. From Figure 6 Left, we find that the KF helps most when the tasks are more object-centric, e.g., SSv2-label. On highly action-centric tasks, e.g., Temporal-SSv2, directly using the action-knowledge-patched representation is preferable to fusing with the backbone representation.
2The scores are calculated between PAXION and KP-Perceiver FT [VTC]. The improvement @R1 for SSv2-label is averaged across  $R1_{v2t}$  and  $R1_{t2v}$ .
Finetune v.s. Fuse
VTC v.s. VTC+DVDM
Figure 6: Left: Impact of the Knowledge Fuser. Comparing finetuning or fusing the same Knowledge Patcher trained with VTC+DVDM losses. Right: Impact of action knowledge patching (DVDM) on downstream tasks. Comparing fusing with the Knowledge Patcher trained with VTC loss only or VTC+DVDM losses. The  $\Delta$  score indicates the relative difference in terms of downstream task accuracy between our original PAXION and the ablated settings (detailed in §4.2).
(2) Patching with action knowledge contributes more to action-centric understanding. From Figure 6 Right, we find that patching with action knowledge, i.e., training with DVDM objectives, contributes to better performance on downstream tasks that are more action-centric. Importantly, this result also indicates that the improvements observed in Tables 2 and 3 do not come solely from adding the KF. However, if the task is more object-centric, such as SSv2-label, VTC training alone is sufficient.
Robustness to domain shift. Learned action knowledge should be generalizable to unseen tasks and domains. However, this goal is difficult to realize with only domain-specific datasets like SSv2[13] which contains only 174 actions. Therefore, in Appendix A we conduct experiments on zero-shot cross-domain transfer which demonstrate that the Knowledge Fuser in PAXION increases robustness to domain shift and can introduce positive transfer during zero-shot inference.
Limitations of vision-language contrastive pretraining. Since CLIP [43], multimodal contrastive losses have been the major pretraining objective for almost all recent image-language [43, 28, 27, 53, 51, 58] and video-language models [55, 35, 10, 60, 12, 50, 57, 52]. Previous work [17, 59, 24, 5] has revealed the limitation of contrastive pretraining on fine-grained compositional understanding, verb understanding, and temporal reasoning. Concurrent work [37, 9] proposed mining hard negatives by rule-based heuristics or large-language models [8] to improve understanding of structured vision-language concepts and verbs. In this work, we focus on general action knowledge which includes verb understanding as well as action temporal understanding. Instead of directly tuning the entire backbone as in [59, 37], PAXION enables fast action knowledge patching while also achieving improved performance on both object-centric and action-centric downstream tasks. It is worth noting that the hard negative mining method proposed by [37] can be easily incorporated with our VAC loss and could potentially result in stronger results.
Parameter-efficient fine-tuning (PEFT). The recent surge in the size of large language models [6, 41, 62, 40, 48] has spurred research on parameter-efficient fine-tuning [18, 31, 61, 46, 19, 33]. Although current video-language models are smaller in scale, we aim to develop PAXION to be applicable to larger models that we anticipate will emerge in the near future. The most similar PEFT-related work to ours is Side-Tuning [61], which we compare against in § 4.1. At a high-level, unlike existing PEFT methods that optimize for specific downstream tasks, PAXION is designed to learn a specific type of knowledge that can benefit various downstream tasks (§ 4.2). Furthermore, it is unclear how to aggregate the task-specific parameters, such as those in adapters [18] or low-rank layers [19], to perform multiple tasks. The versatility of PAXION allows for its use in learning various types of knowledge, each with its own Knowledge Patcher. Subsequently, the patched knowledge-specific representations can be fused together using one Knowledge Fuser. This work serves a proof-of-concept where we focus on action knowledge. We leave the exploration of other types of knowledge and a more comprehensive comparison with PEFT methods as future work.
In this work we propose the ActionBench benchmark for evaluating models' understanding of action knowledge, and reveal a major deficiency in state-of-the-art video-language foundation models in this area. We then propose PAXION to patch in such action knowledge without compromising models' existing capabilities. We show that PAXION significantly improves the model's action understanding while achieving competitive or superior performance on downstream tasks. One limitation of this work is that we only experimented with patching one type of knowledge. We intend to address this in future work, where we plan to expand PAXION to patch broader aspects of physical knowledge such as object affordances and mental simulation, and to explore fusion with multiple learned Knowledge Patchers.
This research is based upon work supported by U.S. DARPA ECOLE Program No. #HR00112390060 and U.S. DARPA KAIROS Program No. FA8750-19-2-1004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
[1] Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016. 2, 6
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. ArXiv preprint, abs/2204.14198, 2022. 5
[3] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812, 2017. 1
[4] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020. 2, 6
[5] Piyush Bagad, Makarand Tapaswi, and Cees G. M. Snoek. Test of time: Instilling video-language models with a sense of time. ArXiv, abs/2301.02074, 2023. 9
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 9
[7] Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the" video" in video-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2917-2927, 2022. 2, 8
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. 9
[9] Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2657-2668, 2023. 9
[10] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097, 2021. 9
[11] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 214-229. Springer, 2020. 4
A Robustness to Domain Shift: Zero-shot Cross-Domain Transfer
Table 4: Evaluating robustness to domain shift. We train the models on SSv2-label and perform zero-shot action classification on out-of-domain datasets, i.e., Moments-In-Time [38] and Temporal-Kinetic [45].  $\Delta$  indicates the relative increase/decrease compared to the backbone.
Method [Patcher Training Loss]	Zero-shot Cross-domain Transfer
Moments-In-Time	Temporal-Kinetic
Val (Acc)	Δ(%)	Val (Acc)	Δ(%)
InternVideo Backbone	23.3	-	57.7	-
KP-Transformer FT [VTC]	16.5	-29%	44.7	-23%
KP-Perceiver FT [VTC]	9.9	-58%	24.7	-57%
Side-Tuning [61] [VTC+DVDM]	21.2	-10%	54.5	-6%
PAXION [VTC+DVDM]	21.6	-7%	49.7	-14%
w/o Knowledge Fuser	4.3	-82%	16.3	-72%
w/ Backbone Ensemble	23.9	+3%	58.1	+1%
Humans acquire action knowledge through multisensory interactions, and have the remarkable ability to generalize to new objects and scenarios. Similarly, our ultimate goal is to learn the underlying rules of action knowledge that is generalizable to unseen domains. However, it is highly challenging when we are given only domain-specific datasets. For instance, the SSv2 dataset [13] only has 174 action classes, which is insufficient to capture the full range of open-world actions. The Ego4d dataset is limited to ego-centric videos, making it difficult to generalize to other types of videos. Training on such domain-specific data can easily lead to overfitting to spurious features and introduce catastrophic forgetting of tasks from other domains. In this section, we further explore whether PAXION is robust to domain shift and whether the learned action knowledge can bring positive transfer to action-centric tasks on unseen domains.
We consider a zero-shot cross-domain transfer setting where we directly apply the models trained on SSv2-label [24] to unseen domains. We consider two zero-shot action classification tasks based on Moments-In-Time  $[38]^3$  and Temporal-Kinetic [45]. Moments-In-Time contains 305 action classes with diverse types of videos that are distinct from SSv2, including movie clips, stock footages, and cartoons. Temporal-Kinetic contains 32 manually selected action classes from Kinetic-400, with a special focus on temporal reasoning. We directly use the action labels (e.g., "bouncing" and "kicking"), as the text candidates for the zero-shot classification [43], which introduces additional domain shifts in terms of text distribution compared with the annotations in SSv2-label (e.g., "book falling like a rock").
Fusing with the backbone improves robustness to domain shift. Table 4 shows the zero-shot action classification accuracy and the relative difference  $\Delta (\%)$  compared with the frozen backbone. We find that adding the Knowledge Fuser effectively increases robustness to domain shift, as reflected by a smaller negative  $\Delta$ . The Side-tuning also demonstrate similar benefit via alpha blending between the Knowledge Patcher and the backbone.
Positive transfer can be achieved by ensembling the Knowledge Fuser (KF) with the backbone. We further propose a simple inference trick, Backbone Ensemble, which combines the output probability from the KF and the backbone model through addition. Specifically, the final prediction of the action class index  $c \in 0,1,\dots,C$  is computed as  $c = \arg \max_{i \in 0,1,\dots,C} (p_a(i = c) + p_b(i = c))$ , where  $C$  is the number of classes,  $p_a$  and  $p_b$  are the predicted probability distribution from the KF and the backbone respectively. We obtain the final prediction by ranking the combined probability of the action text candidates. Our experiments show that this simple inference technique can effectively enhance zero-shot performance and achieve positive transfer on unseen domains.
B Details of Action Dynamics Benchmark (ActionBench)
We construct ActionBench based on two existing video-language datasets with fine-grained action text annotation, Ego4d [14] and SSv2 [13]. To automatically generate the antonym text for the Action
<sup>3</sup>We subsample 2k instances for doing this evaluation.
Table 5: ActionBench Statistics
Dataset	#Train	#Eval	Video Type
ActionBench-Ego4d	274,946	34,369	first-person
ActionBench-SSv2	162,475	23,807	first-person, third-person
Antonym task, we leverage WordNet  $[36]^4$  to find antonyms for verb text tokens. Additionally, we construct an additional verb-to-antonym mapping by leveraging ChatGPT $^5$  and manual curation, since the WordNet database does not cover all verbs in the action taxonomy of the dataset. Furthermore, to ensure that the action antonym indeed forms a negative video-text pair with the original video, we exclude verbs that do not have a semantically reasonable antonym, such as "use" and "look". For Ego4d, we consider a subset of EgoClip [32] annotations, for SSv2 we consider the entire dataset. The final statistics of the training and evaluation splits can be found in Table 5. For SSv2, since the test set does not provide label annotation, i.e., annotation with filled object names, we report scores on the validation set. For Ego4d, we evaluate on the test set. For results in Table 1, we train the Knowledge Patcher variants for one epoch on the training sets and report the accuracy on the evaluation sets. We downsampled the videos into  $224 \times 224$  in scale with a frame rate of 8 fps for both training and evaluation. For human evaluation, we randomly sample 50 instances for the Action Antonym and the Object Replacement task, and another 50 instances for the Video Reversal task. The human evaluation is done by the authors.
C Identifying State-change Salient Videos for Action-Temporal Matching (ATM)
As detailed in § 3.1, we formulate the Action-Temporal Matching (ATM) loss as distinguishing reversed video from the original one given an action text. ATM requires the model to learn the correlation between the correct temporal ordering of the visual observations and the corresponding actions. However, some actions, such as "wiping" and "holding", are repetitive or continuous and may not result in visible state-changes across the frames in the video clip. This can introduce additional noise for the ATM loss when the reversed video is indistinguishable from the original one. To address this issue, we propose two metrics to identify state-change salient videos by leveraging image-language foundation models. We use pretrained BLIP [28] to compute (1) frame-text semantic change  $\delta_{vt}$ , which indicates how the frame-text alignment changes across the first half and second half of the video; (2) frame-frame similarity  $\theta_{vv}$ , which indicates how different the frames from the first half and second half of the video are.
$$
\delta_ {v t} = \left| \frac {1}{N / 2} \left(\sum_ {i \in [ 0, N / 2)} S (\mathbf {v} _ {\mathbf {i}}, \mathbf {t}) - \sum_ {j \in [ N / 2, N)} S (\mathbf {v} _ {\mathbf {j}}, \mathbf {t})\right) \right| \tag {1}
$$
$$
\theta_ {v v} = S \left(\frac {\sum_ {i \in [ 0 , N / 2)} \mathbf {v} _ {\mathbf {i}}}{N / 2}, \frac {\sum_ {j \in [ N / 2 , N)} \mathbf {v} _ {\mathbf {j}}}{N / 2}\right) \tag {2}
$$
where  $N$  is the total number of sampled frames $^6$ ,  $\mathbf{v}$  and  $\mathbf{t}$  are the frame image embedding and the text embedding from pretrained BLIP encoders,  $S$  denotes cosine similarity.
Intuitively, if we observe a large frame-text semantic change  $(\delta_{vt})$  and a small frame-frame similarity  $(\theta_{vv})$ , we could expect to see salient state-changes between the first half and the second half frames. We empirically set a threshold for  $\delta_{vt}$  and  $\theta_{vv}$ . During training, we only compute ATM loss on videos that satisfy  $\delta_{vt} > 0.003$  and  $\theta_{vv} < 0.95$ . The metrics are computed off-line thus do not bring computational overhead during training. Figure 7 shows an example of the videos that are kept and skipped based on the computed metrics.
4We use the WordNet Interface from NLTK https://www.nltk.org/howto/wordnet.html.
<https://openai.com/blog/chatgpt>
<sup>6</sup>We use  $N = 8$  in our experiments.
Figure 7: Example of identifying state-change saliency in videos for forward dynamics modeling.  $\delta_{vt}$  and  $\theta_{vv}$  indicates frame-text semantic change and frame-frame similarity metrics.
Figure 8: Detailed architecture of Knowledge Patcher (Perceiver), Knowledge Patcher (Transformer), Knowledge Fuser and Side-Tuning fuser.
D.1 Architecture Details.
Figure 8 shows detailed architecture of the Knowledge Patcher and Knowledge Fuser in our PAXION framework, as well as the baseline variants being compared in Tables 1, 2 and 3.
Knowledge Patcher (Perceiver). The Perceiver-based Knowledge Patcher contains a single cross-attention layer and a two-layer feedforward network. The Perceiver module performs cross-attention between a sequence of learnable latent queries  $\mathcal{Q} \in \mathbb{R}^{l,d}$  and the raw visual embeddings  $\mathcal{V}^* \in \mathbb{R}^{P,D}$  from the frozen backbone, where  $P$  denotes the visual token length and  $D$  represents the hidden dimension of the visual backbone. Since the user-defined sequence length  $l$  and hidden dimension  $d$  of the learnable latent queries are typically much smaller than  $P$  and  $D$  from the backbone, the Perceiver module serves as an information bottleneck that extracts knowledge-specific features from the raw visual features. For instance, in the case of InternVideo [52] backbone, we set  $l = 16$ ,  $d = 768$  which is much smaller than  $P = 1576$ ,  $D = 1024$  for each video clip with 8 sampled frames. Similar to BLIP-2 [27], when computing the similarity between the visual tokens  $\mathcal{V} \in \mathbb{R}^{l,d}$  from the Knowledge Patcher and the single textual feature vector  $t^* \in \mathbb{R}^d$ , we first compute the pairwise similarity between each visual token and the text feature vector, and then take a maximum across all visual tokens as the final video-text similarity. The results in Table 1 demonstrate the Perceiver-based Knowledge Patcher achieves competitive or better performance compared to the Transformer variant while being 2-3 times smaller. Additionally, we measure the computation overhead of the two variants, and find that the Perceiver variant requires 10 times fewer multiply-add operations than the Transformer variant. This further demonstrates that Perceivers can serve as effective and efficient extractors for knowledge-specific features.
Table 6: Detailed configurations for methods in Tables 2 and 3, and Figure 6.
Method	has Knowledge Fuser?	Trainable Param#	Patching Objectives	Fusing/Finetuning Objectives
KP-Transformer FT	✘	8.4M (1.8%)	VTC	VTC
KP-Perceiver FT	✘	4.2M (0.9%)	VTC	VTC
Side-Tuning	✘	4.2M (0.9%)	VTC + DVDM	VTC
PAXION	✓	8.2M (1.7%)	VTC + DVDM	VTC
KP+Finetune	✘	4.2M (0.9%)	VTC + DVDM	VTC
KP[VTC]+KF	✓	8.2M (1.7%)	VTC	VTC