Title: Paxion: Patching Action Knowledge in Video-Language Foundation Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: video-language model

action knowledge benchmarking

action understanding

temporal understanding
Abstract: Action knowledge involves the understanding of textual, visual, and temporal aspects of actions. We introduce the **Action Dynamics Benchmark (ActionBench)** containing two carefully designed probing tasks: Action Antonym and Video Reversal, which targets multimodal alignment capabilities and temporal understanding skills of the model, respectively. Despite recent video-language models’ (VidLM) impressive performance on various benchmark tasks, our diagnostic tasks reveal their surprising deficiency (near-random performance) in action knowledge, suggesting that current models rely on object recognition abilities as a shortcut for action understanding. To remedy this, we propose a novel framework, **Paxion**, along with a new **Discriminative Video Dynamics Modeling (DVDM)** objective. The Paxion framework utilizes a **Knowledge Patcher** network to encode new action knowledge and a **Knowledge Fuser** component to integrate the Patcher into frozen VidLMs without compromising their existing capabilities. Due to limitations of the widely-used Video-Text Contrastive (VTC) loss for learning action knowledge, we introduce the DVDM objective to train the Knowledge Patcher. DVDM forces the model to encode the correlation between the action text and the correct ordering of video frames. Our extensive analyses show that Paxion and DVDM together effectively fill the gap in action knowledge understanding (~50% → 80%), while maintaining or improving performance on a wide spectrum of both object- and action-centric downstream tasks.

PAXION: Patching Action Knowledge in Video-Language Foundation Models

Abstract

Action knowledge involves the understanding of textual, visual, and temporal aspects of actions. We introduce the Action Dynamics Benchmark (ActionBench) containing two carefully designed probing tasks: Action Antonym and Video Reversal, which targets multimodal alignment capabilities and temporal understanding skills of the model, respectively. Despite recent video-language models' (VidLM) impressive performance on various benchmark tasks, our diagnostic tasks reveal their surprising deficiency (near-random performance) in action knowledge, suggesting that current models rely on object recognition abilities as a shortcut for action understanding. To remedy this, we propose a novel framework, PAXION , along with a new Discriminative Video Dynamics Modeling (DVDM) objective. The PAXION framework utilizes a Knowledge Patcher network to encode new action knowledge and a Knowledge Fuser component to integrate the Patcher into frozen VidLMs without compromising their existing capabilities. Due to limitations of the widely-used Video-Text Contrastive (VTC) loss for learning action knowledge, we introduce the DVDM objective to train the Knowledge Patcher. DVDM forces the model to encode the correlation between the action text and the correct ordering of video frames. Our extensive analyses show that PAXION and DVDM together effectively fill the gap in action knowledge understanding (~50% → 80%), while maintaining or improving performance on a wide spectrum of both object- and action-centric downstream tasks. The code and data will be made publicly available for research purposes at https://github.com/MikeWangWZHL/Paxion.git .

1 Introduction

Recent video-language models (VidLMs) [30, 25, 55, 35, 57, 52] have shown impressive performance on a wide range of video-language tasks. However, such multimodal models are not without deficiencies: [24] points out that many popular video-language benchmarks [56, 3, 16] can be solved by looking at a single frame, and [59] shows that vision-language models struggle to understand compositional and order relations in images, treating images as bags of objects. Such limitations suggest that models' understanding of actions , which may require several frames and comprehension of object relationships, may be lacking.

To test this hypothesis, we first define action knowledge as an understanding of the cause and effect of actions in textual, visual, and temporal dimensions. To quantify a model's action knowledge, we introduce the Action Dynamics Benchmark (ActionBench) . ActionBench contains two probing tasks: distinguishing between (1) a video's caption and the caption with its action verbs replaced by their antonyms; (2) the original and reversed videos. The benchmark also includes a baseline task for controlling the undesired impact from domain mismatch and investigating potential bias towards objects. The baseline task requires the model to differentiate between the original video captions

,

Frozen

VidLM

ActionBench

Data

Integrating with backbone

Learning action knowledge

Action-centric

Loss

action-centric

representation

backbone representation

❄

🔥

🔥

Probing Tasks

and altered versions with randomly replaced objects. We find that state-of-the-art video-language foundation models [52, 57, 24] exhibit near-random performance on our action-oriented probing tasks while excelling on the object-oriented baseline task (Figure 2). This shows that VidLMs lack action knowledge and suggests that their impressive performance on other benchmarks may be attributed to their object recognition ability instead of action understanding.

To address this shortcoming, we propose a novel framework, PAXION (Patching Actions), to patch existing VidLMs with action knowledge without compromising their general vision-language (VL) capabilities. PAXION comprises two main components, the Knowledge Patcher and the Knowledge Fuser. The Knowledge Patcher (KP) is a Perceiver-based [21, 20] lightweight module attached to a frozen VidLM backbone used to augment the VidLM with action-aware representations. Through our preliminary experiments, one major challenge for patching action knowledge is that the widely-used Video-Text Contrastive (VTC) objective [43, 55, 29, 26, 28] is insufficient, which echoes the findings of related work [7, 24, 59]. Hence, inspired by dynamics modeling in robotics and reinforcement learning [1, 4, 15, 23, 39], we introduce the Discriminative Video Dynamics Modeling (DVDM) objective that forces the model to learn the correlation between an action's textual signifier, the action text (e.g. the word 'falling'), and the action's visual depiction (e.g. a clip of a falling book). DVDM includes two new losses, Video-Action Contrastive (VAC) and Action-Temporal Matching (ATM) , which are compatible with VTC without additional parameters. Specifically, we formulate discriminative tasks using action antonyms and reversed videos, with special emphasis on learning from data instances with salient state changes. We demonstrate that the synergy between the Knowledge Patcher and DVDM leads to a dramatic improvement on our ActionBench tasks.

Next, we investigate whether our Knowledge Patcher, which is specialized for action understanding, can be integrated into existing VidLMs for downstream tasks that require both action and object knowledge. To this end, we introduce the Knowledge Fuser (KF) component of PAXION which fuses the action-centric representation from the Knowledge Patcher with the object-centric representation from the frozen backbone using cross-attention. We show that the fused representation from PAXION improves both object and action understanding on a wide spectrum of tasks, including Video-Text Retrieval (SSv2-label [13, 24]), Video-to-Action Retrieval (SSv2-template [24], Temporal [45]), and Causal-Temporal Video Question Answering (NExT-QA [54]). Moreover, our analysis shows that the Knowledge Fuser is essential to maintain a balance between the models' object-related understanding and improving performance on downtream action and temporal-oriented tasks.

We also test the robustness of PAXION by considering a zero-shot cross-domain transfer setting on the Moments-in-Time [38] and Kinetics [22] datasets. We find that the Knowledge Fuser is critical for increasing robustness to domain shifts and that positive transfer to unseen domains can be achieved by further ensembling PAXION with the backbone model.

To the best of our knowledge, this is the first work to systematically evaluate action knowledge and patch it into video-language foundation models. Our main contributions are threefold:

We introduce the Action Dynamics Benchmark (§ 2), which probes action understanding capabilities in video-language models. We evaluate three state-of-the-art video-language foundation models and conclude that they lack a basic grasp of action knowledge.

We propose a novel learning framework called PAXION, which patches the missing action knowledge into frozen video-language foundation models without hurting their gen-

VidLM

"Book

falling

like a rock"

"Book

rising

like a rock"

✔

✘

"Book

falling

like a rock"

Original Action Text

Action Antonym Text

Probing Task: Action Antonym (AA)

Probing Task: Video Reversal (VR)

VidLM

Result

23.2%

Original Action Text

76.8%

Baseline Task: Object Replacement (OR)

GT

"

Book

falling like a rock"

"

Cellphone

falling like a rock"

Original Action Text

Object Replaced Text

VidLM

VidLM

Original

Video

Original Video

Reversed Video

Original

Video

✔

✘

VidLM

Result

GT

✔

✘

VidLM

Result

GT

49.9%

50.1%

77.9%

22.1%

ActionBench-Ego4d

ActionBench-SSv2

100

100

94.2

94.0

90.5

82.4

83.3

80

76.8

76.3

80

69.8

70.2

(%)

Accurarcy

60

58.8

46.6--

49.3

55.0

==47:0

50.1

51.750.6

Accurarcy

60

51.8

48.3

53.6

48.949.6

57.0

50.5

40

40

RandomGuess

Random Guess

ActionAntonym

ActionAntonym

20

20

ObjectReplacement

ObjectReplacement

0

0

InternVideo

CLIP-ViP

Singularity

Avg

InternVideo

CLIP-ViP

Singularity

Avg

eral vision-language capabilities. The key components of PAXION include a Perceiver-based Knowledge Patcher (§ 3) and a cross-attention-based Knowledge Fuser (§ 4).

We propose the DVDM objective (§ 3.1), an improvement over the widely-used VTC loss, which forces the model to encode the correlation between the action text and the correct ordering of video frames. Extensive experiments show that PAXION with DVDM improves the joint understanding of objects and actions while being robust to domain shift.

2 Action Dynamics Benchmark (ActionBench): Do Video-Language Foundation Models Understand Action Knowledge?

To investigate the presence of action knowledge in state-of-the-art video-language foundation models, we propose the Action Dynamics Benchmark (ActionBench) . ActionBench comprises the Action Antonym (AA) and Video Reversal (VR) probing tasks, along with the Object Replacement (OR) baseline task. The probing tasks evaluate the multimodal and temporal correlations between an action text and a video . The baseline task controls for the potential impact of domain mismatch.

We construct this benchmark by leveraging two existing open-domain video-language datasets, Ego4D [14] and Something-Something v2 (SSv2) [13], which provide fine-grained action annotations for each video clip. Compared to a previous verb understanding probing benchmark [42] based on MSRVTT [56] and LSMDC [44], ActionBench is more action-oriented, larger in scale, and contains both ego-centric and third-person videos. Detailed statistics can be found in Appendix B. An illustration of each ActionBench task can be found in Figure 2.

Probing Task: Action Antonym (AA). The Action Antonym task probes the multimodal alignment of the action text and the video representation. We formulate AA as a binary classification task that involves distinguishing the original text annotation from its altered version with the action replaced by its antonym, given the corresponding video clip. For example, if the original text is 'Book falling like a rock' , the action antonym text would be 'Book rising like a rock' . We leverage the WordNet [36] database and manually constructed mappings to automatically construct the antonym texts (details in Appendix B).

Probing Task: Video Reversal (VR). The Video Reversal task probes the temporal understanding of actions. We formulate VR as a binary classification task, where given a video-text pair with at least one action and a reversed version of the video, the goal is to distinguish the original video from the reversed one. Achieving non-trivial performance on the Video Reversal task requires the model to understand the temporal sequence implied by the action. The Video Reversal task also evaluates VLMs' abilities to identify violations of physical knowledge, as some clips defy expectation when reversed (e.g. a falling book becomes one which rises without any discernible cause).

Baseline Task: Object Replacement (OR). Object Replacement is a binary classification task that requires the model to distinguish between the original text annotation and an altered version with objects tokens randomly replaced by other object tokens in the dataset. The Object Replacement task allows us to understand: (1) whether current VidLMs rely on object recognition as a 'shortcut' for video-text matching (i.e., if they have an object-biased representation), and (2) whether poor performance on Action Antonym can be attributed to domain mismatch (i.e., not being trained on Ego4D or SSv2) instead of a lack of action knowledge.

2.1 Evaluating Video-Language Models on ActionBench

We evaluate three recent video-language foundation models, InternVideo [52], CLIP-ViP [57] and Singularity-temporal [24] 1 , on ActionBench. Despite their impressive improvements on videolanguage benchmarks, these models struggle to achieve non-trivial performance on Action Antonym and Video Reversal, as depicted in Figure 2. The fact that they achieve significantly better performance on the Object Replacement task indicates a strong bias towards objects over actions, and affirms that the poor performance on AA is not solely a result of domain mismatch. The near-random performance on the VR task indicates a lack of basic temporal reasoning and physical knowledge.

These observations align with previous approaches [17, 42, 59, 37] which show similar limitations in image-language models [43, 28] and earlier video-language models [11, 35]. We find that high performance on video-language benchmarks does not necessarily equate to a stronger understanding of action knowledge.

3 Patching Action Knowledge in Frozen Video-Language Models

In § 2, we showed that current VidLMs exhibit limitations in their understanding of action knowledge, a crucial component for developing a comprehensive understanding of the external world. This raises the important question: Can we enhance existing VidLMs with this missing knowledge without hurting their general video-language capabilities?

To this end, we propose a novel learning framework, PAXION, which comprises two main components: the Knowledge Patcher (KP) (§ 3) and the Knowledge Fuser (KF) (§ 4). An overview of the PAXION framework can be found in Figure 1. Analogous to releasing patches to fix bugs in published software, the Knowledge Patcher is a Perceiver-based [21, 20] lightweight module attached to a frozen VidLM for steering the VidLM towards action-centric representations. As the widely used videolanguage contrastive (VTC) objective is insufficient for learning action knowledge, we introduce

Video-Text

Contrastive (VTC)

reversed

videos

original

videos

"Book

falling

..."

"Book

rising

..."

Action-Temporal

Matching (ATM)

Adding action antonym text as hard negs

Video-Action

Contrastive (VAC)

Adding reversed videos as hard negs

original text

antonym text

Discriminative

Video

Dynamic

Modeling

(DVDM)

Aligning original text and video

Perceiver

"...

falling

..."

"...

rising

..."

original text

antonym text

original

videos

reversed

videos

VTC

VAC

ATM

OR

OR

key    value

Frozen

VidLM

query

❄

🔥

Paxion: Knowledge Patcher

Losses

Data

backbone

text feature

backbone

video

feature

learnable

latents

patched

features

+ Knowledge

Patcher

trained with

VTC+DVDM

ActionBench-SSv2(backbone)

100

94.2

94.0

83.3

80

70.2

60

51.8

48.3-

53.6

48949.6

40

RandomGuess

ActionAntonym

20

ObjectReplacement

0

InternVideo

CLIP-ViP

Singularity

ActionBench-SSv2(withPatcher)

100

95.0

90.7

87.4

89.3

85.1

82.4

89.9

80

66.0

68.8

60

40

Random Guess

ActionAntonym

20

ObjectReplacement

0

InternVideo

CLIP-ViP

Singularity

Discriminative Video Dynamics Modeling (DVDM) (§ 3.1) objectives that force the model to encode the correlation between the actual action text (e.g., 'falling') and the correct sequence of visual state-changes (i.e., video frames).

Knowledge Patching with Perceivers. Inspired by recent work [2, 27] leveraging Perceivers [21, 20] to extract language-related visual features, we use Perceivers to extract knowledge-specific features. As shown in Figure 3 Knowledge Patcher , we use a lightweight Perceiver which performs cross-attention between a sequence of lower-dimensional, learnable latents Q and the higherdimensional visual embedding V ∗ from a frozen, pretrained VidLM backbone. To further investigate the viability of Perceivers as an alternative to Transformers [49], we include another variant of the KP where we replace the Perceiver with a standard Transformer Encoder. Table 1 shows that the

Perceiver-based KP achieves competitive or better performance compared to the Transformer variant while being 2-3 times smaller in scale. Architecture details of the KPs can be found in Appendix D.1.

Video-Text Contrastive (VTC) is insufficient for learning action knowledge. We initially train both variants of the Knowledge Patcher on the training set of ActionBench with only the Video-Text Contrastive (VTC) loss. VTC loss aligns the visual representation V from the KP with the pooled textual representation t ∗ from the frozen backbone. Results in Table 1 show that training with the VTC loss alone provides marginal to no improvements on Action Antonym (AA) and Video Reversal (VR), particularly on smaller backbone models. This suggests the need for new training objectives (§ 3.1) for learning action knowledge.

3.1 Learning Action Knowledge with Discriminative Video Dynamics Modeling

To address the limitation of the VTC loss in learning action knowledge, we propose two new losses that draw inspiration from dynamics modeling in Robotics and Reinforcement Learning [1, 4, 15, 23, 39]. Specifically, in a typical Markov Decision Process (MDP) setup, forward dynamics modeling aims to predict the next world state ˆ x t +1 given the current world state x t and the action u t . Inverse dynamics modeling aims to predict the current action ˆ u t given the current and next world state x t , x t +1 . Given video frame as a representation of the world states, existing work usually formulates forward dynamics modeling as a generative task [1, 39, 15], directly reconstructing the pixels or the latent embedding of the next frame. For inverse dynamics modeling, the action class is usually predicted using a dedicated classification layer [4, 23, 1]. However, our preliminary experiments show that the existing formulation cannot be directly applied in our setting due to the following unique challenges : (1) Real world videos are much more complex than videos in a lab setting, with constantly changing backgrounds and moving camera angles, causing a large portion of visual features to be unrelated to the main objects and actions. Furthermore, without additional annotation, it is difficult to identify the frames corresponding to the 'current' and 'next' states, as actions may be continuous (e.g., "walking") or repetitive (e.g., 'doing push-ups') within a video. Thus, the training signal from a regression loss becomes extremely noisy. (2) Unlike previous work that has a small fixed number of action classes, we model actions as natural language phrases, making direct classification inapplicable.

To address these unique challenges, we propose a novel 'relaxed' formulation of dynamics modeling, dubbed Discriminative Video Dynamics Modeling (DVDM) , which contains two losses: Video-Action Contrastive (VAC) and Action-Temporal Matching (ATM) . Both VAC and ATM can be directly incorporated into the Video-Text Contrastive (VTC) loss without any additional parameters. As illustrated in Figure 3 Losses , the V AC loss aims to encourage the model to learn the correlation between the visual observations and the actual actions. We formulate the V AC loss as adding action antonym texts as hard negatives. The ATM loss encourages the model to consider the temporal ordering of the visual observations (video frames). Instead of directly generating the next state frames, we formulate ATM as a discriminative task similar to Video Reversal in § 2, where the model distinguishes reversed videos from the original videos, alleviating the need for explicit state annotations. In order to make sure that the reversed videos are indeed distinguishable from the original ones, we further introduce a method (Appendix C) for identifying videos with salient state-changes by leveraging image-language foundation models [28]. The idea is to measure the frame-text and frame-frame similarity between the first and second half of a video. We compute ATM loss only on the videos that have salient state-changes between frames. Experimental results, as shown in Table 1 and Figure 3, indicate that adding the DVDM objectives significantly improves the performance on both probing tasks , suggesting that the resulting representation from the Knowledge Patcher demonstrates a stronger understanding of action knowledge.

4 Leveraging Patched Action Knowledge for Downstream Tasks

In § 3, we showed that the Knowledge Patcher (KP) and DVDM objectives together effectively learn action knowledge-specific representations. However, these representations are highly specialized to action understanding, which may not be optimal for general downstream tasks that require both object and action understanding. Thus, the remaining challenge is to retain the general VL capabilities of the backbone while leveraging the newly learned action knowledge.

One naive idea is to simply use the backbone embeddings whenever the task is less action-centric. However, it is difficult to decide when to use the backbone without prior knowledge of a given task. Further, using the backbone embeddings alone gives up the patched action knowledge that can be essential for certain downstream tasks, such as action recognition. In this section, we demonstrate that we can get the best of both worlds by fusing the action-centric representation from the Knowledge Patcher with the object-centric representation from the frozen backbone.

For this we introduce the second component of PAXION, the Knowledge Fuser (KF) , illustrated in Figure 4 . The KF

Cross-Attn

VTC

key    value

query

🔥

Paxion: Knowledge Fuser

backbone

text

feature

Knowledge

Patcher

features

pooled

backbone

video

feature

fused

video

feature

takes the pooled visual feature ( v ∗ ) from the frozen VL backbone as the input query, and performs cross-attention with the extracted visual tokens ( V ) from the Knowledge Patcher.

4.1 Experimental Setup

To evaluate the model's ability to retain general visual-linguistic capabilities while leveraging newly learned action knowledge, we consider a spectrum of video-language tasks with different emphases on object and action understanding. Specifically, we consider Video-Text Retrieval (SSv2-label [24]), which is object-centric and biased towards static appearances [59, 24]; Causal-Temporal VQA (NExT-QA [54]), which requires joint understanding of static objects and dynamic events; and Videoto-Action Retrieval (SSv2-template [24], Temporal-SSv2 [45]), which is highly action-centric and temporal-intensive. A task is considered to be temporal-intensive if it cannot be solved without correct temporal information [45], e.g., reversed or shuffled frames. For example, as illustrated in Figure 5, the Video-to-Action Retrieval task obscures object names in the text, making it impossible to align text with a video based solely on objects. Moreover, it is impossible to distinguish 'approaching' and 'moving away' without considering the temporal ordering of the frames.

For PAXION , we finetune the Knowledge Fuser jointly with the Knowledge Patcher on downstream tasks using VTC loss. By default, the KP in PAXION is trained with VTC and DVDM losses (§ 3.1). We include the baselines KP-Transformer FT [VTC] and KP-Perceiver FT [VTC] , which are both obtained by continuing to finetune the VTC-only KPs from Table 1 on downstream tasks. Additionally, we compare PAXION with Side-Tuning [61], a Parameter-Efficient Finetuning (PEFT) method that could serve as an alternative to the KF. For the Side-Tuning variant, we initialize the "side-model" using the same Knowledge Patcher as in PAXION and do alpha blending with the frozen backbone. Implementation and configuration details for each method and task can be found in Appendix D. The results are shown in Tables 2 and 3.

4.2 Analysis

PAXION improves joint understanding of objects and actions. Tables 2 and 3 show that PAXION outperforms both the Backbone and the VTC-only baselines (KP-*) . This indicates that PAXION not only retains the original VL capabilities of the backbone, but also fills in the gap of the missing action knowledge by fusing the original representations with the patched ones. We corroborate this finding by observing more significant improvements on action-centric and temporal-intensive tasks,

Method [Patcher Training Loss]

NExT-QA

Original

ATP-hard [7]

C

T

D

all

C

T

all

InternVideo Backbone

43.3

38.6

52.5

43.2

27.0

27.3

27.1

KP-Transformer FT [VTC]

46.1

45.0

61.3

48.1

32.5

33.6

33.0

KP-Perceiver FT [VTC]

46.0

46.0

58.9

48.0

30.1

31.6

30.7

Side-Tuning [61] [VTC+DVDM]

54.9

52.0

69.8

56.3

37.4

36.0

36.8

PAXION

[VTC+DVDM]

56.0

53.0

68.5

57.0

38.8

38.1

38.5

"Approaching something with your camera"

Video-to-Action Retrieval (Temporal-SSv2)

"Moving away from something with your camera"

45.1%

46.8%

60.0%

23.5%

✔

✘

Paxion

VTC-Finetune

Question:

"what did the

baby do after he

approached near

the camera?"

Causal-Temporal VQA (NExT-QA)

A. "raised his hand to take the camera"

16.8%

Paxion

VTC-Finetune

B. "bored"

C. "turn back to the toy"

D. "move his legs"

E. "suck his thumb"

✔

✘

✘

✘

✘

18.2%

28.2%

21.7%

15.1%

29.9%

21.3%

20.2%

18.2%

10.4%

such as Temporal-SSv2 (+20% @R1), compared to object-centric tasks, such as SSv2-label (+11% @R1) 2 . The decomposed results on NExT-QA (Table 3) show that PAXION helps more on the Causal ( C ) and Temporal ( T ) types of questions, and on the harder subset (ATP-hard) where the temporal and action knowledge is emphasized.

PAXION also outperforms Side-Tuning, highlighting the effectiveness of cross-attention for deep fusion. Specifically, the Knowledge Fuser allows us to attend to all extracted visual tokens from the Knowledge Patcher instead of only blending with pooled representations as in Side-Tuning.

Qualitative analysis. Figure 5 shows two qualitative examples on Temporal-SSv2 and NExT-QA. For the Temporal-SSv2 example, we find that the finetuned Knowledge Patcher trained with only VTC fails to distinguish 'Moving away' from 'Approaching,' while PAXION trained with DVDM successfully correlates the seemingly expanding object with the action 'Approaching' . For the NExTQA example, the question asks the model to identify what happens after the action 'approached near the camera' . The VTC baseline incorrectly selects the action 'turn back to the toy, ' which happens before approaching the camera. On the other hand, PAXION successfully chooses 'raised his hand to take the camera' . This indicates a stronger understanding of both action dynamics in words such as 'approach' and the temporal ordering implied by words such as 'after' . Additional qualitative examples and analysis of failure cases can be found in Appendix E.

Disentangling the impact of Knowledge Patching and Fusing. We further investigate the disentangled impact of the Knowledge Fuser (KF) and the Knowledge Patcher (KP) with two ablation settings: (1) KP+Finetune , where instead of adding the KF, we directly finetune the KP trained with DVDM on downstream tasks; (2) KP[VTC]+KF , where we train the KP without DVDM and then add the KF upon it. The results are shown in Figure 6, where the ∆ score represents the relative difference of downstream task performance between our original PAXION (Row 5 in Tables 2 and 3) and the two ablated settings. The key observations are as follows: (1) The Knowledge Fuser contributes more to object understanding. From Figure 6 Left, we find that the KF helps most when the tasks are more object-centric, e.g., SSv2-label. On highly action-centric tasks, e.g., Temporal-SSv2, directly using the action-knowledge-patched representation is preferable to fusing with the backbone representation.

more

action-

centric

more

object-

centric

Finetune

v.s.

Fuse

VTC

v.s.

VTC+DVDM

SSv2-L R1v

+41.7

SSv2-L R1v -4.4

SSv2-L R1t

+26.1

SSv2-L R1t

+2.2

NExT-QAtest

+24.4

NExT-QAtest

+0.0

SSv2-T R1

+8.5

SSv2-T R1

+10.7

Temp-SSv2R1

-1.5

Temp-SSv2R1

+18.5

0

10

20

30

40

-5

0

5

10

15

20

Score (%)

Score (%)

(2) Patching with action knowledge contributes more to action-centric understanding. From Figure 6 Right, we find that patching with action knowledge, i.e., training with DVDM objectives, contributes to better performance on downstream tasks that are more action-centric. Importantly, this result also indicates that the improvements observed in Tables 2 and 3 do not come solely from adding the KF. However, if the task is more object-centric, such as SSv2-label, VTC training alone is sufficient.

Robustness to domain shift. Learned action knowledge should be generalizable to unseen tasks and domains. However, this goal is difficult to realize with only domain-specific datasets like SSv2[13] which contains only 174 actions. Therefore, in Appendix A we conduct experiments on zero-shot cross-domain transfer which demonstrate that the Knowledge Fuser in PAXION increases robustness to domain shift and can introduce positive transfer during zero-shot inference.

5 Related Work

Limitations of vision-language contrastive pretraining. Since CLIP [43], multimodal contrastive losses have been the major pretraining objective for almost all recent image-language [43, 28, 27, 53, 51, 58] and video-language models [55, 35, 10, 60, 12, 50, 57, 52]. Previous work [17, 59, 24, 5] has revealed the limitation of contrastive pretraining on fine-grained compositional understanding, verb understanding, and temporal reasoning. Concurrent work [37, 9] proposed mining hard negatives by rule-based heuristics or large-language models [8] to improve understanding of structured visionlanguage concepts and verbs. In this work, we focus on general action knowledge which includes verb understanding as well as action temporal understanding. Instead of directly tuning the entire backbone as in [59, 37], PAXION enables fast action knowledge patching while also achieving improved performance on both object-centric and action-centric downstream tasks. It is worth noting that the hard negative mining method proposed by [37] can be easily incorporated with our V AC loss and could potentially result in stronger results.

Parameter-efficient fine-tuning (PEFT). The recent surge in the size of large language models [6, 41, 62, 40, 48] has spurred research on parameter-efficient fine-tuning [18, 31, 61, 46, 19, 33]. Although current video-language models are smaller in scale, we aim to develop PAXION to be applicable to larger models that we anticipate will emerge in the near future. The most similar PEFT-related work to ours is Side-Tuning [61], which we compare against in § 4.1. At a high-level, unlike existing PEFT methods that optimize for specific downstream tasks, PAXION is designed to learn a specific type of knowledge that can benefit various downstream tasks (§ 4.2). Furthermore, it is unclear how to aggregate the task-specific parameters, such as those in adapters [18] or low-rank layers [19], to perform multiple tasks. The versatility of PAXION allows for its use in learning various types of knowledge, each with its own Knowledge Patcher. Subsequently, the patched knowledgespecific representations can be fused together using one Knowledge Fuser. This work serves a proof-of-concept where we focus on action knowledge. We leave the exploration of other types of knowledge and a more comprehensive comparison with PEFT methods as future work.

6 Conclusions and Future Work

In this work we propose the ActionBench benchmark for evaluating models' understanding of action knowledge, and reveal a major deficiency in state-of-the-art video-language foundation models in this area. We then propose PAXION to patch in such action knowledge without compromising models' existing capabilities. We show that PAXION significantly improves the model's action understanding while achieving competitive or superior performance on downstream tasks. One limitation of this work is that we only experimented with patching one type of knowledge. We intend to address this in future work, where we plan to expand PAXION to patch broader aspects of physical knowledge such as object affordances and mental simulation, and to explore fusion with multiple learned Knowledge Patchers.

References

Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems , 29, 2016. 2, 6

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. ArXiv preprint , abs/2204.14198, 2022. 5

Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision , pages 5803-5812, 2017. 1

Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038 , 2020. 2, 6

Piyush Bagad, Makarand Tapaswi, and Cees G. M. Snoek. Test of time: Instilling video-language models with a sense of time. ArXiv , abs/2301.02074, 2023. 9

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. 9

Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the" video" in video-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2917-2927, 2022. 2, 8

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. 9

Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2657-2668, 2023. 9

Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097 , 2021. 9

Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16 , pages 214-229. Springer, 2020. 4

Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, Hao Zhang, and Jinwei Yuan. Clip2tv: An empirical study on transformer-based methods for video-text retrieval. arXiv preprint arXiv:2111.05610 , 1(2):6, 2021. 9

Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision , pages 5842-5850, 2017. 2, 4, 9, 14

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18995-19012, 2022. 4, 14

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning , pages 2555-2565. PMLR, 2019. 2, 6

Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A largescale video benchmark for human activity understanding. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015 , pages 961-970. IEEE Computer Society, 2015. 1

Lisa Anne Hendricks and Aida Nematzadeh. Probing image-language transformers for verb understanding. arXiv preprint arXiv:2106.09141 , 2021. 4, 9

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pages 2790-2799. PMLR, 2019. 9

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021. 9

Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795 , 2021. 2, 4, 5

Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning , pages 46514664. PMLR, 2021. 2, 4, 5

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017. 2

Ashish Kumar, Saurabh Gupta, and Jitendra Malik. Learning navigation subroutines from egocentric videos. In Conference on Robot Learning , pages 617-626. PMLR, 2020. 2, 6

Jie Lei, Tamara L Berg, and Mohit Bansal. Revealing single frame bias for video-and-language learning. arXiv preprint arXiv:2206.03428 , 2022. 1, 2, 3, 4, 7, 9, 14, 17, 18

Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In CVPR , 2021. 1

Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven C.H. Hoi. Align and prompt: Video-and-language pre-training with entity prompts. In arxiv , 2021. 2

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023. 5, 9, 16

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ArXiv preprint , abs/2201.12086, 2022. 2, 4, 6, 9, 15

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems , 34:9694-9705, 2021. 2

Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. HERO: Hierarchical encoder for Video+Language omni-representation pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 2046-2065, Online, 2020. Association for Computational Linguistics. 1

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021. 9

Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. Advances in Neural Information Processing Systems , 35:7575-7586, 2022. 15

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems , 35:1950-1965, 2022. 9

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. 17

Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing , 508:293-304, 2022. 1, 4, 9

George A Miller. WordNet: An electronic lexical database . MIT press, 1998. 4, 15

Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. arXiv preprint arXiv:2304.06708 , 2023. 4, 9

Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE transactions on pattern analysis and machine intelligence , 42(2):502-508, 2019. 2, 14, 17, 18

Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. Advances in neural information processing systems , 28, 2015. 2, 6

OpenAI. Gpt-4 technical report. https://arxiv.org/abs/2303.08774 , 2023. 9

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. ArXiv preprint , abs/2203.02155, 2022. 9

Jae Sung Park, Sheng Shen, Ali Farhadi, Trevor Darrell, Yejin Choi, and Anna Rohrbach. Exposing the limits of video-text models through contrast sets. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3574-3586, 2022. 4

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 2021. 2, 4, 9, 14

Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. International Journal of Computer Vision , 123:94-120, 2017. 4

Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time can tell: Discovering temporal data for temporal modeling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 535-544, 2021. 2, 7, 8, 14, 17, 18

Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. arXiv preprint arXiv:2206.06522 , 2022. 9

Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128 , 2023. 18

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. 9

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 5

Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for image-language and video-language tasks. arXiv preprint arXiv:2209.07526 , 2022. 9

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022. 9

Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191 , 2022. 1, 2, 3, 4, 9, 16

Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. ArXiv preprint , abs/2108.10904, 2021. 9

Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9777-9786, 2021. 2, 7, 8, 17

Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6787-6800, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. 1, 2, 9

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 5288-5296. IEEE Computer Society, 2016. 1, 4

Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clipvip: Adapting pre-trained image-text model to video-language representation alignment. arXiv preprint arXiv:2209.06430 , 2022. 1, 2, 3, 4, 9

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. ArXiv preprint , abs/2111.11432, 2021. 9

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations , 2023. 1, 2, 4, 7, 9

Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems , 34, 2021. 9

Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 698-714. Springer, 2020. 7, 8, 9, 14, 17

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. ArXiv preprint , abs/2205.01068, 2022. 9

A Robustness to Domain Shift: Zero-shot Cross-Domain Transfer

Humans acquire action knowledge through multisensory interactions, and have the remarkable ability to generalize to new objects and scenarios. Similarly, our ultimate goal is to learn the underlying rules of action knowledge that is generalizable to unseen domains. However, it is highly challenging when we are given only domain-specific datasets. For instance, the SSv2 dataset [13] only has 174 action classes, which is insufficient to capture the full range of open-world actions. The Ego4d dataset is limited to ego-centric videos, making it difficult to generalize to other types of videos. Training on such domain-specific data can easily lead to overfitting to spurious features and introduce catastrophic forgetting of tasks from other domains. In this section, we further explore whether PAXION is robust to domain shift and whether the learned action knowledge can bring positive transfer to action-centric tasks on unseen domains .

We consider a zero-shot cross-domain transfer setting where we directly apply the models trained on SSv2-label [24] to unseen domains. We consider two zero-shot action classification tasks based on Moments-In-Time [38] 3 and Temporal-Kinetic [45]. Moments-In-Time contains 305 action classes with diverse types of videos that are distinct from SSv2, including movie clips, stock footages, and cartoons. Temporal-Kinetic contains 32 manually selected action classes from Kinetic-400, with a special focus on temporal reasoning. We directly use the action labels (e.g., 'bouncing' and 'kicking' ), as the text candidates for the zero-shot classification [43], which introduces additional domain shifts in terms of text distribution compared with the annotations in SSv2-label (e.g., 'book falling like a rock' ).

Fusing with the backbone improves robustness to domain shift. Table 4 shows the zero-shot action classification accuracy and the relative difference ∆(%) compared with the frozen backbone. We find that adding the Knowledge Fuser effectively increases robustness to domain shift, as reflected by a smaller negative ∆ . The Side-tuning also demonstrate similar benefit via alpha blending between the Knowledge Patcher and the backbone.

Positive transfer can be achieved by ensembling the Knowledge Fuser (KF) with the backbone. We further propose a simple inference trick, Backbone Ensemble , which combines the output probability from the KF and the backbone model through addition. Specifically, the final prediction of the action class index c ∈ 0 , 1 , ..., C is computed as c = arg max i ∈ 0 , 1 ,...,C ( p a ( i = c ) + p b ( i = c )) , where C is the number of classes, p a and p b are the predicted probability distribution from the KF and the backbone respectively. We obtain the final prediction by ranking the combined probability of the action text candidates. Our experiments show that this simple inference technique can effectively enhance zero-shot performance and achieve positive transfer on unseen domains.

B Details of Action Dynamics Benchmark (ActionBench)

We construct ActionBench based on two existing video-language datasets with fine-grained action text annotation, Ego4d [14] and SSv2 [13]. To automatically generate the antonym text for the Action

Antonym task, we leverage WordNet [36] 4 to find antonyms for verb text tokens. Additionally, we construct an additional verb-to-antonym mapping by leveraging ChatGPT 5 and manual curation, since the WordNet database does not cover all verbs in the action taxonomy of the dataset. Furthermore, to ensure that the action antonym indeed forms a negative video-text pair with the original video, we exclude verbs that do not have a semantically reasonable antonym, such as 'use' and 'look'. For Ego4d, we consider a subset of EgoClip [32] annotations, for SSv2 we consider the entire dataset. The final statistics of the training and evaluation splits can be found in Table 5. For SSv2, since the test set does not provide label annotation, i.e., annotation with filled object names, we report scores on the validation set. For Ego4d, we evaluate on the test set. For results in Table 1, we train the Knowledge Patcher variants for one epoch on the training sets and report the accuracy on the evaluation sets. We downsampled the videos into 224x224 in scale with a frame rate of 8 fps for both training and evaluation. For human evaluation, we randomly sample 50 instances for the Action Antonym and the Object Replacement task, and another 50 instances for the Video Reversal task. The human evaluation is done by the authors.

C Identifying State-change Salient Videos for Action-Temporal Matching (ATM)

As detailed in § 3.1, we formulate the Action-Temporal Matching (ATM) loss as distinguishing reversed video from the original one given an action text. ATM requires the model to learn the correlation between the correct temporal ordering of the visual observations and the corresponding actions. However, some actions, such as 'wiping' and 'holding', are repetitive or continuous and may not result in visible state-changes across the frames in the video clip. This can introduce additional noise for the ATM loss when the reversed video is indistinguishable from the original one. To address this issue, we propose two metrics to identify state-change salient videos by leveraging imagelanguage foundation models. We use pretrained BLIP [28] to compute (1) frame-text semantic change δ vt , which indicates how the frame-text alignment changes across the first half and second half of the video; (2) frame-frame similarity θ vv , which indicates how different the frames from the first half and second half of the video are.

where N is the total number of sampled frames 6 , v and t are the frame image embedding and the text embedding from pretrained BLIP encoders, S denotes cosine similarity.

Intuitively, if we observe a large frame-text semantic change ( δ vt ) and a small frame-frame similarity ( θ vv ), we could expect to see salient state-changes between the first half and the second half frames. We empirically set a threshold for δ vt and θ vv . During training, we only compute ATM loss on videos that satisfy δ vt > 0 . 003 and θ vv < 0 . 95 . The metrics are computed off-line thus do not bring computational overhead during training. Figure 7 shows an example of the videos that are kept and skipped based on the computed metrics.

"spreading margarine onto bread"

"holding bulb"

✔

Keep

✘

Skip

026

S

0.004

A

0.979

Cross-Attn

FFN

Cross-Attn

FFN

key value

query

key

value

query

a. Knowledge Patcher

(Perceiver)

VTC

VAC

ATM

losses

VTC

Perceiver

Self-Attn

FFN

c. Knowledge Patcher

(Transformer)

VTC

Transformer

Projection + Pooling

d. Side-Tuning

Frozen

VidLM

Perceiver

🔥

❄

Pooling

Base-model

Side-model

Paxion (Ours)

Baseline

Baseline

D Implementation Details

D.1 Architecture Details.

Figure 8 shows detailed architecture of the Knowledge Patcher and Knowledge Fuser in our PAXION framework, as well as the baseline variants being compared in Tables 1, 2 and 3.

Knowledge Patcher (Perceiver). The Perceiver-based Knowledge Patcher contains a single crossattention layer and a two-layer feedforward network. The Perceiver module performs cross-attention between a sequence of learnable latent queries Q ∈ R l,d and the raw visual embeddings V ∗ ∈ R P,D from the frozen backbone, where P denotes the visual token length and D represents the hidden dimension of the visual backbone. Since the user-defined sequence length l and hidden dimension d of the learnable latent queries are typically much smaller than P and D from the backbone, the Perceiver module serves as an information bottleneck that extracts knowledge-specific features from the raw visual features. For instance, in the case of InternVideo [52] backbone, we set l = 16 , d = 768 which is much smaller than P = 1576 , D = 1024 for each video clip with 8 sampled frames. Similar to BLIP-2 [27], when computing the similarity between the visual tokens V ∈ R l,d from the Knowledge Patcher and the single textual feature vector t ∗ ∈ R d , we first compute the pairwise similarity between each visual token and the text feature vector, and then take a maximum across all visual tokens as the final video-text similarity. The results in Table 1 demonstrate the Perceiver-based Knowledge Patcher achieves competitive or better performance compared to the Transformer variant while being 2-3 times smaller. Additionally, we measure the computation overhead of the two variants, and find that the Perceiver variant requires 10 times fewer multiply-add operations than the Transformer variant. This further demonstrate that Perceivers can serve as effective and efficient extractors for knowledge-specific features.

Knowledge Patcher (Transformer). The Transformer variant of the Knowledge Patcher is a standard Transformer Encoder which contains a self-attention layer and a feedforward layer. The Transformer Encoder performs self-attention on the raw visual embeddings V ∗ ∈ R P,D from the frozen backbone and output an updated visual embedding V ∈ R P,D . To obtain video-text similarity, we first project the visual embeddings into the same dimension as the textual feature vector t ∗ ∈ R d and then do mean pooling before computing dot product.

Knowledge Fuser. The Knowledge Fuser has the same architecture as the Knowledge Patcher which contains a single cross-attention layer and a two-layer feedforward network. In this case, we use the pooled visual feature from the backbone v ∗ ∈ R d to provide query and the Knowledge Patcher output V ∈ R P,D to provide key and value for the cross-attention. The intuition is to obtain a balanced representation for general downstream tasks by fusing the action-centric KP representation ( V ) with the object-centric backbone representation.

Side-Tuning. As an alternative to the Knowledge Fuser, we consider Side-Tuning [61] for further integrating the Knowledge Patcher with the backbone. Side-Tuning contains a base-model and a side-model , where the base-model is pretrained and frozen and the side-model is trainable. In our setting, we treat the backbone as the base-model and initialize the side-model using the trained Knowledge Patcher. We then side-tune the Knowledge Patcher along with the backbone using alpha blending. Specifically, the final fused visual feature vf is obtained by vf = α ( v ∗ ) + (1 -α ) v , where v ∗ is the mean-pooled backbone visual feature, and the v is the mean-pooled Knowledge Patcher feature. And α = Sigmoid ( a ) ∈ [0 , 1] , where a a learnable scalar.

D.2 Knowledge Patcher Training.

We use two Nvidia Tesla V100 (16GB) GPUs for all experiments. For the Knowledge Patcher variants in Table 1, we train them on the training set of the datasets in the ActionBench for one epoch with either VTC loss only or VTC + DVDM (VAC + ATM) loss. We use AdamW [34] optimizer with a learning rate of 1e-5 and a weight decay of 0.05. For the transformer variant, we use a batch size of 8 per GPU. For the Perceiver variant, we are able to increase the batch size to 32 per GPU due to the reduced computation complexity.

D.3 Downstream Task Training.

Tables 6 and 7 shows detailed configurations for downstream task training with methods described in Tables 2 and 3, and Figure 6.

As shown in Table 7, the finetuning dataset for SSv2-label is identical to the SSv2 action knowledge patching dataset where the annotations are filled templates, such as 'Book falling like a rock'. The SSv2-template dataset, on the other hand, contains the object-obscured version of the original SSv2 annotations such as 'Something falling like a rock'. For the Video-to-Action Retrieval tasks, we consider two different subsets from the SSv2 validation set with the object-obfuscated annotations: SSv2-template [24] and Temporal-SSv2 [45]. SSv2-template contains all 174 action classes while Temporal-SSv2 contains 18 manually selected action classes that require more temporally-demanding distinctions, and cannot be distinguished using shuffled frames, such as 'Approaching' and 'Moving away'. In order to investigate the impact of the action knowledge patching, we do not finetune a dedicated model for the 18 action classes for Temporal-SSv2, but instead use the model trained on SSv2-template to directly evaluate on Temporal-SSv2. Therefore, when observed larger improvements on Temporal-SSv2, we can draw the conclusion that patching with action knowledge contributes more to action-centric tasks (§ 4.2).

The hyperparameters, such as the learning rate, are identical to those used during Knowledge Patching training. For Video-Text Retrieval (SSv2-label) and Video-to-Action Retrieval (SSv2-template, Temporal-SSv2), the DVDM (§ 3.1) objective includes VAC and ATM, while for Causal-Temporal VQA (NExT-QA), we only use VAC. This is because the training instances in NExT-QA are not formatted as video-text pairs but instead are in the format of multiple choice QA, making it not suitable for the ATM loss. Each video corresponds to one question and five candidate answers. We apply VAC to NExT-QA by adding action antonym text for each question as hard negative candidate answers.

For the downstream tasks (in Appendix A) for zero-shot cross-domain transfer (Moments-InTime [38] and Temporal-Kinetic [45]), we use the model trained on SSv2 to perform zero-shot evaluation.

E Additional Qualitative Analysis

Figures 9 and 10 show additional qualitative examples on downstream tasks. The examples in demonstrate that PAXION improves understanding of challenging actions that require fine-grained temporal reasoning on the frames. For example, whether it is 'pretending' to do something or actually doing that, and whether an object is moving 'towards' or 'away' from the camera.

In Figure 11, we show failure cases of PAXION to discuss remaining challenges. We find that PAXION still struggle to understand negation and spatial attributes . For example, both VTCFinetune baseline and PAXION fail to distinguish 'without letting it drop down' from 'then letting it drop down' . For questions that require fine-grained spatial information of objects such as 'how many goats can be spotted' , PAXION cannot perform well. Potential solutions including incorporating the patched VidLM with a code language model to disentangle perception and reasoning similar to ViperGPT [47]. By leveraging the strong logical reasoning ability of a code language model, we can easily solve the negation and counting problems by creating code scripts with booleans and loops, and then use the VidLMs as "API calls".

Text Candidates

GT

"Moving something towards the camera"

"Moving something away from the camera"

Dataset

Temporal-

SSv2

✔

✘

34.0%

28.9%

21.5%

42.6%

Paxion

VTC-Finetune

Score | Rank

1

Score | Rank

2

2

1

"Turning something upside down"

"Pretending to turn something upside down"

Temporal-

SSv2

43.2%

46.2%

43.1%

24.9%

2

1

1

2

✔

✘

"pushing tomato so it spins"

"pretending to pick heart up"

SSv2-

label

0.6%

3.1%

4.9%

0.7%

25

1

1

14

✔

✘

"Something colliding with something and both are being deflected"

"Putting something, something and something on the table"

SSv2-

template

4.0%

15.9%

21.2%

2.8%

5

1

1

5

✔

✘

"Sprinkling something onto something"

"Something falling like a rock"

SSv2-

template

8.6%

10.6%

9.2%

1.6%

3

1

1

12

✔

✘

"approaching smart phone with your camera"

"turning the camera right while filming old mobile phone"

SSv2-

label

6.2%

22.6%

20.5%

7.6%

4

1

1

3

✔

✘

Video-to-Action Retrieval

Video-Text Retrieval (V2T)

VTC-Finetune

VTC-Finetune

VTC-Finetune

VTC-Finetune

VTC-Finetune

Paxion

Paxion

Paxion

Paxion

Paxion

Answer Candidates

GT

A. "wants to play with girl"

B. "to throw it"

Question

✔

✘

27.1%

18.2%

25.5%

25.7%

Score | Rank

1

Score | Rank

4

2

1

"why did the baby hold the

ball and moving forward?"

C. "kick off the ground"

21.8%

24.2%

2

3

19.2%

15.1%

3

4

13.7%

9.4%

5

5

D. "give ball to lady"

E. "person inside is walking"

✘

✘

✘

A. "pet its back"

B. "bends down to hug dog"

20.4%

24.3%

16.2%

63.1%

4

2

2

1

"what does the man do as

the dog stood in front of

him?"

C. "resting"

24.8%

11.7%

1

3

20.8%

6.5%

3

4

9.8%

2.6%

5

5

D. "jump over dog"

E. "walk towards the cameraman"

A. "took off goggles"

B. "performed"

27.2%

11.2%

19.3%

26.8%

1

5

2

1

"how did the man on the

most right reacted after the

man in red showed him a

hand gesture?"

C. "adjust the rein"

18.8%

19.1%

4

3

21.3%

17.5%

3

4

21.4%

17.3%

2

5

D. "excited"

E. "grab his shoulders"

✔

✘

✘

✘

✘

✔

✘

✘

✘

✘

VTC-Finetune

Paxion

VTC-Finetune

Paxion

VTC-Finetune

Paxion

Text Candidates

GT

"Lifting something up completely

without

letting it drop down"

"Lifting up one end of something,

then

letting it drop down"

Dataset

Temporal-

SSv2

✔

✘

47.2%

25.4%

39.0%

28.4%

Score | Rank

1

Score | Rank

2

1

2

"bending tube so that it deforms"

"holding soaps over tooth paste"

SSv2-

label

✔

✘

13.9%

0.02%

15.4%

0.05%

1

286

1

213

Answer Candidates

GT

A. "eight"

B. "one"

Question

✔

✘

23.3%

8.5%

17.5%

17.3%

Score | Rank

3

Score | Rank

5

4

5

"how many goats can be

spotted?"

C. "two"

27.2%

20.2%

1

3

24.8%

24.3%

2

1

16.2%

20.6%

4

2

D. "three"

E. "four"

✘

✘

✘

Video-Text Retrieval & Video-to-Action Retrieval  Failure Examples

NExT-QA  Failure Example

VTC-Finetune

Paxion

VTC-Finetune

Paxion

VTC-Finetune

Paxion

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the known issue of CLIP-like models acting similar to bag-of-words models, in which structured information (e.g. relations between objects) is not really captured, so for action recognition the information leveraged is mostly object and scene (e.g. a person besides a guitar vs a person playing a guitar). The authors propose to use some extra augmentations related to action recognition information (both on the visual side and the language side), and also add an adaptor (a perceiver/Q-former) that becomes trainable using the extra augmentations. The underlying L&V model is kept frozen, so the training is efficient and the underlying model is not degraded for other domains.\n\nSTRENGTHS:\n- clear issue, sensible yet innovative solution, good results, all in a well-written paper.\n\nWEAKNESSES:\n- Right now there is little information in the main text that points to the \"patching data\", only the appendix has this info. I think this should be made very clear in the text and in the tables. One imagines that there's domain-specific fine-tuning, but it is not a must, it could be that the calibration set comes from a mixture of sources so the model has general applicability off-the-shelf. That the setting is the former is important. Does this make sense?\n\n- The other implication is: this assumes some CLIP-like model for video, and then uses the \"knowledge patching\" trained on downstream on-domain data. But one could equally decide to go the other way and start with the image CLIP, and train an adaptor to make it work on video using downstream data. This is the approach for example of ST-adapter and AIM. These comparisons are not included in the paper.\n\n(Not important but maybe interesting) The paper (text augmentations) reminded me of the following paper: \"CVPR'23, Teaching Structured Vision & Language Concepts to Vision & Language Models\", which is not discussed.\n\nQUESTIONS:\nThe main question is how this method compares against CLIP + ST-adapter or a similar strategy to adapting CLIP to video through lightweight adapters.\n\nOpen to hearing the author's opinion about the first point (the setting)\n\nLIMITATIONS:\nNo. \nAuthors could comment on the setting (need to have downstream training data), or the standard issues with vision + language pre-training for example.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
In this paper, the authors introduce an interesting ActionBench, aiming to handle action antonym and video reversal problems. To remedy the problem in well-trained VidLM, the authors propose the DVDM objective, along with knowledge patcher and fuser. Extensive experiments demonstrate the effectiveness of the novel objective and modules.

STRENGTHS:
- Novel and internting ActionBench.
- Simple modified contrastive objective to enhance the VidLM abilities for recognizing action antonym and video reversal problems.
- Extensive ablation studies show the effectiveness.

Overall, I appreciate the paper's idea to handle the interesting ActionBench. Though it has been somehow proposed in SthSth, there isn't any work trying to handle the problem. The authors propose the smart fine-tuning method with the perceiver as a knowledge patcher and fuser. And they design a modified contrastive loss for fine-tuning VidLM.

WEAKNESSES:
In fact, the VAC and ATM losses are modified contrastive losses with different positive and negative examples. However, the authors give a complicated motivation in Section 3.1 for their DVDM, which makes the paper harder to read. I suggest to present their idea in more easy-to-follow way.

QUESTIONS:
See weakness

LIMITATIONS:
See weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript presents an Action Dynamics Benchmark (ActionBench) with three new evaluation metrics for existing video-language models. Through ActionBench, the authors find that existing video-language models essentially rely on recognizing objects to recognize actions. Hence, a parameter-efficient component named Knowledge Patcher is connected in parallel to the video-language model and is trained with Discriminative Video Dynamics Modeling to improve the action understanding ability of the model. Finally, the authors present a knowledge fuser to infuse the knowledge learned by the Knowledge Patcher into the video-language model for downstream tasks. Empirical results show that training a knowledge patcher for use is effective in improving the video-text retrieval and temporal VQA tasks.\n\nSTRENGTHS:\n- Although it is proposed before that current video models may rely on spatial information to recognize actions in videos, the manuscript is the first to present such a benchmark with proper evaluation metrics to show the problem. This can inspire further work to solve the problem in video understanding. \n\n- The Discriminative Video Dynamics Modeling is straightforward and effective in training the video-language model to be aware of the action and the temporal direction of the videos. \n\n- The approach that the manuscript presents is similar to post-pre-training, and it is shown in Table 2 and Table 3 that such a post-pre-training strategy can benefit downstream retrieval tasks. \n\n- The writing is good and the organization of the manuscript is clear.\n\nWEAKNESSES:\n- The evaluation in the downstream task is limited to video-text retrieval on SSV2 and VQA on NExT-QA, which makes it difficult to compare the presented approach with existing methods. Since the evaluation is based on InternVideo backbone, it is possible to provide some other comparisons where InternVideo has some published results. \n\n- Further on the above note, I am not quite familiar with VQA but on SSV2, the performance seems oddly low. Especially for video-to-text retrieval, which is essentially a text-driven classification for SSv2 videos, the performance has reached 61% for EVL-B/16 [1] when 8 frames are used, which also connect a side network to the pre-trained encoder. I didn't find details in both the manuscript and the supplemental material why this is the case. Could you provide an explanation for this? \n\n[1] Frozen CLIP Models are Efficient Video Learners\n\nQUESTIONS:\n- To help object-centric understanding, why don't you use the pre-trained representations from the backbone as well (e.g., concatenate or add them to the KP during fine-tuning)?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of improving video-language understanding models, which a particular emphasis on their ability to align described actions to video segments and also their ability to model temporal dynamics. The authors first propose ActionBench, which is a modification of two datasets (Ego4D, SS-v2) for probing the degree to which action antonyms and reversed videos impact the models, and show that frozen VidLMs do not perform these probing tasks well. The authors then propose Paxion + a DVDM training objective: Paxion has a knowledge patcher network (better action encoding) and a knowledge fuser (incorporating into frozen VidLMs), and DVDM extends the standard contrastive VTC loss to better correlate action text with the specific ordering of the video frames. The authors then benchmark their approach on downstream datasets and tasks (SS-v2, NExT-QA, etc.)\n\nSTRENGTHS:\n`+` Action/temporal understanding in large-scale pretrained video-language models is an important topic.\n\n`+` ActionBench contains probing tasks that seem like they may be interesting for future investigations, and the authors plan to release data and code for reproducibility.\n\n`+` The proposed Paxion/DVDM improvements seems to make a difference for the probing tasks relative to the base frozen VidLM, and there are ablations to show this.\n\n`+` Downstream evaluations on both SS-v2 and NExT-QA datasets to characterize the proposed model.\n\nWEAKNESSES:\n`-` There seems to be a limitation of how much the proposed techniques/tasks make a difference in the final downstream setting (e.g., side-tuning seems to ~match Paxion, within 0.3 points). This calls into question how over-specific the design of the proposed techniques and objectives are to the particular definitions of the proposed probing tasks in ActionBench (and by extension, how useful those formulations of the probing tasks are in downstream settings beyond SS-v2). (e.g., in [36] there is stronger impact on a larger range of downstream  video distributions/tasks)\n\n`-` The analysis for NExT-QA also seems incomplete. Prior/concurrent work (e.g., [5, 36]) report more detailed breakdowns of the accuracy into the specific causal/temporal splits of NExT-QA (are the improvements coming in the right categories?). Further, these other works report accuracy on a harder subset (ATP-hard) that makes the improvements from their verb/temporal augmentation techniques clearer. There is significant space in the current paper (Table 3) for these numbers (and they do not require additional compute, since they are all subsets), so it is unclear why they are not reported especially since they may help to bolster the core claims of the work.\n\n`-` The probing tasks seem closely related to ones proposed in prior work [5], but there doesn't seem to be a full discussion / comparison in this work of what these tasks add to those other ones (there is a brief citation in related work, but no discussion of this kind). Having a clearer sense of the potential complementarity would help to better clarify the contribution of this work (relatedly, how text/visual biases are controlled for in the probing tasks proposed here).\n\nQUESTIONS:\nPlease see the weaknesses section above for questions / areas to address in the rebuttal. Because the preliminary rating is borderline, it will be very helpful to finalize/potentially improve the rating.\n\nLIMITATIONS:\nThe authors provide a brief discussion of the limitation on focusing on one type of knowledge patch in the conclusion + dataset cards in the supplement.\n\n---\n\n---\n\n**Post-rebuttal update:** \n\nThe authors partially address the weaknesses/comments described in the initial review:\n\n`+` The additional analysis on the NExT-QA results helps to better support their method/claims (compared to the original result, which showed minor differences), and we can see a larger gap on some settings for temporal/causal/etc where it is helping now.\n\n`+` The additional discussion with related work provides helpful context of how these works can be viewed as complementary to one another.\n\n`-` The limitations around the impact to downstream tasks, however, still remain. The authors make the argument that there are not a lot of temporal datasets, which is understandable. However:\n1. Given one of the highlighted contributions of the work is proposing a new *benchmark* for probing (ActionBench), it would have been much stronger to make a connection to at least a couple other datasets that represent a spectrum of temporal/action understanding. Does ActionBench performance help to differentiate downstream settings where temporal/action understanding correlates well? Why or why not? (This is something that is illustrated by the proxy tasks in [5] well, in Table 5 of their work; it would have been interesting to see if this benchmark helps to complement/refine that other analysis).\n2. Furthermore, the fact that many datasets are not temporal does not mean there aren't *any* additional datasets (or subsets of datasets) for event reasoning that could have been interesting to consider (e.g., [5] considers AGQAv2; there are others like STAR as well; [36] identifies an action-focused split of Kinetics). \n3. Finally, it's worth noting the current work considers *two* evaluation datasets (SSv2 and NExT-QA), and SSv2 is also the basis for the original probing task (so that leaves only one fully new dataset to make a connection to). In contrast, related and concurrent work consider *many more* to help establish the broad efficacy of their proposed learning objectives, techniques, and analysis tasks (e.g., [5] considers 5+ downstream, [36] considers at least 4). In general, I don't think it is strictly necessary for papers to benchmark on tons of datasets, but given that a core contribution of the work is centered around a proposed probing task (and corresponding remedy), establishing this connection with at least one other dataset would have significantly improved the contribution of the work.\n\n`-` The author response that \"the way we construct the probing datasets...is fully automatic, meaning that it is not confined to a particular dataset such as SSv2\" is not necessarily persuasive, since this pipeline is tied to the specific (templated) language distribution of SSv2, and  more importantly, some of the tasks may not make sense on other video distributions besides SSv2 (e.g., reversing the frame order may only make sense on SSv2's short videos of gestures/object interactions, where the reversed video can map to a realistic new video (\"push object\" $\\leftrightarrow$ \"pull object\"), whereas if you show a model a video of people walking backwards or \"uncooking\" a pancake, it isn't quite as effective as a probing task since that's not a realistic new action).\n\n---\n\n**Overall (post-rebuttal):** I think the rebuttal response did help to address some key concerns, and I do think there is value in considering this work in the broader context of other related efforts [5, 36], so I am increasing my rating leaning towards acceptance. I hope the points mentioned above regarding continued areas of improvement will be helpful for future revisions and/or later work.\n\n---", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}