SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 7: Detailed training configurations for tasks in Tables 2, 3, and 4.
Downstream Task	Patching Dataset	Patching #Epochs	Fusing/Finetuning Dataset	Fusing/Finetuning #Epochs
SSv2-label [24]	SSv2	1	SSv2	1
SSv2-template [24]	SSv2	1	SSv2-template	2
Temporal-SSv2 [45]	SSv2	1	SSv2-template	2
NExT-QA [54]	NExT-QA	1	NExT-QA	4
Moments-In-Time [38]	SSv2	1	SSv2	1
Temporal-Kinetic [45]	SSv2	1	SSv2	1
Knowledge Patcher (Transformer). The Transformer variant of the Knowledge Patcher is a standard Transformer Encoder which contains a self-attention layer and a feedforward layer. The Transformer Encoder performs self-attention on the raw visual embeddings  $\mathcal{V}^{*}\in \mathbb{R}^{P,D}$  from the frozen backbone and output an updated visual embedding  $\mathcal{V}\in \mathbb{R}^{P,D}$ . To obtain video-text similarity, we first project the visual embeddings into the same dimension as the textual feature vector  $t^{*}\in \mathbb{R}^{d}$  and then do mean pooling before computing dot product.
Knowledge Fuser. The Knowledge Fuser has the same architecture as the Knowledge Patcher which contains a single cross-attention layer and a two-layer feedforward network. In this case, we use the pooled visual feature from the backbone  $\mathfrak{v}^* \in \mathbb{R}^d$  to provide query and the Knowledge Patcher output  $\mathcal{V} \in \mathbb{R}^{P,D}$  to provide key and value for the cross-attention. The intuition is to obtain a balanced representation for general downstream tasks by fusing the action-centric KP representation  $(\mathcal{V})$  with the object-centric backbone representation.
Side-Tuning. As an alternative to the Knowledge Fuser, we consider Side-Tuning [61] for further integrating the Knowledge Patcher with the backbone. Side-Tuning contains a base-model and a side-model, where the base-model is pretrained and frozen and the side-model is trainable. In our setting, we treat the backbone as the base-model and initialize the side-model using the trained Knowledge Patcher. We then side-tune the Knowledge Patcher along with the backbone using alpha blending. Specifically, the final fused visual feature  $\mathfrak{v}_f$  is obtained by  $\mathfrak{v}_f = \alpha (\mathfrak{v}^*) + (1 - \alpha)\mathfrak{v}$ , where  $\mathfrak{v}^*$  is the mean-pooled backbone visual feature, and the  $\mathfrak{v}$  is the mean-pooled Knowledge Patcher feature. And  $\alpha = \operatorname{Sigmoid}(a) \in [0,1]$ , where  $a$  a learnable scalar.
D.2 Knowledge Patcher Training.
We use two Nvidia Tesla V100 (16GB) GPUs for all experiments. For the Knowledge Patcher variants in Table 1, we train them on the training set of the datasets in the ActionBench for one epoch with either VTC loss only or VTC + DVDM (VAC + ATM) loss. We use AdamW [34] optimizer with a learning rate of 1e-5 and a weight decay of 0.05. For the transformer variant, we use a batch size of 8 per GPU. For the Perceiver variant, we are able to increase the batch size to 32 per GPU due to the reduced computation complexity.
D.3 Downstream Task Training.
Tables 6 and 7 shows detailed configurations for downstream task training with methods described in Tables 2 and 3, and Figure 6.
As shown in Table 7, the finetuning dataset for SSv2-label is identical to the SSv2 action knowledge patching dataset where the annotations are filled templates, such as "Book falling like a rock". The SSv2-template dataset, on the other hand, contains the object-obscured version of the original SSv2 annotations such as "Something falling like a rock". For the Video-to-Action Retrieval tasks, we consider two different subsets from the SSv2 validation set with the object-obfuscated annotations: SSv2-template [24] and Temporal-SSv2 [45]. SSv2-template contains all 174 action classes while Temporal-SSv2 contains 18 manually selected action classes that require more temporally-demanding distinctions, and cannot be distinguished using shuffled frames, such as "Approaching" and "Moving away". In order to investigate the impact of the action knowledge patching, we do not finetune a dedicated model for the 18 action classes for Temporal-SSv2, but instead use the model trained on SSv2-template to directly evaluate on Temporal-SSv2. Therefore, when observed larger improvements on Temporal-SSv2, we can draw the conclusion that patching with action knowledge contributes more to action-centric tasks (§ 4.2).
The hyperparameters, such as the learning rate, are identical to those used during Knowledge Patching training. For Video-Text Retrieval (SSv2-label) and Video-to-Action Retrieval (SSv2-template, Temporal-SSv2), the DVDM ( $\S 3.1$ ) objective includes VAC and ATM, while for Causal-Temporal VQA (NExT-QA), we only use VAC. This is because the training instances in NExT-QA are not formatted as video-text pairs but instead are in the format of multiple choice QA, making it not suitable for the ATM loss. Each video corresponds to one question and five candidate answers. We apply VAC to NExT-QA by adding action antonym text for each question as hard negative candidate answers.
For the downstream tasks (in Appendix A) for zero-shot cross-domain transfer (Moments-In-Time [38] and Temporal-Kinetic [45]), we use the model trained on SSv2 to perform zero-shot evaluation.
E Additional Qualitative Analysis
Figures 9 and 10 show additional qualitative examples on downstream tasks. The examples in demonstrate that PAXION improves understanding of challenging actions that require fine-grained temporal reasoning on the frames. For example, whether it is 'pretending' to do something or actually doing that, and whether an object is moving 'towards' or 'away' from the camera.
In Figure 11, we show failure cases of PAXION to discuss remaining challenges. We find that PAXION still struggle to understand negation and spatial attributes. For example, both VTC-Finetune baseline and PAXION fail to distinguish "without letting it drop down" from "then letting it drop down". For questions that require fine-grained spatial information of objects such as "how many goats can be spotted", PAXION cannot perform well. Potential solutions including incorporating the patched VidLM with a code language model to disentangle perception and reasoning similar to ViperGPT [47]. By leveraging the strong logical reasoning ability of a code language model, we can easily solve the negation and counting problems by creating code scripts with booleans and loops, and then use the VidLMs as "API calls".
Figure 9: Additional qualitative examples (Retrieval).
Video-to-Action Retrieval
Video-Text Retrieval (V2T)
Figure 10: Additional qualitative examples (VQA).
Causal-Temporal VQA (NExT-QA)
Video-Text Retrieval & Video-to-Action Retrieval Failure Examples
NExT-QA Failure Example
Figure 11: Failure examples.