SlowGuess/NAACL_2022_OCR huggingface dataset:
Jae Sung Park
Sheng Shen<sup>2</sup>
Ali Farhadi
Yejin Choi $^{1,3}$
Anna Rohrbach
<sup>1</sup> Paul G. Allen School of Computer Science & Engineering, University of Washington  
<sup>2</sup> University of California, Berkeley <sup>3</sup> Allen Institute for Artificial Intelligence
{jspark96,ali，yejin}@cs.washington.edu
{sheng.s, trevordarrell, anna.rohrbach}@berkeley.edu
Recent video-text models can retrieve relevant videos based on text with a high accuracy, but to what extent do they comprehend the semantics of the text? Can they discriminate between similar entities and actions? To answer this, we propose an evaluation framework that probes video-text models with hard negatives. We automatically build contrast sets, where true textual descriptions are manipulated in ways that change their semantics while maintaining plausibility. Specifically, we leverage a pre-trained language model and a set of heuristics to create verb and person entity focused contrast sets. We apply these in the multiple choice video-to-text classification setting. We test the robustness of recent methods on the proposed automatic contrast sets, and compare them to additionally collected human-generated counterparts, to assess their effectiveness. We see that model performance suffers across all methods, erasing the gap between recent CLIP-based methods vs. the earlier methods.
Relating video and text modalities is one of the important goals in vision and language. Video is a complex signal where people and objects act and interact with each other through space and time. Thus correctly associating a textual description and a video requires understanding of entities, their actions and much more, making it a hard problem.
One of the popular ways of training and evaluating video-text models is via cross-modal matching. Often the task is formulated as a retrieval problem, where the goal is to select the correct match among many (e.g. thousand) candidates, and distractors are picked randomly (Yu et al., 2018). Another way is via multiple-choice prediction, where the goal is to pick the true match out of several (e.g. 5) candidates (Torabi et al., 2016). The latter allows
Verb Manipulation
A: A girl feeding a brown horse.
B: A girl rides a brown horse.
C: Football team playing football on a field.
D: The man is drinking beer.
E: Two men playing a video game.
Entity Manipulation
A: Over her shoulders, HARRY glances at
RON, who lowers his gaze for a moment.
B: Over her shoulders RON glances at HARD.
HARRY, who lowers his gas.
C: He tries to shake it. D: LARRY's study.
D: HARRY studies it.  
E: BON whispers to HARRY.
GT
Predicted
Figure 1: Samples of our video-to-text tasks on the MSR-VTT (Xu et al., 2016) and LSMDC dataset (Rohrbach et al., 2017; Park et al., 2020). A hard negative option is added by manipulating verb (top) and entity (bottom) in the ground truth sentence. Two SOTA methods MMT (Gabeur et al., 2020) and CLIP4CLIP (Luo et al., 2021) incorrectly choose the manipulated sentence (option B) in both these cases.
for more controlled choice of negatives, which are typically selected from other videos. Commonly, the retrieval setting is used during training to avoid capturing any specific multiple-choice patterns or biases, while both are used for evaluation.
Recent methods that leverage the large-scale CLIP model (Radford et al., 2021) show significant improvement in cross-modal matching, specifically, in the retrieval setting (Fang et al., 2021; Luo et al., 2021). They outperform the prior state-of-the-art methods, often based on the Multimodal Transformer design (Miech et al., 2020; Gabeur et al., 2020; Lei et al., 2021). However, we know that often model performance is "over-estimated" due to the lack of challenging samples in evaluation. For instance, Gardner et al. (2020) show that model performance on several NLP tasks and one image-text task is much lower on contrast sets, which are test samples with small perturbation done by human experts in a way that changes the gold label.
In this work, we are investigating whether the video-text models also struggle in an evaluation framework that probes them with hard negatives. Instead of using human-designed contrast sets that are not easily scalable, we propose an automated pipeline that can generate contrast sets via verb and
1Code is available in https://github.com/
jamespark3922/video-lang-contrast-set
Human Language Technologies, pages 3574 - 3586
human entity manipulation. Our manipulations are carefully designed to preserve fluency but change semantics of the textual descriptions, making them invalid for a given video. We focus on entities and verbs to evaluate if the model can truly understand "who did what" in a video. Inspired by (Li et al., 2020; Morris et al., 2020), we leverage a generative T5 language model (Raffel et al., 2020) to manipulate the verb phrase and use heuristics to swap person entities. Note that our pipeline does not require a trained video-text model in the loop.
We apply our automatic manipulations to two popular video-text benchmarks, MSR-VTT (Xu et al., 2016) and LSMDC (Rohrbach et al., 2017). We additionally collect human generated contrast sets to compare with our automatic ones. To make sure that our automatic negatives are of high quality, we also confirm that humans can successfully select the correct description for a given video with our hard negatives. Finally, we benchmark several video-text models on our contrast sets. We find that all methods degrade in performance with the introduction of hard negatives in the multiple-choice setting (Figure 1). This includes the recent CLIP-based works that demonstrated large gains in the retrieval setting. This shows that all methods have difficulty discriminating between entities and verbs when the remaining context is unchanged. We observe that model performance drops especially on cases such as verb antonym swaps, where fine-grained action understanding is important.
Defending and generating adversarial examples (Jia et al., 2019; Jin et al., 2020) have been mostly explored in NLP since the reign of pre-trained language models (LMs) (Devlin et al., 2019). Li et al. (2020); Garg and Ramakrishnan (2020); Morris et al. (2020) show that substituting words in a sentence with masked LMs (Devlin et al., 2019; Liu et al., 2019) can successfully mislead the classification and entailment model predictions to be incorrect. Template-based (McCoy et al., 2019; Glockner et al., 2018) and manually crafted (Gardner et al., 2020) perturbations on evaluation datasets have also been studied for textual entailment. Ribeiro et al. (2020) have curated a list of checklists to reveal bugs present in NLP models.
Language-based adversarial examples can be collected to study the robustness of vision-language models as well. Shekhar et al. (2017) intro
duces FOIL-COCO dataset to evaluate the vision-language model's decision when associating images with both correct and "foil" captions. Akula et al. (2020) measure the robustness of visual referring expression models by checking if grounding is performed correctly after word manipulation. Hendricks and Nematzadeh (2021) show that vision-language Transformers are worse at verb understanding than nouns. New versions of the VQA dataset (Antol et al., 2015) are proposed to study robustness of VQA models (Shah et al., 2019; Li et al., 2021). Bitton et al. (2021) automatically generate contrast sets from scene graphs to probe compositional consistency of VQA models. Our work is different in that we use pre-trained LMs to introduce perturbations and evaluate robustness of video-language models.
3 Designing Contrast Sets
In this section we present our approach to automatically constructing text-based contrast sets for video-language tasks. Suppose we are given a video  $V_{i}$  and description  $s_i$ . Contrast sets  $\hat{C}_i = \{s_1, \dots, s_i\}$  are designed such that  $\hat{s}_i$  is semantically inconsistent with  $V_{i}$  and yet models incorrectly select  $\hat{s}_i$  over  $s_i$  in a video-to-text multiple-choice setting. While there are different ways to create valid  $\hat{C}_i$ , we investigate manipulating 1) person entities and 2) verb phrases in the original descriptions. Qualitative examples of  $\hat{C}$  are shown in Table 1.
3.1 Contrast Sets for Person Entities
First, we investigate automatically swapping the name (or identity) of a person. The LSMDC dataset (Rohrbach et al., 2017; Park et al., 2020) includes movie descriptions with character identities (e.g. Harry Potter), and a list of characters present in each movie along with their gender. We replace each character's ID with one from the same movie and with the same gender, to prevent the language statistics alone from detecting the swapped IDs.
For the MSR-VTT dataset (Xu et al., 2016) we do not have the identities; however,  $80\%$  of videos have gender cues in the descriptions. Thus the contrast sets are created by swapping the gender of a person mentioned in a sentence and the corresponding pronouns (e.g., A woman is pushing her stroller  $\rightarrow$  A man is pushing his stroller). This is done with a template that maps gender-sensitive words and pronouns to their counterparts (see Appendix).
Dataset	Original	Person Entity	Verb Phrase
MSRVTT	1. Two men are doing wrestling.	Two women are doing wrestling.	Two men are dancing.
2. A man in black shirt is talking with his two friends.	A woman in black shirt is talking with her two friends.	A man in black shirt is running with his two friends.
LSMDC-ID	3. His gaze steely, Jenko lowers his gun.	His gaze steely, Schmidt lowers his gun.	His gaze steely, Jenko raises his gun.
4. Jenko and Schmidt sit in the rear pew.	Zach and Schmidt sit in the rear pew.	Jenko and Schmidt stand in the rear pew.
Table 1: Examples of person entity and verb phrase based contrast sets in MSR-VTT and LSMDC-IDs dataset.
3.2 Language Model Generated Verb Contrast Sets
The above rule-based strategies cannot be directly translated to create contrast sets for verb phrases: 1) a substitute verb phrase is not guaranteed to be inconsistent with a video, and 2) the sentence may look unnatural and no longer be textually plausible. Based on their success in adversarial attack generation (Li et al., 2020; Garg and Ramakrishnan, 2020; Morris et al., 2020), we instead leverage pretrained language models (LMs) to automatically manipulate the verb phrases.
We identify verb phrases in a sentence using Spacy (Honnibal and Montani, 2017), replace them with a mask token [MASK], and select top  $K$  phrases that best fit the mask token using probability scores from a LM. Different from prior work (Li et al., 2020), we use T5-base model (Raffel et al., 2020) instead of masked language models (Devlin et al., 2019; Liu et al., 2019) to easily support generating multi-word candidates. We additionally finetune T5 to learn verb phrases in the downstream training data with unsupervised denoising objective (Raffel et al., 2020). This is done to mitigate the possible distribution shift between ground truth and manipulated descriptions, which could be exploited to distinguish between the two.
We then filter the  $K$  sentence candidates with the following criteria: 1) There is no verb in the sentence. 2) Verbs are rare or unseen in training descriptions. 3) The sentence has a high perplexity measured by GPT2-XL (Radford et al., 2019) to ensure grammaticality and plausibility (Morris et al., 2020). Lastly, we check that the semantics of a candidate is inconsistent with the original sentence. This is when  $a$ ) a candidate verb is an antonym² of an original verb, or  $b$ ) a word embedding (Mrkšić et al., 2016) of candidate and original verbs and
their sentence encodings (Reimers and Gurevych, 2019) both have low cosine similarity scores. We handle the antonyms separately, as the embedding-based scores do not adequately capture these, i.e., a sentence with an antonym verb may still be considered semantically close to an original sentence.
3.3 Human-Generated Verb Contrast Sets
Are language models capable of generating contrast sets of good quality? To answer this question, we follow the original contrast sets work (Gardner et al., 2020), and also create negatives manually to see if the performance on machine and human generated contrast sets is similar. We use the Amazon Mechanical Turk (AMT) platform and ask workers to modify a verb phrase such that a sentence becomes inconsistent with a video (see Appendix).
4.1 Datasets and Multiple Choice Design
MSR-VTT (Xu et al., 2016) is composed of 10K YouTube videos each paired with 20 natural descriptions and is typically evaluated on retrieval performance with 1000 video text pairs as candidates in the test set. The multiple choice version (Yu et al., 2018) has 2,990 test videos as queries, and a positive caption with 4 random captions from other videos as 5 answer options. We label this split as the Random MC. We design another MC problem by replacing one negative option with one from our contrast sets. In particular, Gender MC swaps gender in an original sentence; VerbLM MC and VerbH MC include verb-based negatives generated by our approach and by humans.
LSMDC (Rohrbach et al., 2017) includes short movie clips and captions. Characters in these captions are labeled as SOMEONE and we cannot construct contrast sets for person-entities. We instead use captions in (Park et al., 2020) that include
Extracted using VerbNet (Schuler, 2005).
Approach	V → T (R@1)	Random MC	Gender MC	VerbLM MC	VerbH MC
CLIP zero-shot	27.2	91.1	69.6	65.4	64.1
MMT	27.0	97.6	84.0	83.4	80.3
MMT-CLIP	30.8	97.2	84.0	80.9	78.3
CLIP4CLIP	43.1	98.4	82.7	83.7	80.2
CLIP2Video	43.3	98.3	78.5	81.1	79.0
Human	-	-	-	92.7	94.5
the character identities. We create a new split using the same movies in training and test so that the test identities have been seen during training. We call this modified dataset LSMDC-IDs. Using this set, Random MC is newly defined with 4 negative captions drawn randomly from different clips of the same movie. ID MC swaps the character IDs (Section 3.1) as negatives, and Verb MC includes the verb contrast sets, as before.
4.2 Video-Text Models and Evaluation
We benchmark Transformer (Vaswani et al., 2017) based video-language models in our experiments. Portillo-Quintero et al. (2021) apply frozen CLIP features (Radford et al., 2021) to perform zero-shot video to text retrieval (CLIP zero-shot). Multi Modal Transformer (MMT) (Gabeur et al., 2020) learns the joint representation between text and multiple modalities in videos. Inspired by Dzabraev et al. (2021), we also extend MMT to take frozen CLIP features as input, denoted as MMT-CLIP. CLIP4CLIP (Luo et al., 2021) and CLIP2Video (Fang et al., 2021) directly finetune CLIP with temporal pooler and are the state-of-the-art in retrieval tasks. ViT-B/32 model is used for all CLIP experiments (see Appendix C for details). We train the above models with a contrastive loss to learn the joint video-text representation. In MC settings, we mark it as correct, if a ground truth sentence is scored the highest. In addition, we also evaluate humans on the MC task. We report video-to-text  $(\mathrm{V} \rightarrow \mathrm{T})$  Recall@1 for retrieval evaluation.
Table 2 shows results on the MSR-VTT dataset. In video-to-text retrieval, we see a significant gap in performance between the CLIP-finetuned models and all other models. Moreover, CLIP zero-shot matches MMT in this metric. Next, we see that Random MC is nearly solved by almost all models. However there is a significant drop in performance
Table 2: Method comparison on MSR-VTT dataset. Human is majority vote over 3 judges.
Approach	V → TR@1	Random MC	ID MC	VerbLM MC	VerbH MC
CLIP zero-shot	4.3	53.3	39.8	38.9	35.7
MMT	17.7	73.2	65.2	56.2	56.9
MMT-CLIP	23.8	74.8	70.1	56.9	58.7
CLIP4CLIP	25.0	72.9	69.1	54.1	57.5
Human	-	-	-	90.2	92.8
Table 3: Method comparison on LSMDC-IDs dataset. Human is majority vote over 3 judges.
across all models when evaluated on contrast-set based MC. Interestingly, the performance gap between MMT and the finetuned CLIP models with high retrieval performance (CLIP4CLIP and CLIP2Video) is gone in this setting, meaning stronger retrieval performance does not guarantee robustness to word-level manipulations. We also observe that models with frozen CLIP features perform better on Gender MC than Verb MC, and finetuning the CLIP features on video-language task can make the model less sensitive to gender information. Finally, to verify that the automated verb-based contrast sets are valid, we note that: models on VerbLM MC perform on par with the human produced ones Verb  $H$  MC, and humans maintain accuracy greater than  $90\%$  on both contrast sets.3
Table 3 presents results on the LSMDC-IDs dataset. We find that distinguishing different clips of the same movie (Random MC) is not "solved" by the models unlike the MSR-VTT. We also notice that the ID swaps are significantly easier than the verb swaps, and CLIP features are particularly helpful in distinguishing different character IDs (MMT vs. MMT-CLIP). Table 4 shows that model accuracy drops by at least  $13.9\%$  when the "negative" IDs appear more frequently in the training data than the original IDs, meaning the models struggle to identify IDs in the long-tail. The results on verb contrast sets are similar to the MSR-VTT dataset. The performance is much lower on contrast-set MC cases than Random MC. There is no significant gap between  $\text{Verb}_{LM} MC$  and  $\text{Verb}_H MC$ . Our automated contrast sets are still valid as humans perform above  $90\%$  for both cases.
Does Semantic Proximity in Verb Contrast Sets Affect the Model Performance? To answer this, we first considered a subset containing verb antonyms. For the remaining ones, we use the off-the-shelf sentence encoders, SentBERT (Reimers and Gurevych, 2019) and CLIP text transformer
3We report majority vote over 3 human judges.
Original: A lady is holding a taco salad.
Original: A woman opening a package and pulling something out.
Verb Contrast Set: A lady stirs a taco salad.
Original: Helen wipes her nose.
Verb Contrast Set: Helen points to her nose.
Figure 2: Example failure cases where MMT and CLIP4CLIP prefer a negative sentence over an original one. Despite "watching" the video, the models have difficulty distinguishing fine grained actions (e.g. hold vs. stir, wipe vs. point), and verb antonyms (pulling vs. pushing, raise vs. lower.
Verb Contrast Set: A woman opening a package and pushing something out.
Original: Todd raises an arm victoriously, and so does Olive.
Verb Contrast Set: Todd lowers an arm victoriously, and so does Olive.
Overall	Rare	Δ
MMT	65.2	48.4	16.8
MMT-CLIP	70.1	56.2	13.9
CLIP4CLIP	69.1	54.2	14.9
Table 4: Accuracy for ID MC in LSMDC-IDs dataset. We show the overall accuracy and accuracy when the original ID was more rare than the swapped ID (Rare).  $\Delta$  is the difference between the two accuracies.
Verb Antonyms	SentBERT	CLIP-Text
High	Low	High	Low
CLIP zero-shot	53.8	63.5	76.9	60.4	72.3
MMT	67.6	79.6	95.3	80.2	89.8
CLIP4CLIP	70.6	77.6	95.7	77.0	92.4
Human	92.9	92.2	94.1	91.6	94.3
Table 5: Model accuracy on  $Verb_{LM}$  MC in MSR-VTT. We show contrast sets with verb antonyms, and remaining subsets with the highest (High) and lowest (Low)  $15\%$  semantic similarity with the original sentence (High and Low). Similarity scores are calculated using: SentBERT (Reimers and Gurevych, 2019) and CLIP text encoder (Radford et al., 2021).
(Radford et al., 2021), to measure the semantic proximity b.w. the original and negative sentences, and select the ones with the highest and lowest  $15\%$  according to these scores (High/Low) $^4$ . We present the results on MSR-VTT in Table 5. We notice that the models especially struggle with antonyms, such
as dropping from  $83.7\%$  (in Table 2) to  $70.6\%$  for CLIP4CLIP. Humans on the other hand get  $92.9\%$  accuracy and show no difference in their performance. The best models achieve high accuracy on par with humans on semantically different examples (Low) as measured by both SentBERT and CLIP-Text. However, model performance is much lower for contrast sets with high semantic similarity (High), whereas human performance is not as affected (e.g. CLIP4CLIP drops to  $77.6\%$  and humans maintain  $92.2\%$  accuracy on SentBERT). In Figure 2, we show failure cases where the SOTA models are misled by semantically close sentences and verb antonyms, due to their lack of fine-grained understanding of actions in the video.
We present a pipeline to build automatic contrast sets for video and language tasks, focused on manipulating person entities and verb phrases. We show that models struggle on contrast sets compared to random negatives, and stronger retrieval models do not show better robustness to hard negatives. For verb contrast sets, we find that model performance is strongly correlated with semantic proximity, unlike humans. We leave it as future work to use automatic contrast sets in training to improve model robustness, and designing contrast sets for different concepts/parts of speech.
These subsets are disjoint from the antonym set to avoid scoring antonyms as semantically similar (see Section 3.2).
Our goal is to diagnose performance of video-language models on hard negative samples w.r.t. verbs and person entities. Overall, we envision positive impact from this work, as it aims to expose limitations of the existing models. Some of our entity swaps focus on apparent gender (as described by humans in the video-text datasets), but we do not predict biological sex or gender identity. We construct our verb-focused contrast sets automatically, using a large generative language model, thus potentially some biases present in such a model could propagate into our hard negative samples. Practitioners who wish to use our contrast sets should be mindful of such sources of bias.
This work was funded by DARPA MCS program through NIWC Pacific (N66001-19-2-4031), and in part by DARPA's LwLL, and/or SemaFor programs, and Berkeley Artificial Intelligence Research (BAIR) industrial alliance programs.
Arjun Reddy Akula, Spandana Gella, Yaser Al-Onaizan, Song-Chun Zhu, and Siva Reddy. 2020. Words aren't enough, their order matters: On the robustness of grounding visual referring expressions. In ACL.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In ICCV.
Yonatan Bitton, Gabriel Stanovsky, Roy Schwartz, and Michael Elhadad. 2021. Automatic generation of contrast sets from scene graphs: Probing the compositional consistency of gqa. ArXiv, abs/2103.09591.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805.
Maksim Dzabraev, Maksim Kalashnikov, Stepan Alekseevich Komkov, and Aleksandr Petiushko. 2021. Mdmmt: Multidomain multimodal transformer for video retrieval. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3349-3358.
Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. 2021. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020. Multi-modal transformer