Venue: NAACL 2022

# Exposing the Limits of Video-Text Models through Contrast Sets

Jae Sung Park\({}^{1}\) &Sheng Shen\({}^{2}\) Ali Farhadi\({}^{1}\)   Trevor Darrell\({}^{2}\) &Yejin Choi\({}^{1,3}\) &Anna Rohrbach\({}^{2}\)

###### Abstract

Recent video-text models can retrieve relevant videos based on text with a high accuracy, but to what extent do they comprehend the semantics of the text? Can they discriminate between similar entities and actions? To answer this, we propose an evaluation framework that probes video-text models with hard negatives. We automatically build _contrast sets_, where true textual descriptions are manipulated in ways that change their semantics while maintaining plausibility. Specifically, we leverage a pre-trained language model and a set of heuristics to create verb and person entity focused contrast sets. We apply these in the multiple choice video-to-text classification setting. We test the robustness of recent methods on the proposed automatic contrast sets, and compare them to additionally collected human-generated counterparts, to assess their effectiveness. We see that model performance suffers across all methods, erasing the gap between recent CLIP-based methods vs. the earlier methods. 1

Footnote 1: Code is available in [https://github.com/jamespark922/video-lang-contrast-set](https://github.com/jamespark922/video-lang-contrast-set)

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3574 - 3586

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3574 - 3586

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3574 - 3586

## 1 Introduction

Relating video and text modalities is one of the important goals in vision and language. Video is a complex signal where people and objects act and interact with each other through space and time. Thus correctly associating a textual description and a video requires understanding of entities, their actions and much more, making it a hard problem.

One of the popular ways of training and evaluating video-text models is via cross-modal matching. Often the task is formulated as a retrieval problem, where the goal is to select the correct match among many (e.g. thousand) candidates, and distractors are picked randomly Yu et al. (2018). Another way is via multiple-choice prediction, where the goal is to pick the true match out of several (e.g. 5) candidates Torabi et al. (2016). The latter allows for more controlled choice of negatives, which are typically selected from other videos. Commonly, the retrieval setting is used during training to avoid capturing any specific multiple-choice patterns or biases, while both are used for evaluation.

Recent methods that leverage the large-scale CLIP model Radford et al. (2021) show significant improvement in cross-modal matching, specifically, in the retrieval setting Fang et al. (2021); Luo et al. (2021). They outperform the prior state-of-the-art methods, often based on the Multimodal Transformer design Miech et al. (2020); Gabeur et al. (2020); Lei et al. (2021). However, we know that often model performance is "over-estimated" due to the lack of challenging samples in evaluation. For instance, Gardner et al. (2020) show that model performance on several NLP tasks and one image-text task is much lower on _contrast sets_, which are test samples with small perturbation done by human experts in a way that changes the gold label.

In this work, we are investigating whether the video-text models also struggle in an evaluation framework that probes them with hard negatives. Instead of using human-designed contrast sets that are not easily scalable, we propose an automated pipeline that can generate contrast sets via verb and

Figure 1: Samples of our video-to-text tasks on the MSR-VTT Xu et al. (2016) and LSMDC dataset Rohrbach et al. (2017); Park et al. (2020). A hard negative option is added by manipulating verb (top) and entity (bottom) in the ground truth sentence. Two SOTA methods MMT Galeour et al. (2020) and CLIP4CLIP Luo et al. (2021) incorrectly choose the manipulated sentence (option B) in _both_ these cases.



human entity manipulation. Our manipulations are carefully designed to preserve fluency but change semantics of the textual descriptions, making them invalid for a given video. We focus on _entities_ and _verbs_ to evaluate if the model can truly understand "who did what" in a video. Inspired by Li et al. (2020); Morris et al. (2020), we leverage a generative T5 language model Raffel et al. (2020) to manipulate the verb phrase and use heuristics to swap person entities. Note that our pipeline does not require a trained video-text model in the loop.

We apply our automatic manipulations to two popular video-text benchmarks, MSR-VTT Xu et al. (2016) and LSMDC Rohrbach et al. (2017). We additionally collect human generated contrast sets to compare with our automatic ones. To make sure that our automatic negatives are of high quality, we also confirm that humans can successfully select the correct description for a given video with our hard negatives. Finally, we benchmark several video-text models on our contrast sets. We find that all methods degrade in performance with the introduction of hard negatives in the multiple-choice setting (Figure 1). This includes the recent CLIP-based works that demonstrated large gains in the retrieval setting. This shows that all methods have difficulty discriminating between entities and verbs when the remaining context is unchanged. We observe that model performance drops especially on cases such as verb antonym swaps, where fine-grained action understanding is important.

## 2 Related Work

Defending and generating adversarial examples Jia et al. (2019); Jin et al. (2020) have been mostly explored in NLP since the reign of pre-trained language models (LMs) Devlin et al. (2019). Li et al. (2020); Garg and Ramakrishnan (2020); Morris et al. (2020) show that substituting words in a sentence with masked LMs Devlin et al. (2019); Liu et al. (2019) can successfully mislead the classification and entailment model predictions to be incorrect. Template-based McCoy et al. (2019); Glockner et al. (2018) and manually crafted Gardner et al. (2020) perturbations on evaluation datasets have also been studied for textual entailment. Ribeiro et al. (2020) have curated a list of checklists to reveal bugs present in NLP models.

Language-based adversarial examples can be collected to study the robustness of vision-language models as well. Shekhar et al. (2017) introduces FOIL-COCO dataset to evaluate the vision-language model's decision when associating images with both correct and "foil" captions. Akula et al. (2020) measure the robustness of visual referring expression models by checking if grounding is performed correctly after word manipulation. Hendricks and Nematzadeh (2021) show that vision-language Transformers are worse at verb understanding than nouns. New versions of the VQA dataset Antol et al. (2015) are proposed to study robustness of VQA models Shah et al. (2019); Li et al. (2021). Bitton et al. (2021) automatically generate contrast sets from scene graphs to probe compositional consistency of VQA models. Our work is different in that we use pre-trained LMs to introduce perturbations and evaluate robustness of _video_-language models.

## 3 Designing Contrast Sets

In this section we present our approach to automatically constructing _text-based_ contrast sets for videologanguage tasks. Suppose we are given a video \(V_{i}\) and description \(s_{i}\). Contrast sets \(\hat{C}_{i}=\{\hat{s_{1}},...,\hat{s_{i}}\}\) are designed such that \(\hat{s_{i}}\) is semantically inconsistent with \(V_{i}\) and yet models incorrectly select \(\hat{s_{i}}\) over \(s_{i}\) in a video-to-text multiple-choice setting. While there are different ways to create valid \(\hat{C}_{i}\), we investigate manipulating 1) _person entities_ and 2) _verb phrases_ in the original descriptions. Qualitative examples of \(\hat{C}\) are shown in Table 1.

### Contrast Sets for Person Entities

First, we investigate automatically swapping the name (or _identity_) of a person. The LSMDC dataset Rohrbach et al. (2017); Park et al. (2020) includes movie descriptions with character identities (e.g. _Harry Potter_), and a list of characters present in each movie along with their gender. We replace each character's ID with one from the same movie and with the same gender, to prevent the language statistics alone from detecting the swapped IDs.

For the MSR-VTT dataset Xu et al. (2016) we do not have the identities; however, 80% of videos have gender cues in the descriptions. Thus the contrast sets are created by swapping the gender of a person mentioned in a sentence and the corresponding pronouns (e.g., _A woman is pushing her stroller \(\rightarrow\) A man is pushing his stroller_). This is done with a template that maps gender-sensitive words and pronouns to their counterparts (see Appendix) 



### Language Model Generated Verb Contrast Sets

The above rule-based strategies cannot be directly translated to create contrast sets for verb phrases: 1) a substitute verb phrase is not guaranteed to be inconsistent with a video, and 2) the sentence may look unnatural and no longer be textually plausible. Based on their success in adversarial attack generation Li et al. (2020); Garg and Ramakrishnan (2020); Morris et al. (2020), we instead leverage pretrained language models (LMs) to automatically manipulate the verb phrases.

We identify verb phrases in a sentence using Spacy Honnibal and Montani (2017), replace them with a mask token [MASK], and select top \(K\) phrases that best fit the mask token using probability scores from a LM. Different from prior work Li et al. (2020), we use T5-base model Raffel et al. (2020) instead of masked language models Devlin et al. (2019); Liu et al. (2019) to easily support generating multi-word candidates. We additionally finetune T5 to learn verb phrases in the downstream training data with unsupervised denoising objective Raffel et al. (2020). This is done to mitigate the possible distribution shift between ground truth and manipulated descriptions, which could be exploited to distinguish between the two.

We then filter the \(K\) sentence candidates with the following criteria: 1) There is no verb in the sentence. 2) Verbs are rare or unseen in training descriptions. 3) The sentence has a high perplexity measured by GPT2-XL Radford et al. (2019) to ensure grammaticality and plausibility Morris et al. (2020). Lastly, we check that the semantics of a candidate is _inconsistent_ with the original sentence. This is when _a)_ a candidate verb is an antonym2 of an original verb, or _b)_ a word embedding Mrksic et al. (2016) of candidate and original verbs and their sentence encodings Reimers and Gurevych (2019) both have low cosine similarity scores. We handle the anonyms separately, as the embedding-based scores do not adequately capture these, i.e., a sentence with an antonym verb may still be considered semantically close to an original sentence.

Footnote 2: Extracted using VerbNet Schuler (2005).

### Human-Generated Verb Contrast Sets

Are language models capable of generating contrast sets of good quality? To answer this question, we follow the original contrast sets work Gardner et al. (2020), and also create negatives manually to see if the performance on machine and human generated contrast sets is similar. We use the Amazon Mechanical Turk (AMT) platform and ask workers to modify a verb phrase such that a sentence becomes inconsistent with a video (see Appendix).

## 4 Experiments

### Datasets and Multiple Choice Design

**MSR-VTT Xu et al. (2016)** is composed of 10K YouTube videos each paired with 20 natural descriptions and is typically evaluated on retrieval performance with 1000 video text pairs as candidates in the test set. The multiple choice version Yu et al. (2018) has 2,990 test videos as queries, and a positive caption with \(4\) random captions from other videos as \(5\) answer options. We label this split as the Random MC. We design another MC problem by replacing one negative option with one from our contrast sets. In particular, Gender MC swaps gender in an original sentence; Verb\({}_{LM}\) MC and Verb\({}_{HI}\) MC include verb-based negatives generated by our approach and by humans.

**LSMDC Rohrbach et al. (2017)** includes short movie clips and captions. Characters in these captions are labeled as SOMEONE and we cannot construct contrast sets for person-entities. We instead use captions in Park et al. (2020) that include

\begin{table}
\begin{tabular}{l l p{142.3pt} p{142.3pt}} \hline \hline
**Dataset** & **Original** & **Person Entity** & **Verb Phrase** \\ \hline
**MSRVTT** & 1. Two men are doing wrestling. & Two women are doing wrestling. & Two men are dancing. \\  & 2. A man in black shirt is talking with his two friends. & A woman in black shirt is talking with her two friends. & A man in black shirt is running with his two friends. \\
**LSMDC-ID** & 3. His gaze steely, Jenko lowers his gun. & His gaze steely, Schmidt lowers his gun. & His gaze steely, Jenko raises his gun. \\  & 4. Jenko and Schmidt sit in the rear paw. & Zach and Schmidt sit in the rear pew. & Jenko and Schmidt stand in the rear pew. \\ \hline \hline \end{tabular}
\end{table}
Table 1: Examples of person entity and verb phrase based contrast sets in MSR-VTT and LSMDC-IDs dataset.



the character identities. We create a new split using the same movies in training and test so that the test identities have been seen during training. We call this modified dataset **LSMDC-IDs**. Using this set, Random MC is newly defined with \(4\) negative captions drawn randomly from different clips of the same movie. ID MC swaps the character IDs (Section 3.1) as negatives, and Verb MC includes the verb contrast sets, as before.

### Video-Text Models and Evaluation

We benchmark Transformer Vaswani et al. (2017) based video-language models in our experiments. Portillo-Quintero et al. (2021) apply frozen CLIP features Radford et al. (2021) to perform zero-shot video to text retrieval (CLIP zero-shot). Multi Modal Transformer (MMT) Gabeur et al. (2020) learns the joint representation between text and multiple modalities in videos. Inspired by Dzabraev et al. (2021), we also extend MMT to take frozen CLIP features as input, denoted as MMT-CLIP. CLIP4CLIP Luo et al. (2021) and CLIP2Video Fang et al. (2021) directly finetune CLIP with temporal pooler and are the state-of-the-art in retrieval tasks. ViT-B/32 model is used for all CLIP experiments (see Appendix C for details). We train the above models with a contrastive loss to learn the joint video-text representation. In MC settings, we mark it as correct, if a ground truth sentence is scored the highest. In addition, we also evaluate humans on the MC task. We report video-to-text (V \(\rightarrow\) T) Recall@1 for retrieval evaluation.

### Results

Table 2 shows results on the MSR-VTT dataset. In video-to-text retrieval, we see a significant gap in performance between the CLIP-finetuned models and all other models. Moreover, CLIP zero-shot matches MMT in this metric. Next, we see that Random MC is nearly solved by almost all models. However there is a significant drop in performance across all models when evaluated on constrast-set based MC. Interestingly, the performance gap between MMT and the finetuned CLIP models with high retrieval performance (CLIP4CLIP and CLIP2Video) is gone in this setting, meaning stronger retrieval performance does not guarantee robustness to word-level manipulations. We also observe that models with frozen CLIP features perform better on Gender MC than Verb MC, and fine-tuning the CLIP features on video-language task can make the model less sensitive to gender information. Finally, to verify that the automated verb-based contrast sets are valid, we note that: models on VerbLM MC perform on par with the human produced ones VerbH MC, and humans maintain accuracy greater than \(90\%\) on both contrast sets.3

Footnote 3: We report majority vote over 3 human judges.

Table 3 presents results on the LSMDC-IDs dataset. We find that distinguishing different clips of the same movie (Random MC) is not "solved" by the models unlike the MSR-VTT. We also notice that the ID swaps are significantly easier than the verb swaps, and CLIP features are particularly helpful in distinguishing different character IDs (MMT vs. MMT-CLIP). Table 4 shows that model accuracy drops by at least 13.9% when the "negative" IDs appear more frequently in the training data than the original IDs, meaning the models struggle to identify IDs in the long-tail. The results on verb contrast sets are similar to the MSR-VTT dataset. The performance is much lower on contrast-set MC cases than Random MC. There is no significant gap between VerbLM MC and VerbHC. Our automated contrast sets are still valid as humans perform above 90% for both cases.

Does Semantic Proximity in Verb Contrast Sets Affect the Model Performance?To answer this, we first considered a subset containing verb antonyms. For the remaining ones, we use the off-the-shelf sentence encoders, SentBERT Reimers and Gurevych (2019) and CLIP text transformer

\begin{table}
\begin{tabular}{l|c|c|c|c|c} \hline \hline  & V → T & Random & Gender & VerbLM & VerbH \\

- & 92.7 & 94.5 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Method comparison on **MSR-VTT** dataset. Human is majority vote over 3 judges.

\begin{table}
\begin{tabular}{l|c|c|c|c|c} \hline \hline  & V → T & Random & ID & VerbLM & VerbH \\

- & 90.2 & 92.8 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Method comparison on **LSMDC-IDs** dataset. Human is majority vote over 3 judges.
 

(Radford et al., 2021), to measure the semantic proximity b.w. the original and negative sentences, and select the ones with the highest and lowest 15% according to these scores (High/Low)4. We present the results on MSR-VTT in Table 5. We notice that the models especially struggle with antonyms, such as dropping from 83.7% (in Table 2) to 70.6% for CLIP4CLIP. Humans on the other hand get 92.9% accuracy and show no difference in their performance. The best models achieve high accuracy on par with humans on semantically different examples (Low) as measured by both SentBERT amd CLIP-Text. However, model performance is much lower for contrast sets with high semantic similarity (High), whereas human performance is not as affected (e.g. CLIP4CLIP drops to 77.6% and humans maintain 92.2% accuracy on SentBERT). In Figure 2, we show failure cases where the SOTA models are misled by semantically close sentences and verb antonyms, due to their lack of fine-grained understanding of actions in the video.

Footnote 4: These subsets are disjoint from the antonym set to avoid scoring antonyms as semantically similar (see Section 3.2).

## 5 Conclusion

We present a pipeline to build automatic contrast sets for video and language tasks, focused on manipulating person entities and verb phrases. We show that models struggle on contrast sets compared to random negatives, and stronger retrieval models do not show better robustness to hard negatives. For verb contrast sets, we find that model performance is strongly correlated with semantic proximity, unlike humans. We leave it as future work to use automatic contrast sets in training to improve model robustness, and designing contrast sets for different concepts/parts of speech.

\begin{table}
\begin{tabular}{c|c c} \hline \hline  & Overall Rare & \(\Delta\) \\ \hline MMT & 65.2 & 48.4 & 16.8 \\ MMT-CLIP & 70.1 & 56.2 & 13.9 \\ CLIP4CLIP & 69.1 & 54.2 & 14.9 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Accuracy for ID MC in **LSMDC-IDs** dataset. We show the overall accuracy and accuracy when the original ID was more rare than the swapped ID (Rare). \(\Delta\) is the difference between the two accuracies.

Figure 2: Example failure cases where MMT and CLIP4CLIP prefer a negative sentence over an original one. Despite “watching” the video, the models have difficulty distinguishing fine grained actions (e.g. _hold vs. stirr_, _pipe vs. point_), and verb antonyms (_pulling vs. pushing_, _raise vs. lower_).

 

Ethical Considerations

Our goal is to diagnose performance of video-language models on hard negative samples w.r.t. verbs and person entities. Overall, we envision positive impact from this work, as it aims to expose limitations of the existing models. Some of our entity swaps focus on apparent gender (as described by humans in the video-text datasets), but we do not predict biological sex or gender identity. We construct our verb-focused contrast sets automatically, using a large generative language model, thus potentially some biases present in such a model could propagate into our hard negative samples. Practitioners who wish to use our contrast sets should be mindful of such sources of bias.

#### Acknowledgements

This work was funded by DARPA MCS program through NIWC Pacific (N66001-19-2-4031), and in part by DARPA's LwLL, and/or SemaFor programs, and Berkeley Artificial Intelligence Research (BAIR) industrial alliance programs.

## References

* A. Reddy Akula, S. Gella, Y. Al-Onaizan, S. Zhu, and S. Reddy (2020)Words aren't enough, their order matters: On the robustness of grounding visual referring expressions. In ACL, Cited by: SS5.
* S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh (2015)Vqa: Visual question answering.. In ICCV, Cited by: SS5.
* Y. Bitton, G. Stanovsky, R. Schwartz, and M. Elhadad (2021)Automatic generation of contrast sets from scene graphs: probing the compositional consistency of gqa. ArXivabs/2103.09591. External Links: Link, 2103.09591 Cited by: SS5.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. ArXivabs/1810.04805. External Links: Link, 1810.04805 Cited by: SS5.
* M. Dzabrev, M. Kalashnikov, S. Alekseevich Komkov, and A. Petiushko (2021)Mdmmt: multidomain multimodal transformer for video retrieval. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3349-3358. External Links: Document, ISSN 1749-749X Cited by: SS5.
* H. Fang, P. Xiong, L. Xu, and Y. Chen (2021)Clip2video: mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097. External Links: Link, 2106.11097 Cited by: SS5.
* V. Gabeur, C. Sun, K. Alahari, and C. Schmid (2020)Multi-modal transformer for video retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 214-229. External Links: ISBN 978-3-319 

Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. Bert-attack: Adversarial attack against bert using bert. In _The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _ArXiv_, abs/1907.11692.
* Luo et al. (2021) Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianruti Li. 2021. Clip4clip: An empirical study of clip for end to end video clip retrieval. _arXiv preprint arXiv:2104.08860_.
* McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3428-3448.
* Miech et al. (2020) Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual representations from uncurated instructional videos. _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 9876-9886.
* Miech et al. (2018) Antoine Miech, Ivan Laptev, and Josef Sivic. 2018. Learning a text-video embedding from incomplete and heterogeneous data. _arXiv preprint arXiv:1804.02516_.
* Micch et al. (2019) Antoine Michel, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howt100m: Learning a text-video embedding by watching hundred million narrated video clips. _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_, pages 2630-2640.
* Morris et al. (2020) John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In _EMNLP_.
* Mrksic et al. (2016) Nikola Mrksic, Diparmuid O Seaghdha, Blaise Thomson, Milica Gasic, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In _Proceedings of HLT-NAACL_.
* Park et al. (2020) Jae Sung Park, Trevor Darrell, and Anna Rohrbach. 2020. Identity-aware multi-sentence video description. In _European Conference on Computer Vision_, pages 360-378. Springer.
* Portillo-Quintero et al. (2021) Jesus Andres Portillo-Quintero, Jose Carlos Ortiz-Bayliss, and Hugo Terashima-Marin. 2021. A straightforward framework for video retrieval using clip.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In _ICML_.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Sherry Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. In _ACL_.
* Rohrbach et al. (2017) Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Chris Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2017. Movie description. _International Journal of Computer Vision_.
* Schuler (2005) Karin Kipper Schuler. 2005. _VerbNet: A broad-coverage, comprehensive verb lexicon_. University of Pennsylvania.
* Shah et al. (2019) Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. 2019. Cycle-consistency for robust visual question answering. _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 6642-6651.
* Shekhar et al. (2017) Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Autrelie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. 2017. Foil it! find one mismatch between image and language caption. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 255-265.
* Torabi et al. (2016) Atousa Torabi, Niket Tandon, and Leon Sigal. 2016. Learning language-visual embedding for movie understanding with natural-language. _arXiv:1609.08124_.
* Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. _ArXiv_, abs/1706.03762.
* Xie et al. (2018) Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. 2018. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification 

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 5288-5296.
* Yu et al. (2018) Youngjaeu Yu, Jongseok Kim, and Gunhee Kim. 2018. A joint sequence fusion model for video question answering and retrieval. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pages 471-487.

 



## Appendix A Contrast Set Construction

Here, we provide more details on construction of each contrast set.

### Gender Contrast Sets

Table 6 shows the mapping of gender-sensitive words. We use these rules to swap only a single word in the sentence. This is to guarantee that swapping gender leads to different semantics (e.g. _man and woman walk together_\(\rightarrow\)_woman and man walk together_ both apply to the same video if all words are swapped). If there are more than one possible mappings, we randomly sample one from a uniform distribution. Lastly, we swap all gender-sensitive pronouns that have the same gender as original noun. These contrast sets are used for the MSR-VTT dataset Xu et al. (2016).

### Person ID Contrast Sets

The first character ID in a sentence is replaced by a different character ID that appears in the same movie and has the same gender. Among all the candidates, the manipulated ID is sampled from a uniform distribution. The following character IDs in the same sentence have uniform chance of being kept or swapped using the same strategy. These contrast sets are used for the LSMDC-IDs dataset.

### Verb Contrast Sets

Attack SelectionWe use Spacy to get the POS tags, and find verb phrases that match a list of pre-defined patterns (verb; verb + preposition).

Candidate GenerationWe use T5 model and performed beam search (beam size = 50) to generate \(K=50\) multi-word candidates.

Candidate ConstraintsWe keep a candidate if the lemmatized verbs 5 in it appeared more than 30 times in the training set. For fluency, we calculate perplexity score of original and manipulated sentence using GPT2-XL Radford et al. (2019), which we call \({\text{ppl}}_{o}\) and \({\text{ppl}}_{m}\). We calculate the normalized difference of perplexity scores \({\text{pppl}}_{\text{diff}}=\frac{{\text{ppl}}_{o}-{\text{ppl}}_{m}}{{\text{ ppl}}_{o}}\) to remove a candidate that is less plausible than the original. Specifically, candidates are kept if \({\text{ppdl}}_{\text{diff}}<0.6\), or \({\text{ppdl}}_{\text{diff}}<1.4\cap{\text{ppl}}_{m}<750\). Lastly, the semantic inconsistency constraints are satisfied if the word embedding Mrksic et al. (2016) of the lemmatized verbs in the candidate and original sentence have cosine similarity score lower than 0.4, and the sentence embeddings Reimers and Gurevych (2019) have cosine similarity score lower than 0.8.

Footnote 5: [https://www.nltk.org/modules/nltk/stem/wordnet.html](https://www.nltk.org/modules/nltk/stem/wordnet.html)

## Appendix B Human vs Machine Generated Verb Contrast Sets

Figure 3 shows a distribution of machine and human generated verb contrast sets. Each instance is the number of lemmatized verbs divided by total number of verbs in the contrast sets. We see that machine generated contrast set is more skewed to the left, and doesn't share the same distribution of verbs as in the human generated contrast sets. (e.g. human contrast sets have more occurrences of _cry_ and _throw_ in MSR-VTT, and _jump_ and _drop_ in LSMDC-IDs). Despite the difference, note that models have similar performances in both contrast sets.

## Appendix C Implementation Details

* **MMT**Gabeur et al. (2020): We use the following features extracted from video6: motion from S3D Xie et al. (2018), audio from VGGish Hershey et al. (2017), scene embeddings, face, OCR, Speech, and Appearance. We refer to Miech et al. (2018); Gabeur et al. (2020) for more details about the features. Footnote 6: [https://github.com/albanie/collaborative-experts](https://github.com/albanie/collaborative-experts)

For MSR-VTT, we use the released checkpoint from their code7, which is pre-trained on HowTo100M dataset Miech et al. (2019) and further finetuned on MSR-VTT.

\begin{table}
\begin{tabular}{l|l} \hline
**Male Nouns** & **Female Nouns** \\ \hline man \(\rightarrow\) woman & woman \(\rightarrow\) man \\ men \(\rightarrow\) women & women \(\rightarrow\) men, guys \\ boy \(\rightarrow\) girl & girl \(\rightarrow\) boy, guy \\ boys \(\rightarrow\) girls & girls \(\rightarrow\) boys, guys \\ guy \(\rightarrow\) woman, girl & lady \(\rightarrow\) man, guy \\ guys \(\rightarrow\) women, girls, ladies & ladies \(\rightarrow\) men, guys \\ \hline \end{tabular}
\end{table}
Table 6: List of gender sensitive words mapped to a different gender. Note, that singular and plural form is maintained.



For LSMDC-IDs which needs re-training, we used their finetuning code for LSMDC dataset (Rohrbach et al., 2017). The model is trained with max margin ranking loss on 1 Nvidia RTX-6000 GPU for 12 hours. Hyperparameter search was done to find margin of 0.05, batch size of 32, and Adam optimizer (Kingma and Ba, 2015) with learning rate \(5e^{-5}\). The best model was selected by the video-to-text retrieval performance with Recall@1. We found training from scratch performs better than using pre-trained model. This has been also observed by Gabeur et al. (2020) for the LSMDC dataset.
* **MMT-CLIP**: We replace the appearance features in MMT with frozen CLIP ViTB/32 features and train with the same architecture.
* **CLIP zero-shot:** In (Portillo-Quintero et al., 2021) CLIP(ViTB-32) (Radford et al., 2021) features are aggregated via mean pooling to approximate video representation. This video representation and text embedding from CLIP are combined to perform retrieval and MC in a zero shot manner.
* **CLIP4CLIP**(Luo et al., 2021): We use the hyperparameters from the finetuning code8 to reproduce their results. We use mean pooling for the similarity calculator and CLIP model is initialized with ViTB-32 weights. The model was trained with 4 Nvidia RTX-6000 GPUs for 5 epochs (48 gpu hours). The best model was selected by using Recall@1 in video-to-text retrieval. Footnote 8: [https://github.com/ArrowLuo/CLIP4Clip](https://github.com/ArrowLuo/CLIP4Clip)

Figure 3: Distribution of machine and human generated verb contrast sets in the MSR-VTT and LSMDC-IDs dataset. Each instance is the ratio of lemmatized verbs within each contrast set.

 

* **CLIP2Video**Fang et al. (2021): We used the released checkpoint on MSR-VTT using their code base9. This model is not used for LSMDC-IDs because finetuning code was not provided. CLIP model is initialized with ViTB-32 weights. Footnote 8: [https://github.com/CyhanFang/CLIP2Video](https://github.com/CyhanFang/CLIP2Video)

## Appendix D Multiple Choice Details

Here we provide more details about our evaluation data. Note, that we use 5 text candidates (1 positive and 4 negative) for all multiple choice (MC) settings.

### Msr-Vtt

We use the standard train/val/test split in MSRVT dataset Xu et al. (2016).

* Retrieval: 1,000 ground truth video-text pairs in the test set Yu et al. (2018).
* Random MC: 2,990 videos and all negative options are drawn randomly from other videos Yu et al. (2018).
* Gender MC: 2,477 video-text instances. Using the original descriptions from Random MC, a single negative is drawn from gender contrast sets to replace one of the options in Random MC (the remaining 3 are kept). Note, that not all videos involved people or contained gender-sensitive words in descriptions, hence some instances are filtered.
* Verb\({}_{LM}\) MC: 2,554 video-text instances. Constructed using the same strategy as in Gender MC but a single negative is drawn from verb contrast sets generated by language models. Instances are filtered when there are no valid verb contrast sets satisfying constraints in Section A.3.
* Verb\({}_{H}\) MC: 2,554 video-text instances. We use the instances in Verb\({}_{LM}\) MC, and a negative is drawn from human designed verb contrast sets.

### LSDmc-IDs

We define a new split using LSMDC-ID descriptions with character IDs (proper names) Park et al. (2020). Note, that Rohrbach et al. (2017); Park et al. (2020) use development and test sets where videos come from distinct movies than the training data, meaning that IDs in test data are not seen in training. To overcome this issue, we split their _training_ descriptions into 80%/10%/10%/ ratio to create new training/validation/test sets that _share_ the same movies and identities across splits.

* Retrieval: 7,010 ground truth video-text pairs.
* Random MC: 7,010 videos, negative text options drawn randomly from different videos but the same movie.
* ID MC: 7,010 video-text instances. We replace one negative in Random MC with the one from ID contrast sets.
* Verb\({}_{LM}\) MC: 7,010 video-text instances. We replace one negative in Random MC with one from the language model generated verb contrast sets.
* Verb\({}_{H}\) MC: 3,500 video-text instances. We replace one negative in Random MC with one from the human designed verb contrast sets (we only crowdsourced 3,500 instances).

## Appendix E Human Annotation Details

We ran two different human annotations, one to evaluate our Verblm MC and another to manually design verb contrast sets. Figures 4 and 5 show the respective HIT UIs. We use Amazon Mechanical Turk interface to get a pool of annotators from native English speaking countries and with high approval rate, and pay them $15 hour on average which is above a minimum wage.

 



## Overview

Thanks for participating in this HIT! In this HIT, you'll be given an **video** and **5 candidate sentences**. Your task is to select the **best sentence** describing the video.

### Note:

* Please be forgiving of minor spelling errors.
* There might be more than one statement (or None) that matches the content. Try to do your best to choose the most plausible option.
* **Names** in text correspond to characters in movies, which could be used to disambiguate **different genders**. BUT, we do not expect you to determine if the character is doing the right action, and the correct answer should be clear without knowing the names.
* If you are not sure about your answer for the above reasons, you can check the **"not clear"** box

Figure 4: AMT UI for conducting human evaluation in the MC setting with contrast sets.





## References

* [1] M. A. Abadi, J. R. Barlow, and D. C. B. Call. The world: a survey of machine learning techniques for machine learning. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 35-46, 2007. ACM, 2008.
* [2] S. Choudhary, P. K. De Vries, and T. G. Vinyals. Learning to improve neural networks by reducing the number of hidden layers. In _Advances in Neural Information Processing Systems_, pages 93-108, 2009.
* [3] H. Chen, Y. Li, X. Zhang, and L. Wang. Deep residual learning for deep convolutional neural network. In _NIPS Conference on Computer Vision and Pattern Recognition_, volume 1, pages 1-10, 2016. IEEE, 2016.
* [4] W. Dong, Z. Liu, and Q. Sun. Deep residual learning for image recognition. In _CVPR International Conference on Computer Vision and Pattern Recognition_, volume 1, pages 1-10, 2016. IEEE, 2016.
* [5] W. Dong, Z. Liu, and L. Wang. Deep residual learning for image recognition. In _ECCV_, volume 1, pages 1-10, 2016. IEEE, 2016.
* [6] W. Dong, Z. Liu, and L. Wang. Deep residual learning for image recognition. In _ICCV Workshops on Computer Vision and Pattern Recognition_, volume 1, pages 1-10, 2016. IEEE, 2016.

[MISSING_PAGE_POST]