SlowGuess/NAACL_2022_OCR huggingface dataset:
for video retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 214-229. Springer.
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020. Evaluating models' local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020.
Siddhant Garg and Goutham Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classification. ArXiv, abs/2004.01970.
Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650-655.
Lisa Anne Hendricks and Aida Nematzadeh. 2021. Probing image-language transformers for verb understanding. arXiv preprint arXiv:2106.09141.
Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin W. Wilson. 2017. Cnn architectures for largescale audio classification. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 131-135.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Robin Jia, Aditi Raghunathan, Kerem Goksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions. In EMNLP/IJCNLP (1).
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331-7341.
Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu. 2021. Adversarial vqa: A new benchmark for evaluating the robustness of vqa models. ArXiv, abs/2106.00245.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. Bert-attack: Adversarial attack against bert using bert. In *The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2021. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2104.08860.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448.
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual representations from uncurated instructional videos. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9876-9886.
Antoine Miech, Ivan Laptev, and Josef Sivic. 2018. Learning a text-video embedding from incomplete and heterogeneous data. arXiv preprint arXiv:1804.02516.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2630-2640.
John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In EMNLP.
Nikola Mrkšić, Diarmuid Řeaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of HLT-NAACL.
Jae Sung Park, Trevor Darrell, and Anna Rohrbach. 2020. Identity-aware multi-sentence video description. In European Conference on Computer Vision, pages 360-378. Springer.
Jesús Andrés Portillo-Quintero, José Carlos Ortiz-Bayliss, and Hugo Terashima-Marín. 2021. A straightforward framework for video retrieval using clip.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Marco Tulio Ribeiro, Tongshuang Sherry Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. In ACL.
Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Chris Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2017. Movie description. International Journal of Computer Vision.
Karin Kipper Schuler. 2005. VerbNet: A broad-coverage, comprehensive verb lexicon. University of Pennsylvania.
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. 2019. Cycle-consistency for robust visual question answering. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6642-6651.
Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurelie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. 2017. Foil it! find one mismatch between image and language caption. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 255-265.
Atousa Torabi, Niket Tandon, and Leon Sigal. 2016. Learning language-visual embedding for movie understanding with natural-language. arXiv:1609.08124.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv, abs/1706.03762.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. 2018. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288-5296.
Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018. A joint sequence fusion model for video question answering and retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pages 471-487.
Male Nouns	Female Nouns
man → woman	woman → man
men → women	women → men, guys
boy → girl	girl → boy, guy
boys → girls	girls → boys, guys
guy → woman, girl	lady → man, guy
guys → women, girls, ladies	ladies → men, guys
Table 6: List of gender sensitive words mapped to a different gender. Note, that singular and plural form is maintained.
A Contrast Set Construction
Here, we provide more details on construction of each contrast set.
A.1 Gender Contrast Sets
Table 6 shows the mapping of gender-sensitive words. We use these rules to swap only a single word in the sentence. This is to guarantee that swapping gender leads to different semantics (e.g. man and woman walk together  $\rightarrow$  woman and man walk together both apply to the same video if all words are swapped). If there are more than one possible mappings, we randomly sample one from a uniform distribution. Lastly, we swap all gender-sensitive pronouns that have the same gender as original noun. These contrast sets are used for the MSR-VTT dataset (Xu et al., 2016).
A.2 Person ID Contrast Sets
The first character ID in a sentence is replaced by a different character ID that appears in the same movie and has the same gender. Among all the candidates, the manipulated ID is sampled from a uniform distribution. The following character IDs in the same sentence have uniform chance of being kept or swapped using the same strategy. These contrast sets are used for the LSMDC-IDs dataset.
A.3 Verb Contrast Sets
Attack Selection We use Spacy to get the POS tags, and find verb phrases that match a list of predefined patterns (verb; verb + preposition).
Candidate Generation We use T5 model and performed beam search (beam size  $= 50$ ) to generate  $K = 50$  multi-word candidates.
Candidate Constraints We keep a candidate if the lemmatized verbs  $^5$  in it appeared more than 30 times in the training set. For fluency, we calculate perplexity score of original and manipulated sentence using GPT2-XL (Radford et al., 2019), which we call  $\mathrm{ppl}_o$  and  $\mathrm{ppl}_m$ . We calculate the normalized difference of perplexity scores  $\mathrm{ppl}_{\mathrm{diff}} = \frac{\mathrm{ppl}_o - \mathrm{ppl}_m}{\mathrm{ppl}_o}$  to remove a candidate that is less plausible than the original. Specifically, candidates are kept if  $\mathrm{ppl}_{\mathrm{diff}} < 0.6$ , or  $\mathrm{ppl}_{\mathrm{diff}} < 1.4 \cap \mathrm{ppl}_m < 750$ . Lastly, the semantic inconsistency constraints are satisfied if the word embedding (Mrkšić et al., 2016) of the lemmatized verbs in the candidate and original sentence have cosine similarity score lower than 0.4, and the sentence embeddings (Reimers and Gurevych, 2019) have cosine similarity score lower than 0.8.
B Human vs Machine Generated Verb Contrast Sets
Figure 3 shows a distribution of machine and human generated verb contrast sets. Each instance is the number of lemmatized verbs divided by total number of verbs in the contrast sets. We see that machine generated contrast set is more skewed to the left, and doesn't share the same distribution of verbs as in the human generated contrast sets. (e.g. human contrast sets have more occurrences of cry and throw in MSR-VTT, and jump and drop in LSMDC-IDs). Despite the difference, note that models have similar performances in both contrast sets.
- MMT (Gabeur et al., 2020): We use the following features extracted from video<sup>6</sup>: motion from S3D (Xie et al., 2018), audio from VGGish (Hershey et al., 2017), scene embeddings, face, OCR, Speech, and Appearance. We refer to Miech et al. (2018); Gabeur et al. (2020) for more details about the features.
For MSR-VTT, we use the released checkpoint from their code<sup>7</sup>, which is pre-trained on HowTo100M dataset (Miech et al., 2019) and further finetuned on MSR-VTT.
$^{5}$ https://www.nltk.org/_modules/nltk/stem/wordnet.html
https://github.com/albanie/collaborative-experts
<sup>7</sup>https://github.com/gabeur/mmt
(a) Verb contrast sets in MSR-VTT
(b) Verb contrast sets in LSMDC-IDs
Figure 3: Distribution of machine and human generated verb contrast sets in the MSR-VTT and LSMDC-IDs dataset. Each instance is the ratio of lemmatized verbs within each contrast set.
For LSMDC-IDs which needs re-training, we used their finetuning code for LSMDC dataset (Rohrbach et al., 2017). The model is trained with max margin ranking loss on 1 Nvidia RTX-6000 GPU for 12 hours. Hyperparameter search was done to find margin of 0.05, batch size of 32, and Adam optimizer (Kingma and Ba, 2015) with learning rate  $5e^{-5}$ . The best model was selected by the video-to-text retrieval performance with Recall@1. We found training from scratch performs better than using pre-trained model. This has been also observed by Gabeur et al. (2020) for the LSMDC dataset.
- MMT-CLIP: We replace the appearance features in MMT with frozen CLIP ViTB/32 features and train with the same architecture.
- CLIP zero-shot: In (Portillo-Quintero et al., 2021) CLIP(ViTb-32) (Radford et al., 2021) features are aggregated via mean pooling to approximate video representation. This video representation and text embedding from CLIP are combined to perform retrieval and MC in a zero shot manner.
- CLIP4CLIP (Luo et al., 2021): We use the hyperparameters from the finetuning code to reproduce their results. We use mean pooling for the similarity calculator and CLIP model is initialized with ViTB-32 weights. The model was trained with 4 Nvidia RTX-6000 GPUs for 5 epochs (48gpu hours). The best model was selected by using Recall@1 in video-to-text retrieval.
<sup>8</sup>https://github.com/ArrowLuo/CLIP4Clip
- CLIP2Video (Fang et al., 2021): We used the released checkpoint on MSR-VTT using their code base<sup>9</sup>. This model is not used for LSMDC-IDs because finetuning code was not provided. CLIP model is initialized with ViTB-32 weights.
D Multiple Choice Details
Here we provide more details about our evaluation data. Note, that we use 5 text candidates (1 positive and 4 negative) for all multiple choice (MC) settings.
D.1 MSR-VTT
We use the standard train/val/test split in MSRVTT dataset (Xu et al., 2016).
- Retrieval: 1,000 ground truth video-text pairs in the test set (Yu et al., 2018).
- Random MC: 2,990 videos and all negative options are drawn randomly from other videos (Yu et al., 2018).
- Gender MC: 2,477 video-text instances. Using the original descriptions from Random MC, a single negative is drawn from gender contrast sets to replace one of the options in Random MC (the remaining 3 are kept). Note, that not all videos involved people or contained gender-sensitive words in descriptions, hence some instances are filtered.
- VerbLM MC: 2,554 video-text instances. Constructed using the same strategy as in Gender MC but a single negative is drawn from verb contrast sets generated by language models. Instances are filtered when there are no valid verb contrast sets satisfying constraints in Section A.3.
-  $\operatorname{Verb}_H$  MC: 2,554 video-text instances. We use the instances in  $\operatorname{Verb}_{LM}$  MC, and a negative is drawn from human designed verb contrast sets.
D.2 LSDMC-IDs
We define a new split using LSMDC-ID descriptions with character IDs (proper names) (Park et al., 2020). Note, that Rohrbach et al. (2017); Park et al. (2020) use development and test sets where videos come from distinct movies than the training data,
meaning that IDs in test data are not seen in training. To overcome this issue, we split their training descriptions into  $80\% / 10\% / 10\%$  ratio to create new training/validation/test sets that share the same movies and identities across splits.
- Retrieval: 7,010 ground truth video-text pairs.
- Random MC: 7,010 videos, negative text options drawn randomly from different videos but the same movie.
- ID MC: 7,010 video-text instances. We replace one negative in Random MC with the one from ID contrast sets.
-  $\mathsf{Verb}_{LM}$  MC: 7,010 video-text instances. We replace one negative in Random MC with one from the language model generated verb contrast sets.
-  $\operatorname{Verb}_H$  MC: 3,500 video-text instances. We replace one negative in Random MC with one from the human designed verb contrast sets (we only crowdsourced 3,500 instances).
E Human Annotation Details
We ran two different human annotations, one to evaluate our  $\mathrm{Verb}_{\mathrm{LM}}$  MC and another to manually design verb contrast sets. Figures 4 and 5 show the respective HIT UIs. We use Amazon Mechanical Turk interface to get a pool of annotators from native English speaking countries and with high approval rate, and pay them $15 hour on average which is above a minimum wage.
We include additional information on the MSR-VTT (Xu et al., 2016) and LSMDC (Rohrbach et al., 2017) datasets. MSR-VTT contains diverse YouTube videos and corresponding crowdsourced descriptions in English language. LSMDC contains movie clips and associated descriptions from scripts or Audio Description, also in English. Both datasets are distributed for research use. The license, personally identifiable information (PII), and consent details of each dataset are in the respective papers. Since LSMDC contains clips from movies, some may contain nudity or violence, etc.
$^{9}$ https://github.com/CryhanFang/CLIP2Video
Instructions (click to expand)
In this HIT, you'll be given an video and 5 candidate sentences. Your task is to select the best sentence describing the video.
- Please be forgiving of minor spelling errors.
- There might be more than one statement (or None) that matches the content. Try to do your best to choose the most plausible option.
- Names in text correspond to characters in movies, which could be used to disambiguate different genders. BUT, we do not expect you to determine if the character is doing the right action, and the correct answer should be clear without knowing the names.
- If you are not sure about your answer for the above reasons, you can check the "not clear" box
Figure 4: AMT UI for conducting human evaluation in the MC setting with contrast sets.
$\bigcirc$  a boy explaining how to plug something into his computer
a group is dancing
$\bigcirc$  a boy explaining how to edit something into his computer
$\bigcirc$  asian man discusses technology in the younger generations
$\bigcirc$  two men on wave runner in ocean rescuing a surfer
Not Clear (More than one or None of the statement applies).
Optional feedback? (expand/collapse)
Overview [Update: 10/25/21]
Intro: AI systems have made a great progress in understanding what we see in the media, such as video, using natural language. One such application is to have a machine go through and find the best video that matches the text description.
But it is still not clear how "good" they are and can understand media in the same level as us.
In this HIT, we are interested if these machines can detect INCORRECT details in text that require more subtle understanding of the video. To do so, we will have to first collect such incorrect descriptions.
Task: You will be given a video and a original sentence describing the content.
Please MODIFY the HIGHLIGHTED WORD such that it is INCORRECT with respect to what happens in the video.
You are free to change other words, but the highlighted word should ALWAYS BE MODIFIED.
Your written sentence should have the following PROPERTIES:
(Actually) Incorrect: Written sentence should include details that are inconsistent with the video. While you are trying to write something to fool the machine, the original sentence should sound more plausible than the modified one.
[BAD] Sentence that is NOT Incorrect:
Person is fixing the computer.
Bad: Person is repairing the machine (try to avoid synonyms).
- Good: Person is breaking the computer (antonyms are great examples to use).
The dancers are performing on the stage.
- Bad: The dancers are dancing on the stage.
- Good: The dancers are singing on the stage (if they are not singing).
- Plausible: Written sentence should grammatically make sense and sound plausible. We should not be able to tell your sentence is incorrect without watching the video.
[BAD] Inplausible Examples:
A woman pushing her stroller.
Bad: A woman eating her stroller.
Good: A woman carrying her baby
- Bad: A dog is talking (usually dogs don't talk in real life).
- Good: A dog is running towards the owner. (if dog running is not shown in the video.)
NOTE: You are always welcome to modify multiple words, or even the entire sentence as long as the above properties are met.
More Examples (click to expand/collapse)
HIT:
Figure 5: AMT UI for collecting human-generated verb contrast sets.
- Please look at the examples before you begin!
- Please make sure to ALWAYS CHANGE the HIGHLIGHTED word.
You are encouraged to change additional words to make the sentence INCORRECT and still sound PLAUSIBLE (see requirements in instruction).
- Please AVOID changing the name of a person.
If a video is not played, please still do your best to write sentence incorrect from image.
Original Sentence: Jenko smirks and Schmidt beams.
Your Incorrect Sentence
Jenko smirks and Schmidt beams.
[Optional]
Check to write your sentence!
Your Incorrect Sentence (not required, but if you want to come up with more than one)