SlowGuess/NAACL_2022_OCR huggingface dataset:
The visual gate is dynamically computed using the contextualized video representations and is used to trim the video signal to the relevant bits. This helps in regulating the contribution of the two modalities for the final prediction as per the input. The concatenation represents the late-fusion of text-only embeddings and video-contextualized text embeddings. This merged representation is then sent to the classifier layer for classification i.e.  $\hat{Z} = \text{softmax}(W_c * S_{clf} + b_c)$ .
5.4 Sequence Labelling with Joint Video-Text Pretraining
In order to leverage joint modelling of video and text modalities through large-scale pretraining, we adapt the pretrained HERO (Li et al., 2020) and ClipBERT (Lei et al., 2021) for sequence tagging.
HERO. For each sample in video-based intent detection, we send the video clip and the corresponding subtitle for context as well as query, as input to HERO.  $V_{cross}$  represents the cross-contextualized frame embeddings from the Cross-modal Transformer module, which is then concatenated with query embeddings  $W_{emb}^{q}$  before being sent to the Temporal Transformer  $f_{temp}$  in HERO
Type	Model	Video Embeddings	Tool Intents	Creative Intents
P	R	F	P	R	F
Unimodal	CRF	-	0.43	0.55	0.48	0.17	0.1	0.13
RoBERTa	-	0.53	0.65	0.58	0.21	0.39	0.27
GPT2	-	0.41	0.67	0.51	0.12	0.25	0.17
Multimodal(Unimodal Pretraining)	RoBERTa + Naive Fusion	3D ResNext	0.48	0.62	0.54	0.19	0.48	0.27
2D ResNet	0.54	0.65	0.59	0.23	0.38	0.28
RoBERTa + Late Fusion	SlowFast	0.58	0.65	0.61	0.22	0.40	0.29
3D ResNext	0.55	0.64	0.59	0.23	0.41	0.29
2D ResNet	0.58	0.62	0.60	0.24	0.26	0.25
SlowFast	0.60	0.66	0.62	0.24	0.41	0.30
Multimodal Pretraining	HERO	2D ResNet + SlowFast	0.57	0.65	0.61	0.23	0.43	0.30
HERO + Late Fusion	2D ResNet + SlowFast	0.62	0.61	0.62	0.30	0.31	0.30
ClipBERT	-	0.53	0.66	0.59	0.19	0.35	0.25
ClipBERT + Late Fusion	-	0.54	0.67	0.60	0.21	0.29	0.27
Table 4: Partial-match based results on the test split of the Behance Intent Discovery dataset.
for global contextualization. Thus, the output is:
$$
S _ {t e m p} = f _ {t e m p} ([ V ^ {c r o s s}; W _ {e m b} ^ {q} ])
$$
$$
S _ {o u t} = S _ {t e m p} [ N _ {v}: (N _ {v} + N _ {t}),: ]
$$
where  $N_v$  and  $N_t$  are the number of frames and tokens in video and query respectively. The output of  $f_{temp}$  is masked to select the representations pertaining to the query only. In the naive fusion setting,  $S_{out}$  is then sent to the classifier layer.
ClipBERT. Similarly, the output  $S_{out}$  from the Cross-modal Transformer  $f_{cross}$  in ClipBERT is masked and sent to the classifier layer for prediction i.e.  $S_{out} = f_{cross}([V;W_{emb}^{q}])[:N_{t},:]$ .
Late Fusion. We integrate the late fusion approach into HERO and ClipBERT as follows:
$$
S _ {g a t e} = \operatorname {s i g m o i d} \left(W _ {g} * S _ {o u t} + b _ {g}\right)
$$
$$
S _ {c l f} = \left[ S _ {g a t e} * S _ {o u t}; W _ {e m b} ^ {q} \right]
$$
where the visual gate is computed as in Sec. 5.3 (see Fig. 4) and  $S_{clf}$  is sent to the classifier layer.
Evaluation. Since the transcribed phrases in Behance Livestreams are the result of an automatic speech recognition (ASR) system, the exact span match metrics might be distorted by ASR errors. Hence, we use a more lenient  $75\%$  partial match-based Precision/Recall/F-score metric i.e., if there is more than  $75\%$  overlap between the ground truth and predicted span, we consider it as a match.
Video Representations. We experiment with 3D ResNext-101 (Xie et al., 2017) ( $fps = 6$ ), SlowFast (Feichtenhofer et al., 2019) (clip length  $= 2s$ ) and 2D ResNet-152 (He et al., 2016) (clip length  $= 2s$ ) following preprocessing steps in Li et al. (2020).
Models. We use the RoBERTa  $LARGE$  (Liu et al., 2019) models for the unimodal experiments, as well as the multimodal experiments that are based on unimodal pretrained models. We use the pretrained HERO (Li et al., 2020) and ClipBERT (Lei et al., 2021) in the remaining experiments; their language encoders are initialized from pretrained RoBERTa  $B A S E$  and BERT  $B A S E$  (Devlin et al., 2019) models. Each model is trained end-to-end using fully-supervised training and is subjected to grid-search based hyper-parameter optimization. The best checkpoints are selected based on overall F-Score. See Appendix for bounds.
In this section, we discuss results from various models on the Behance Intent Discovery dataset (see Table 4).
The text baselines. Starting with the text-only baselines, we see the best performance from the RoBERTa models, i.e.,  $58\%$  and  $27\%$  partial match F-scores on the tool and creative intents, respectively (rows 2 and 3 in Table 4). Notably, the tool intent predictor is biased with high recall but low precision performance i.e. it retains too many candidates, many of which do not correspond to any intents. These results also demonstrate that large pretrained language models like RoBERTa and GPT2 struggle to comprehend the abstract ideas represented in creative intents.
The Naïve Fusion models. The Naïve Fusion approach with pretrained RoBERTa yields up to  $2\%$  improvement over the text-only baselines. In some cases, such as the 3D ResNext representations, this approach degrades the performance, especially in
the harder creative intent set. We attribute this to the difference in informativeness between the text and the video signal, as discussed in Sec. 5.2.
The Late Fusion models. With the Late Fusion approach, we see significant improvements in almost all cases. Compared to the corresponding Naïve Fusion models, Late Fusion models mainly improve precision for tool intents. This result supports our hypothesis that the video signal is most useful as a gate to filter out non-intent candidates from the textual signal. The SlowFast representations prove especially beneficial for creative intents, as seen in row 9 in Table 4. With the use of multimodal pretrained models like HERO and ClipBERT, we observe significant improvements in prediction of tool intents and smaller improvements for creative intents with a simple adaptation of the prediction head for sequence labelling (see Sec. 5.4). HERO uses video representations from pretrained encoders while ClipBERT operates on raw videos; both approaches work well with the software-based video domain yielding up to  $3\%$  and  $1\%$  improvement on tool intents respectively (rows 10, 12 in Table 4) over the unimodal RoBERTa models. Larger improvements are seen from further augmenting these models with late fusion i.e.  $1\%$  improvement on tool intents (rows 11, 12 in Table 4). The late fusion RoBERTa model using SlowFast features (row 9 in Table 4) performs best for creative intents, with  $3\%$  improvement over the text-only baseline.
We see similar trends from experiments on the validation set of the Behance Intent Discovery dataset. See results in Appendix.
8 Analysis & Discussion
In this section, we perform qualitative analysis of the late fusion approach and examine the effect of video clip length. We also discuss a semiautomated approach to creating annotations for intent extraction and use the data in combination with manual annotations for improved results. See Appendix for more analyses.
8.1 Qualitative Analysis
In order to understand the inner workings of the late fusion architecture, we examine the cross-attention and visual gate modules of the RoBERTa+Late Fusion model trained with 2D ResNet features. Each row of the attention score matrix  $M \in R^{n \times f}$  (for  $n$  tokens and  $f$  video segments) in text-to-video
Figure 5: Visualization of (a) temporal attention over video segments from 12 attention heads and, (b) mean  $\pm$  SD of visual gate values for each token (blue for intent span), using the RoBERTa+Late Fusion model.
I am wondering if I should actually paint it in or I should try.
Figure 6: Wrong predictions (red) from unimodal RoBERTa which are solved by adding video signal.
My workflows change a lot of noise..
cross-attention module corresponds to the temporal attention over video clips (represented by a sequence of ResNet feature vectors) for a given token. We plot this score matrix for the 12 attention heads in the RoBERTa model in Figures 5(a) and 7(a). The attention heads are activated in the intent region suggesting a strong interaction between two modalities in important segments of the video.
To understand how the video signal helps the prediction, we first plot the mean and standard deviation of visual gate values  $(S_{gate})$  for each token in Figures 5(b) and 7(b). Results show that the visual gate preferentially relies on the video modality for tokens outside the intent span. Furthermore, in Fig. 6, we show example phrases where the text
(b)
Figure 7: Visualization of (a) temporal attention over video segments from twelve attention heads and (b) mean and standard deviations for the distribution of visual gate values for each token, using the best RoBERTa+Late Fusion model.
only model classifies wrongly as intent while the joint model does not. The phrases themselves appear to be intent but the lack of action in the visual frame indicates that these are chit-chat interactions. Both analyses support our hypothesis that the late fusion model utilizes the video signal to filter intent candidates and improve precision.
8.2 Video Clip Length
As we discuss in Sec. 4, the video clip durations for the tool and creative intents are not specified. We observe that the intended action can span anywhere between 1 second to several minutes. Longer clip lengths are relevant for many creative intents like "make it into something fantasy", "add the arm to this little guy", etc. Hence, we experiment with various clip lengths (10, 20 and 60 secs), but find that larger clip lengths do not lead to further improvements. In fact, with 60 second clips the performance of RoBERTa+Late Fusion model drops below the performance of text-only RoBERTa. This issue could be alleviated with long-range video understanding models (Sener et al., 2020).
8.3 Semi-automated Intent Annotations
Since manual annotation of procedural intents is time-intensive and expensive, we explore a semiautomatic pipeline for creation of intent annotations. The Behance Livestreams corpus contains tool timelines for each livestream, which enumerates the tools used within the software at different points in the livestream. We compute the tfidf scores for co-occurrence of 896, 287 action-object phrases (from dependency parses of sentences) and corresponding tools in the tool timelines, in order to find the phrases that are frequently used for describing particular tool actions, such as "grab the smudge tool". After filtering the phrases for those with high tfidf scores, the pool of intent candidates was further cleaned manually, resulting in a final set of 3,697 tool intent candidates. Using this pool of candidates, 24,300 phrases from the Behance Livestreams corpus were identified as tool intent samples. Since it is not straightforward to extract creative intents using similar methods, we first identified key phrases for creative intents from the set of action-object phrases with high term frequency. We then subjected it to manual cleaning (two annotators per sample;  $\kappa = 0.986$ ) followed by embedding similarity to select creative intents (see Appendix for full pipeline). Using this method, we recovered 7,135 phrases containing creative intents.
We use these semi-automatically collected annotations as additional training data in our experiments with Late Fusion RoBERTa models. Since the manually annotated Behance Intent Discovery dataset is skewed towards negative samples i.e.  $< 25\%$  samples contain intent, we balance the training data by adding 5,000 samples (containing tool or creative intents) from the aforementioned semi-automatically annotated dataset to it. With this balanced data, we see up to  $2\%$  improvement in the Late Fusion RoBERTa models. See Appendix.
In this paper, we explore the novel task of video-based multimodal intent discovery. We present the unlabelled Behance Livestream corpus consisting of instructional videos for software tools, and the Behance Intent Discovery dataset annotated with tool and creative intents. We propose a late-fusion approach for integration of the video signal with the text signal in a controlled manner for this task, and show significant improvements with unimodal and multimodal pretrained models.
We would like to thank Tracy King for her detailed feedback and Hailin Jin for making the Behance transcript available. We would also like to thank the reviewers for their useful feedback. This work was partially done while AM was interning at Adobe Research and later extended at UNC, where it was supported by ARO Award W911NF2110220 and DARPA KAIROS Grant FA8750-19-2-1004. The views contained in this article are those of the authors and not of the funding agency.
11 Ethics/Broader Impacts
From an ethics standpoint, we provide a detailed overview of the methods used to create the Behance Livestreams corpus and Behance Intent Discovery dataset in Sec. 4 and more details in the Appendix. We also provide some analyses of the data in Table 3. All of the language data consists of simple English sentences. The dataset comprises livestreamed video tutorials by users of the Behance platform. Behance users grant full usage rights of their content and agree to not hold copyright claims on content in the livestreams videos or transcripts. This content is being made available for free distribution for academic research purposes only and does not allow for redistribution. Aside from the name of the instructor in each video (which is public information), real names of livestream session users or other identifying information does not appear in any of the transcripts. We provide full descriptions of the models used in this paper in Sec. 5. Detailed hyperparameters and bounds for hyperparameter search are included in the Appendix.
Video-based intent discovery serves to enhance the information exploration experience of users on any video-based platform. Since we focus on extracting procedural intent relevant to the goal of the video and in the software domain, we do not anticipate this technology to cause any harm to users, or have any unintended consequences.
Swati Agarwal and Ashish Sureka. 2016. Characterizing linguistic attributes for automatic classification of intent based racist/radicalized posts on twitter micro-blogging website. In European Intelligence and Security Informatics Conference (EISIC) 2016.
Alessio Palmero Aprosio, Stefano Menini, and Sara Tonelli. 2020. Creating a multimodal dataset of
images and text to study abusive language. arXiv preprint arXiv:2005.02235.
Ruichu Cai, Binjun Zhu, Lei Ji, Tianyong Hao, Jun Yan, and Wenyin Liu. 2017. An cnn-lstm attention approach to understanding user query intent from online health communities. In 2017 iiee international conference on data mining workshops (icdmw), pages 430-437. IEEE.
Ying-Yu Chen and Shu-Kai Hsieh. 2020. An analysis of multimodal document intent in instagram posts. In Proceedings of the 32nd Conference on Computational Linguistics and Speech Processing (ROCLING 2020), pages 193-207.
Zhiyuan Chen, Bing Liu, Meichun Hsu, Malu Castellanos, and Riddhiman Ghosh. 2013. Identifying intention posts in discussion forums. In Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1041-1050.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and-language tasks via text generation. In ICML.
Alice Coucke, Alaa Saade, Adrien Ball, Theodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190.
Deborah A Dahl, Madeleine Bates, Michael K Brown, William M Fisher, Kate Hunicke-Smith, David S Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the atist task: The atis-3 corpus. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202-6211.
C Ailie Fraser, Joy O Kim, Alison Thornsberry, Scott Klemmer, and Mira Dontcheva. 2019. Sharing the studio: How creative livestreaming can inspire, educate, and engage. In Proceedings of the 2019 on Creativity and Cognition, pages 144-155.
Vineet Gupta, Devesh Varshney, Harsh Jhamtani, Deepam Kedia, and Shweta Karwa. 2014. Identifying purchase intent from social posts. In *Eighth International AAAI Conference on Weblogs and Social Media*.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, and Chuang Gan. 2020. Location-aware graph convolutional networks for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11021-11028.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatiotemporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758-2766.
Menglin Jia, Zuxuan Wu, Austin Reiter, Claire Cardie, Serge Belongie, and Ser-Nam Lim. 2021. Intentonomy: a dataset and study towards human intent understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12986-12996.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in Neural Information Processing Systems, 33.
Junyeong Kim, Minuk Ma, Kyungsu Kim, Sungjin Kim, and Chang D Yoo. 2019. Progressive attention memory network for movie story question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8337-8346.
Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. 2017. Deepstory: video story qa by deep embedded memory networks. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 2016-2022.
Christoph Kofler, Martha Larson, and Alan Hanjalic. 2014. Intent-aware video search result optimization. IEEE transactions on multimedia, 16(5):1421-1433.
Christoph Kofler, Martha Larson, and Alan Hanjalic. 2016. User intent in multimedia search: a survey of the state of the art and future challenges. ACM Computing Surveys (CSUR), 49(2):1-37.
Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky, and Ajay Divakaran. 2019. Integrating text and image: Determining multimodal document intent in instagram posts. In Proceedings of the
2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4622-4632.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331-7341.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. Tvqa: Localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1369-1379.
Jie Lei, Licheng Yu, Tamara Berg, and Mohit Bansal. 2020a. What is more likely to happen next? video-and-language future event prediction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8769-8784.
Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020b. Tvr: A large-scale dataset for video-subtitle moment retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 447-463. Springer.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020. Hero: Hierarchical encoder for video+ language omni-representation pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2046-2065.
Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu. 2020. Violin: A large-scale dataset for video-and-language inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10900-10910.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353.
Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. 2017. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6884-6893.