SlowGuess/NAACL_2022_OCR huggingface dataset:
S Meyer. 2015. LinkedIn's blockbuster deal with lynda. com: What it means to the online learning industry/s. Retrieved February, 15:2016.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-2640.
Saiteja Nalla, Mohit Agrawal, Vishal Kaushal, Ganesh Ramakrishnan, and Rishabh Iyer. 2020. Watch hours in minutes: Summarizing videos with user intent. In European Conference on Computer Vision, pages 714-730. Springer.
Mingtao Pei, Yunde Jia, and Song-Chun Zhu. 2011. Parsing video events with goal inference and intent prediction. In 2011 International Conference on Computer Vision, pages 487-494. IEEE.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.
Fadime Sener, Dipika Singhania, and Angela Yao. 2020. Temporal aggregate representations for long-range video understanding. In European Conference on Computer Vision, pages 154-171. Springer.
Aaron Smith, Skye Toor, and Patrick Van Kessel. 2018. Many turn to youtube for children's content, news, how-to lessons. Pew Research Centre, 7.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473.
Zineng Tang, Jie Lei, and Mohit Bansal. 2021. Decembert: Learning from noisy instructional videos via dense captions and entropy minimization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2415-2426.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4631-4640.
Hans Van der Meij. 2017. Reviews in instructional video. Computers & education, 114:164-174.
Nikhita Vedula, Nedim Lipka, Pranav Maneriker, and Srinivasan Parthasarathy. 2019. Towards open intent discovery for conversational text. arXiv preprint arXiv:1904.08524.
Jinpeng Wang, Gao Cong, Xin Wayne Zhao, and Xiaoming Li. 2015. Mining user intents in twitter: A semi-supervised approach to inferring intent categories for tweets. In Twenty-Ninth AAAI Conference on Artificial Intelligence.
Di Wu, Liang Ding, Fan Lu, and Jian Xie. 2020. Slotrefine: A fast non-autoregressive model for joint intent detection and slot filling. EMNLP.
Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and S Yu Philip. 2018. Zero-shot user intent detection via capsule neural networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3090-3099.
Saining Xie, Ross Girshick, Piotr Dolkar, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500.
Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura. 2020. Bert representations for video question answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1556-1565.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. 2020. Clevrer: Collision events for video representation and reasoning. In ICLR.
Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3342-3352.
Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. 2020. Learning 2d temporal adjacent networks for moment localization with natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12870-12877.
Linchao Zhu and Yi Yang. 2020. Actbert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8746-8755.
For the semi-automatically created annotations described in Sec. 8.3, we empirically select a window of 10 seconds for computing the scores and retain intent phrases with a term frequency of 5 or higher in the corpus and tfidf scores of 0.3 or higher with one or more tools. See the full semi-automated pipeline of dataset creation in Fig. 8.
Data Processing Pipeline
Figure 8: Semi-automated data processing pipeline.
Model	Video Embeddings	Acc.
3-Layer MLP	3D ResNext	0.706
SlowFast	0.743
3D ResNext-101	-	0.762
Table 5: Results from pilot experiments on usefulness of video modality for multimodal intent discovery.
Probing Experiments. We conducted pilot experiments to probe the usefulness of video signals for intent detection in Behance Livestreams. We prepare a video-only classification dataset for intent classification containing 3,000 samples each for creative, tool and no-intents. We use off-the-shelf ResNext features with a 3-layer MLP classifier as well as finetune ResNext on this task. Using only pretrained video representations, the 3-layer MLP classifier was able to detect the presence of an intent with  $70\%$  and  $74\%$  accuracy using 3D ResNext and SlowFast features, respectively, while  $66\%$  being the chance baseline. With finetuned ResNext, the accuracy improved to  $76\%$ . However, the accuracy of classifying between tool and creative intents remained close to random for all models, suggesting the complex nature of creative intents. See Table 5.
For HERO and ClipBERT models, we use the recommended hyperparameters for finetuning in their Github repository. For RoBERTa-based models, see the hyperparameters common to all models in Table 8. We performed grid-search based optimization of the variable hyperparameters using the bounds in Table 8. The best performing batch size for all models was found to be 32.
See partial match results for the validation split of Behance Intent Discovery in Table 6.
D.1 Finetuned Video Representations
We see large improvements with sparsely-sampled 2D ResNet video embeddings (see Table 4 which are extracted from ResNet pretrained on the ImageNet dataset. This begs the question, if larger improvements can be had by finetuning the feature extractors on the domain of Behance Livestreams. To facilitate this, we create a dataset of 10,000 images containing snapshots of video livestreams and classified them into one of 50 tool categories using the tool timeline. We finetune ResNet-152 on this dataset with a resulting classification accuracy of  $47\%$ . We use the finetuned ResNet to extract sparsely sampled video embeddings and re-run the late fusion experiment with RoBERTa. We see  $2\%$  improvement for tool intents and  $1\%$  drop in performance on creative intents. This suggests that finetuning feature extractors on the target domain can be beneficial for low-level intents.
D.2 Semi-automated Intent Annotations
As discussed in Sec. 8.3, we use semi-automatically collected annotations as additional training data in our experiments with Late Fusion RoBERTa models. Since the manually annotated Behance Intent Discovery dataset is skewed towards negative samples i.e.  $< 25\%$  samples contain intent, we balance the training data by adding 5,000 samples (containing tool or creative intents) from the aforementioned semi-automatically annoated dataset to it. With this balanced data, we see up to  $2\%$  improvement in the Late Fusion RoBERTa models as seen in Table 7. However, with increasing amount of
$^{6}$ https://github.com/linjieli222/HERO
<sup>7</sup>https://github.com/jayleicn/ClipBERT
Type	Model	Video Embeddings	Tool Intents	Creative Intents
P	R	F	P	R	F
Unimodal	CRF	-	0.36	0.57	0.44	0.16	0.09	0.12
RoBERTa	-	0.48	0.78	0.59	0.34	0.52	0.41
GPT2	-	0.40	0.61	0.48	0.15	0.19	0.15
Multimodal(Unimodal Pretraining)	RoBERTa + Naïve Fusion	3D ResNext	0.46	0.75	0.57	0.3	0.34	0.32
2D ResNet	0.47	0.77	0.59	0.34	0.64	0.44
SlowFast	0.52	0.76	0.62	0.36	0.56	0.44
RoBERTa + Late Fusion	3D ResNext	0.48	0.78	0.59	0.34	0.52	0.41
2D ResNet	0.48	0.78	0.59	0.34	0.52	0.41
SlowFast	0.54	0.77	0.62	0.38	0.60	0.44
Multimodal Pretraining	HERO	2D ResNet + SlowFast	0.51	0.72	0.6	0.34	0.31	0.33
HERO + Late Fusion	2D ResNet + SlowFast	0.56	0.73	0.63	0.37	0.53	0.43
ClipBERT	-	0.53	0.71	0.61	0.28	0.47	0.35
ClipBERT	-	0.56	0.73	0.63	0.31	0.48	0.37
Table 6: Partial-match based results on the validation split of the Behance Intent Discovery dataset.
Model	Video Embeddings	Dataset	Tool Intents	Creative Intents
P	R	F	P	R	F
RoBERTa + Late Fusion	SlowFast	20K Manual Only	0.60	0.66	0.62	0.24	0.41	0.30
SlowFast	20K Manual + 5K Semi	0.59	0.69	0.64	0.20	0.44	0.29
SlowFast	20K Manual + 10K Semi	0.41	0.70	0.51	0.19	0.43	0.25
Table 7: Partial-match based results on the test split of the Behance Intent Discovery dataset using manual annotations and semi-automatically created annotations.
Hyperparameter	Value
Common Hyperparameters
#Training Epochs	10
Max Gradient Norm	1.0
Weight Decay	0.0
Max. Sequence Length	70
Seed	0
Warmup Steps	200
LR Decay	Linear
Optimizer	AdamW (ε=1e-8, β1=0.9, β2=0.98)
Search Bounds
Learning Rate	[1e-4, 1e-5, 5e-5, 1e-6, 5e-6]
Batch Size	[8, 16, 32]
Table 8: Common Training Hyperparameters and Search Bounds for RoBERTa models
semi-automatically data, we drastic decline in the precision of the model for both tool and creative intents (see row 3 in Table 7). With the use of better methods for filtering out the useful signal from the noisy data, there might be better results with semi-automatically created annotations. This line of research is important because it promotes scalable annotations which can cover a diverse population of livestreamers from many livestream videos.