SlowGuess/NAACL_2022_OCR huggingface dataset:
Adyasha Maharana<sup>1</sup>, Quan Tran<sup>2</sup>, Franck Dernoncourt<sup>2</sup>, Seunghyun Yoon<sup>2</sup>, Trung Bui<sup>2</sup>, Walter Chang<sup>2</sup>, Mohit Bansal<sup>1</sup>
<sup>1</sup>University of North Carolina at Chapel Hill  <sup>2</sup>Adobe Research
{adyasha, mbansal}@cs.unc.edu, {qtran, dernonco, syoon, bui, wachang}@adobe.com
Individuals, educational institutions, and businesses are prolific at generating instructional video content such as "how-to" and tutorial guides. While significant progress has been made in basic video understanding tasks, identifying procedural intent within these instructional videos is a challenging and important task that remains unexplored but essential to video summarization, search, and recommendations. This paper introduces the problem of instructional intent identification and extraction from software instructional livestreams. We construct and present a new multimodal dataset consisting of software instructional livestreams and containing manual annotations for both detailed and abstract procedural intent that enable training and evaluation of joint video and text understanding models. We then introduce a multimodal cascaded cross-attention model to efficiently combine the weaker and noisier video signal with the more discriminative text signal. Our experiments show that our proposed model brings significant gains compared to strong baselines, including large-scale pretrained multimodal models. Our analysis further identifies that the task benefits from spatial as well as motion features extracted from videos, and provides insight on how the video signal is preferentially used for intent discovery. We also show that current models struggle to comprehend the nature of abstract intents, revealing important gaps in multimodal understanding and paving the way for future work. $^{1}$
Instructional videos have become increasingly ubiquitous as users generate diverse "how-to", DIY, and tutorial videos. A Pew Research Center 2018 survey of U.S. adult YouTube users (Smith et al., 2018) found that over half of surveyed users use
video content to learn how to do things they had not done before. These instructional videos convey both abstract and specific intent for physical tasks such as cooking where e.g., an abstract culinary intent is "let's bring out the flavor" and a detailed intent is "add a pinch of nutmeg". Thus, a key task in instructional video understanding is to discover both abstract and detailed intents. By discovering these intents, we can enable or improve important tasks such as semantic indexing of videos (Kofler et al., 2016), knowledge graph creation for video search and recommendations (Pei et al., 2011; Kofler et al., 2014), intent highlighting, and video summarization (Nalla et al., 2020).
An important domain with rich and complex examples of both abstract and detailed intent types are software training videos for creative tasks such as making photo or video effects. These types of software training videos have been shown to be effective for enhanced learning (Van der Meij, 2017) and are also considered a valuable resource in the era of online learning (Meyer, 2015). Existing video and phrase datasets such as HowTo100M (Miech et al., 2019) cover a wide variety of tutorials for visual tasks demonstrated by humans; however, software-based instructional videos are not a part of such corpora. Hence, in this paper, we present a new corpus of software-instructional videos containing instructional intents, which are derived from Behance Livestreams demonstrating the use of Adobe Photoshop software.
Intent detection has been well-studied in dialogue systems (Wu et al., 2020), but is less explored for instructional video content, especially emerging livestream content (Fraser et al., 2019). While rich in complex procedural instruction and intent, the interactive and social nature of livestreams poses unique challenges. Analyzing language features alone will provide only limited information about
$^{2}$ https://www_behance.net/live, https:// www.adobe.com/products/photoshop.html
$^{1}$ Code and data are available at https://github.com/adymaharana/VideoIntentDiscovery.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 476 - 489 July 10-15, 2022 ©2022 Association for Computational Linguistics
the actual instructional intent and the tools and commands used. For instance, the phrase "flipping the canvas" in "Are you flipping the canvas?" indicates a tool intent, but a closer look at the video clip reveals that it is in fact part of livestream chit-chat and does not take place on-screen. Incorporating both language and video modalities can enhance intent extraction of such ambiguous intents. Hence, in this paper, we present a new joint language-video intent discovery task and a multimodal dataset consisting of: Behance Intent Discovery, and the Behance Livestream video and transcript corpus that intents are found in. We frame intent discovery as a sequence labelling task; each sample in the intent discovery dataset contains a transcribed phrase annotated with token-level tags for abstract and detailed intents, and an associated video timestamp. Our goal is to predict the instructional intents from the transcript in each video.
To perform intent discovery within instructional videos, we propose a multimodal cascaded cross-attention model to predict both the abstract and detailed procedural intents that are present. Additionally, we use late fusion of multimodal embeddings to prevent the visual modality from overwhelming the textual signal, and show significant improvements on the video-based intent detection task using unimodal and multimodal pretrained models like HERO (Li et al., 2020). Further, we compare the performance of various video feature extractors as well as different video lengths, and present benchmark results on the proposed dataset. We find that discovery of tool intents benefit from sparsely-sampled spatial features while creative intents benefit from densely-sampled motion features. In the absence of motion features, most models struggle to utilize the video signal for identification of creative intents. Further, visualization of cross-attention and visual gate modules in the late fusion model suggests strong and meaningful interaction between the two modalities. Our contributions are:
- We introduce and explore the novel task of video-based multimodal intent discovery, and present an annotated dataset consisting of nearly 20K sentences from 66 livestreams for extraction of procedural intents from instructional videos.
- We release a large corpus of software-based instructional videos (2,049 sessions, 3,128 hours total), accompanied by timestamped transcripts, that can be used for pretraining
multimodal models.
- We propose the multimodal cascaded crossattention model and demonstrate the effectiveness of late fusion of multimodal embeddings in this task.
- We present empirical results for the proposed dataset using unimodal and multimodal approaches, and provide insights from analysis of modelling choices for future research.
Intent discovery has been widely studied in the context of dialog modelling and generation wherein it has been framed as a binary or multi-class classification problem. The SNIPS (Coucke et al., 2018) and ATIS (Dahl et al., 1994) datasets consist of concise single-sentence texts containing intents with constrained vocabulary and attributes. Several works have explored intent classification of internet posts in the context of racial/radicalized intent (Agarwal and Sureka, 2016), purchase intents (Gupta et al., 2014; Wang et al., 2015), discussion forums (Chen et al., 2013) and health queries (Cai et al., 2017). Vedula et al. (2019) propose open intent discovery with unconstrained vocabulary as a sequence tagging task. Using this framework, we present our dataset on instructional intents.
In the wake of exploding visual social-media content, several image-based multi-modal intent datasets have been previously proposed. Kiela et al. (2020); Aprosio et al. (2020) study abusive language and hateful intent in memes and photo posts. Jia et al. (2021) explore intent categories derived from social psychology and use object localization to integrate visual context in task models. Instagram posts are another interesting source for multimodal content (Chen and Hsieh, 2020; Kruk et al., 2019). We introduce the task of video-based multimodal intent discovery, which has been unexplored.
Several tasks have been proposed in the recent years to probe joint video and text understanding. Lei et al. (2018); Kim et al. (2017); Maharaj et al. (2017); Jang et al. (2017); Tapaswi et al. (2016) and Yi et al. (2020) introduce video-based question answering datasets created from various sources of creative visual content, i.e. movies, TV shows, GIFs etc. Lei et al. (2020b) and Lei et al. (2020a) propose the task of video-moment retrieval and next frame prediction respectively, based on query subtitles, while Liu et al. (2020) present the multimodal version of natural language inference. Early
Figure 1: Examples of the output predictions for candidate creative and tool intents given an instructional livestream video and its associated transcript.
models for performing these tasks involve combining pretrained image representations from sparsely sampled videos, and text encodings from pretrained encoders (Devlin et al., 2019; Liu et al., 2019) in architectures for modelling global-local interactions (Zhu and Yang, 2020; Yang et al., 2020), temporal localization (Kim et al., 2019; Zhang et al., 2020), graph-based reasoning (Huang et al., 2020) etc. More recent attempts involve pretraining models on large video+text corpora (Miech et al., 2019) and finetuning on downstream tasks (Sun et al., 2019; Cho et al., 2021; Tang et al., 2021; Lei et al., 2021; Luo et al., 2020). We explore late-fusion of video and text embeddings (Yu et al., 2020) for intent detection in pretrained and non-pretrained multimodal settings.
3 Problem Setting: Intent Discovery from Livestreams
In our setting, each video captures a Behance livestream in which an instructor demonstrates the steps needed to accomplish various image editing or compositing tasks. We specifically focus on rich creative instructional or tutorial livestreams that teach photo editing and compositing methods using an image application such as Photoshop, which consists of over 1,300 basic menu commands and subcommands, tool icons, panels, and galleries.
The livestream itself consists of a screencast of the instructor's application software, a smaller video window showing the instructor, a time-coded transcript of the dialog within the session, and a
Figure 2: Example of dependency structure of an intent.
Attribute	Statistics
#Sessions	3,356
Min/Max/Avg. session length	1/426/80 mins.
Min/Max/Avg. phrase length	1/142/8 words
Min/Max/Avg. #phrases per session	1/4,552/587
#Distinct tools in corpus	282
Table 1: Statistics of the Behance Livestreams corpus for sessions and transcribed phrases.
tool timeline which is a time-coded log of the specific application tools used during the livestream. Given the transcript of an instructional video, the video itself, and optional Tool Usage (TU) information from the tool timeline, Fig. 1 shows examples of the Creative Intentions (CI, shown at 01:36.16 and 01:37:09) and Tool Intentions (TI, shown at 01:36:21, 01:36:29 and 01:36:46) we seek to discover. Further, we wish to combine joint language and video knowledge to gain improvements in detecting candidate intents that are false positives such as the text at 01:37:09, which is only a parenthetical comment by the instructor.
4 Behavior Datasets
Dataset Collection. We first obtain 2,049 videos along with their transcripts and tool timelines from the Behance platform. The tool timeline contains a time-stamped record of the tools used in the software during the tutorial. The average session length is 80 minutes with an average of 587 transcribed phrases per session (see Table 1). The tool timelines contain 282 distinct tools with varying frequencies; Color, Select Brush, Select Layer are some of the most frequent ones. The instructional software-based domain of this dataset is significantly different from existing large corpora drawn from YouTube instructional videos (Miech et al., 2019) and TV content (Lei et al., 2018, 2020b), but it is an important learning resource. Hence, we include the unlabelled Behavior Livestreams corpus as an addition to the pool of video+text corpora that can be leveraged for continued pretraining of multimodal models and finetuning on downstream tasks relevant to software-based livestream videos.
In order to prepare the intent discovery dataset,
1:34:28.55 -> 1:34:33.45
Select the inverse and give a little bit of detail right to that edge.
1:48:14.45 -> 1:48:21.65
I am going to make a new layer and use a clipping mask again on the wood and I might use a textured brush.
3:40:48.59-->3:50:51.76
So I am blurring some of my groups.
Figure 3: Examples of tool intent (blue) and creative intents (magenta) from the Behance Intent Discovery dataset.
4:47:40.83 -> 4:47:48.12
I'm happy with that, so I merge it down and then we'll make a new layer and let's add some more plants and stuff.
Attribute	Training/Validation/Test
#Sentences	13989/2,105/3,917
#Tool intents	3,478/414/825
#Creative intents	674/106/189
#Livestream videos	54/6/6
we extract candidate intent phrases from the transcripts of the Behance Livestream corpus. Following Vedula et al. (2019), we define an intent as a text phrase consisting of: (i) an action word or phrase, which constitutes a definite task, goal or activity and (ii) an object, which represents those words or phrases that the action is going to act or operate upon. We generate the dependency graph of sentences, and extract the VERB node as action and the direct object of the VERB as the object, along with all other children nodes (see example in Fig. 2). Through manual analysis, we identified two major categories of meaningful intent: tool and creative. Tool intents are low-level intents that can be typically mapped to a single tool in the software. Creative intents are abstract intents used to describe a high-level creative goal that consist of a complex set of actions or tool intents. For instance, in Fig. 3, "make a new layer" is a tool intent that can be mapped to the tool Create Layer, while "add more plants and stuff" is a creative intent. All other intents in the corpus, predominantly from chit-chat statements, are irrelevant to our task. We frame the task of intent discovery as a sequence-tagging problem and tag the intent phrases within each sentence with IOB (inside, outside, beginning) span annotations for the two classes: tool and creative intents. Each sample consists of a timestamped sentence with span annotations and the video session it is extracted from.
Based on the above defined framework, we col
Table 2: Statistics of the various splits in the Behance Intent Discovery dataset.
Top Unique Verbs in Action
Tool Intents	merge, select, add, use, paint, duplicate, make, delete, painting, do, flip, decrease, using, lower, figure, erase
Creative Intents	add, make, give, change, paint, create, convey, fill, animate, use, have
Top Unique Nouns in Object
Tool Intents	layer, color, mask, things, brush, shapes, selection, tool, shift, opacity, canvas, adjustment, stuff, thing
Creative Intents	colors, details, light, shadow, texture, contrast, highlights, vibe, depth, sense, bounce, feeling, elements
Table 3: Unique words in the intent discovery dataset.
We employed two annotators using the UpWork crowdsourcing platform and trained them for identifying intents from Behance videos and transcripts. They were instructed to annotate spans for tool and creative intents within each sentence. The annotations were created using the open-source Doccano annotation tool. In total, we collected annotations for 20,011 sentences from 66 Behance videos. The resulting Behance Intent Discovery dataset contains 13,989/2,105/3,917 samples in training, development and test splits respectively (see statistics in Table 2). We do not specify the duration of video clips for each sentence in the annotation. In our experiments, we explore varying clip duration and empirically choose a window of 10 seconds  $(\pm 5s)$  around the sentence's timestamp (see Sec. 8.2). The full video sessions are released for further research.
Dataset Analysis. We analyzed tool and creative intents to find the most frequent, unique verbs and nouns mentioned in the phrases. While there are action verbs which are distinctly tool-specific, such as merge, select, and duplicate, there are many verbs
$4$ https://www.upwork.com/. Annotators were compensated per the  $\$ 20/$ hr rate.
<sup>5</sup>https://github.com/doccano/doccano
<sup>3</sup>https://spacy.io/api/dependencyparser
which are common to both tool and creative intents such as add, make, and paint. Hence, the task model needs to learn the difference between tool and creative intents to be able to classify intents with similar action verbs into the correct categories. Further, we examined the unique nouns occurring in the intents and found lesser overlap between the two intent classes. Creative intents contain abstract and subjective visual concepts which pose a unique and interesting challenge to multimodal models. See Appendix for probing experiments.
5 Methods for Intent Discovery
Intuitively, as in a lot of instructional sources like text books, the text or audio serves as the primary mode of high-level information transfer, while the video/image signal provides detailed context or demonstration. Thus, we start our exploration using text models, which are built on two pretrained models: RoBERTa (Liu et al., 2019) and GPT2 (Radford et al., 2019). There are several previous works focusing on a limited set of intents (Xia et al., 2018), and thus, treat the problem of intent discovery as a classification problem. In our case, given the vast possibilities of potential intents in our sources, we cast the problem as a span detection problem, and design our models accordingly.
5.1 Unimodal Sequence Labelling
Our text models are designed similar to Named-Entity-Recognition models with a pretrained embedding layer and a sequence classification layer on top. Each phrase in the transcript is annotated separately in the intent dataset, leading to efficient processing. Although it is possible to process longer spans of text, in our annotations, we found out that each sentence usually gives enough information to extract the intent inside it, and extra context (neighboring sentences) does not significantly help the decision. We denote an input sentence as  $X = [x_0,\dots,x_N]$  with  $N$  as the length of the input sentence,  $Z = [z_{0},\dots,z_{N}]$  denotes the common IOB tags of two classes: creative intent and tool intent. Using text encoder  $f_{enc}$ , we extract text encodings  $E$  i.e.  $E = f_{enc}(X)$ . The encodings are then passed to the classifier layer for computing tag probabilities i.e.  $\hat{Z} = \text{softmax}(W_c * E + b_c)$  where  $W_{c}$ ,  $b_{c}$  are parameters of the classifier layer. The model is trained end-to-end using cross-entropy loss i.e.  $\mathcal{L}_{\theta} = -\frac{1}{N}\sum_{i=1}^{N}z_i\log(\hat{z}_i)$ , where  $\theta$  represents parameters of the entire model.
5.2 Multimodal Sequence Labelling: Naive Fusion
Seeking to leverage the video information, in our first attempt, we tried a simple feature fusion between the text signal and the video signal in the sequence labelling framework. We add a cross-attention layer on top of the pretrained text encoder in this naive joint video-text model and use the output of the cross-attention layer for sequence label classification. Let's denote the video features as  $V$ . Our model (see Fig. 4 (a)) is described as follows:
$$
\hat {Z} = s o f t m a x (W _ {c} * f _ {s e l f} (f _ {c r o s s} (E, V)) + b _ {c})
$$
where  $E$ ,  $f_{\text{self}}$  and  $f_{\text{cross}}$  are text encodings, self-attention and cross-attention layers respectively. This naive fusion model, however, does not provide any significant improvements compared to text-only baselines (see Sec. 7). Analysis of the results revealed that the textual features dominate the final decision, especially in the creative intent classes. To understand this behaviour, we performed a pilot task in which a human annotator looks through the video segments and tries to guess the intent without any transcript or audio. Our annotator found the task very difficult, and only possible after watching a very long context window, which partially explains the low performance of this model. The video signal is much more ambiguous than the text signal, and when presented with two sources where one is vastly less informative than the other, the model learns to rely only on the text, leading to no improvement compared to the text-only baseline. Joining two sources of features with different predictive utility is difficult. Given the fact that the video feature extractor is not trained on similar data, the video feature might not contain enough information for a direct intent detection task. Fortunately, our pilot task also reveals an important insight, i.e., the video signal is good at identifying whether an intent is present or not. Many intent candidates identified by the text models are not creative or tool intents, but are chitchat utterances from the instructor interacting with the audience. In these cases, we posit that the inactivity presented in the video signal is a strong indication that a creative/tool intent does not occur at the current time window. Using this idea, we propose a cascaded model with deeper interaction between video and text signal.
(a) Multimodal RoBERTa: Naive Fusion
(b) HERO for Sequence Labelling
(c) Multimodal RoBERTa: Late Fusion with Cascaded Cross-Attention
Figure 4: Demonstration of multimodal models: (a) Multimodal RoBERTa with naive fusion of video and text encodings; (b) Adaptation of HERO for sequence labelling with late fusion, see (c) for visual gate; (c) Multimodal RoBERTa with cascaded cross-attention and late fusion;  $\sigma$ , +, * represent sigmoid function, concatenation and matrix multiplication respectively.
5.3 Cascaded Cross-Attention & Late Fusion
Using the intuition that the text signal would provide candidates for the vision model, which is subsequently used for filtering out the cases without intent, we design the cascade cross-attention model as follows: First, we extract the set of contextualized embeddings  $E$  from the text encoder  $f_{enc}$  and transform it through two self-attention layers to create a two-stream architecture (see Fig. 4). In the first stream, the text encodings are processed through a single-layer of self-attention to produce  $E_1$ . In the second stream, the output from self-attention i.e.  $E_2$ , is combined with video embeddings through a cascaded cross-attention module. Let  $V = [v_1, v_2, \dots, v_k]$  be the input sequence of video embeddings. The cascaded module contains three cross-attention layers: video-to-text  $f_{v2t}(\cdot)$ , text-to-video  $f_{t2v}(\cdot)$  and text-to-text cross-attention  $f_{t2t}(\cdot)$ , with outputs computed as:
$$
S _ {1} = f _ {v 2 t} \left(W _ {m} V + b _ {m}, E _ {2}\right)
$$
$$
S _ {2} = f _ {t 2 t} \left(E _ {2}, S _ {1}\right)
$$
$$
S _ {3} = f _ {t 2 i} \left(E _ {s 2}, W _ {m} V + b _ {m}\right)
$$
where  $W_{m}, b_{m}$  are the parameters of a linear layer for transforming video embeddings. Next, the outputs from cross-attention layers are concatenated, linearly mapped and transformed into 0-1 values using a sigmoid, to generate the visual gate (see Fig. 4 (c)). Finally, the output from cross
attention layer is multiplied with this gate, i.e.
$$
S _ {g a t e} = \operatorname {s i g m o i d} \left(W _ {g} \left[ S _ {2}; S _ {3} \right] + b _ {g}\right)
$$
$$
S _ {c l f} = [ S _ {g a t e} * S _ {3}; E _ {s 1} ]
$$