SlowGuess/NAACL_2021_OCR huggingface dataset:
Yi Zhang†, Sujay Kumar Jauhar§, Julia Kiseleva§, Ryen W. White§ and Dan Roth†
†University of Pennsylvania, Philadelphia, PA, USA
†{yizhang5, danroth} @cis.upenn.edu, §{sjauhar, jukisele, ryenw} @microsoft.com
People rely on digital task management tools, such as email or to-do apps, to manage their tasks. Some of these tasks are large and complex, leading to action paralysis and feelings of being overwhelmed on the part of the user. The micro-productivity literature has shown that such tasks could benefit from being decomposed and organized, in order to reduce user cognitive load. Thus in this paper, we propose a novel end-to-end pipeline that consumes a complex task and induces a dependency graph from unstructured text to represent sub-tasks and their relationships. Our solution first finds nodes for sub-tasks from multiple 'how-to' articles on the web by injecting a neural text generator with three key desiderata – relevance, abstraction, and consensus. Then we resolve and infer edges between these sub-task nodes by learning task dependency relations. We collect a new dataset of complex tasks with their sub-task graph to develop and evaluate our solutions. Both components of our graph induction solution are evaluated in experiments, demonstrating that our models outperform a state-of-the-art text generator significantly. Our generalizable and scalable end-to-end solution has important implications for boosting user productivity and assisting with digital task management.
People today increasingly rely on digital modalities and applications to organize, track and complete tasks from their work and life. They depend on email to structure their communications as a way of tracking pending tasks (Bellotti et al., 2003), issue commands to their digital assistants<sup>1</sup> for timely task reminders (Brewer et al., 2017), and use task management applications (Bellotti et al., 2004)<sup>2</sup>
Figure 1: The subtask graph of "plan a birthday party", where nodes are subtasks, and the directed edges represent the dependencies between subtasks.
to organize grocery runs, thesis writing and everything in between.
In this work, we focus on tasks that are complex (Hassan Awadallah et al., 2014), and – which research in micro-productivity has shown (Kirsh, 2000; Teevan et al., 2016a) – may benefit from thoughtful organization. For example, consider Figure 1, where we show how the complex task “plan a birthday party”<sup>3</sup> can be broken down into more manageable pieces and structured by mutual temporal dependencies, in order to create an actionable plan that is simpler and more effective.
In this paper we propose to help automate generating such actionable plans in order to reduce cognitive load on users. While prior research (Cheng et al., 2015; Teevan et al., 2016b) has shown the benefits of tracking and acting on micro-tasks, little effort has been expended on finding automated solutions for actually breaking down complex tasks into tractable sub-tasks. Thus we design a novel end-to-end solution that is capable of decomposing complex tasks and structuring sub-task dependencies.
We model our end-to-end solution as a graph induction problem, in which we first find nodes to represent sub-tasks, then infer the temporal dependency edges between them, yielding a flow diagram like the one in Figure 1. All of this is done from unstructured text ubiquitously found on the web, making our approach general and scalable.
3Contrast this with a simple task such as "send the PPT deck to Sarah".
*Most of this work was done while the first author was an intern at Microsoft Research.
Siri, Cortana, Google Assistant etc.
2Google Keep, Todoist, Microsoft To Do etc.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2726-2735 June 6-11, 2021. ©2021 Association for Computational Linguistics
In the first component (that of finding nodes), we learn to synthesize information from multiple 'how-to' articles across the web and generate text fragments for sub-tasks. In particular, we extend a state-of-the-art neural text generator by injecting it with three desiderata for these fragments: relevance (to the complex task), abstraction (by summarizing content in articles), and consensus (for appearing across multiple sources). In the second component (that of finding edges), we infer temporal dependencies between sub-tasks.
Existing corpora of 'how-to' articles (most notably WikiHow (Koupaee and Wang, 2018)) do not contain this latent dependency structure. Moreover, articles in these corpora are structured and formatted consistently and uniformly, making them ill-suited to our approach, which seeks to synthesize the content of multiple heterogeneous web pages. We therefore devise a simple annotation framework through which we gather a new dataset of complex tasks, and their associated subtasks and mutual dependencies, from multiple 'how-to' web articles using non-expert crowd workers. Finally, we use this data to fine-tune our augmented neural text generator, as well as predict dependency edges between the sub-tasks it generates.
In experiments, we demonstrate that our optimal solution - which encodes relevance, abstraction and consensus - yields significant improvements over a state-of-the-art text generator on both subtask generation and dependency prediction.
The focus of this paper is on Complex Tasks; however, our research has impact beyond intelligent task management. For example, learning to decompose complex natural language expressions could have impact on complex question answering (Chali et al., 2009; Luo et al., 2018), where question decomposition, multi-hop reasoning, information synthesis, and implicit knowledge all play an important role. More generally, the ability to model mappings between short text fragments and elements in multiple documents could benefit research in areas such as topic-focused multi-document summarization (Wan et al., 2007) and event timeline extraction of evolving news stories (Do et al., 2012).
In summary, our key contributions are (i) building an end-to-end pipeline for complex task decomposition as a graph induction problem from unstructured text; (ii) constructing a new dataset for complex tasks that contain sub-tasks as well
as the temporal dependencies between them; and (iii) extending a neural text generator by injecting signals for relevance, abstraction and consensus, thereby making it more capable at tackling task decomposition.
We begin by defining some key concepts. We refer to a task as a text fragment that represents a goal people want to track, remind themselves of, or learn how to do; for example, "buy a Christmas present", "eat healthier" or "change a tire". In order to disambiguate the intent of tasks (consider the fragment "Harry Potter", which could equally refer to "read [the book]" or "watch [the movie]") we scope our investigation to tasks that contain at least one verb.
A task is considered as a complex task when two or more individual steps – themselves also worth tracking, remembering or learning how to do – need to be performed in its completion. Therefore, “plan a birthday party”, which involves creating a guest list and buying food and beverages (see Figure 1) is a complex task. While “throw out the trash” is not such a complex task, even though it may involve opening the front door and walking to the trash bins. We refer to the individual steps of a complex task as sub-tasks.
Sub-tasks may sometimes depend on other sub-tasks being completed before they can be tackled. Consider the example from Figure 1 again, which illustrates how one must "set up a time" and "make a guest list" before "send(ing) out invitations". We refer to these relations as temporal dependencies, and are pair-wise notated as sub-task B depending on  $(\leftarrow)$  A.
Given these key concepts, we define a complex task graph as follows. Let the sub-tasks of a complex task  $t$  be denoted by  $ST(t) = \{s_i\}_{i=1}^n$ . Then define  $G_s(t) = (V, E)$  as the complex task graph of  $t$ . Here  $G_s(t)$  is a directed graph, where  $V = ST(t)$  is the set of sub-task nodes, and  $E$  represents the set of temporal dependency edges such that  $(s_i, s_j) \in E, s_i \gets s_j$ .
2.1 Modeling Overview
Given these definitions, the problem of decomposing and organizing a complex task becomes inducing a graph  $G_{s}(t) = (V,E)$  from a complex task
input  $t^4$ . To construct the graph, the key steps are (1) generating the sub-task nodes  $V$ , and (2) inferring the temporal dependency edges  $E$  between nodes. We propose to do both from unstructured text.
In particular, the web has made a large number of instructional texts on a variety of topics and activities freely available for public consumption; some of them are in purpose-built websites, such as WikiHow (Koupaee and Wang, 2018), while others appear in personal blogs, social fora, educational portals and a number of other heterogeneous sources. We leverage these resources to find relevant information for complex tasks. Specifically, given a task  $t$ , we query a search engine with the term "how to  $t$ " ("plan a birthday party" becomes "how to plan a birthday party"), and store the  $k$  most relevant results in a collection  $D_{k}(t)$ . Our graph induction problem then becomes finding the optimal graph  $G_{s}(t) = (V,E)$  given the evidence in  $D_{k}(t)$ . We elaborate on solutions for node generation and edge inference in what follows.
Sub-task Generation Formally, given a complex task  $t$  and a collection of relevant articles  $D_{k}(t)$ , we attempt to generate the sub-tasks  $ST(t)$ . We argue that the text fragments for sub-task nodes we generate must satisfy three desiderata:
(1) Relevance, so that generated sub-tasks are directly related to the complex task  $t$ .
(2) Abstraction, because 'how-to' articles often explain and expand on sub-tasks.
(3) Consensus, since sub-tasks that are cited across multiple sources are more likely to be important.
Our model for sub-task generation builds on BART (Lewis et al., 2019), a state-of-the-art sequence-to-sequence model for text generation, and injects it with our three desiderata. Concretely, we make BART capable of handling multi-source input, design a custom relevance-aware cross attention layer and implement a cluster encoding technique to guide the generation process. Details for the model are presented in Section 4.1.
Dependency Inference Given the generated set of sub-tasks  $V = ST(t)$ , the next step in our end-to-end graph induction solution consists of inferring the temporal dependency edges  $E$ . We formulate this as a binary classification problem, where we attempt to predict the existence of a dependency edge  $\forall (s_i, s_j) \in V$ .
Specifically, we use the concatenated intermediate representations for sub-tasks  $(s_i,s_j)$  from our enhanced BART model and add a final linear layer to learn a binary classifier. We train this classifier on a new dataset of complex tasks that contains pairwise temporal dependency information (see Section 3). More details are given in Section 4.2.
To build and evaluate our solutions, we need data. The most relevant existing dataset is WikiHow (Koupae and Wang, 2018), which is derived from the popular how-to website. However WikiHow, while very useful for parts of our modeling paradigm, is ill-suited to others. Namely, its articles are manually curated, with consistent structure and format, making them a mis-match to the heterogeneous, noisy and free-form articles we expect to encounter on the web. Moreover, they contain no dependency information between sub-tasks beyond a simple numbered ordering.
Thus, to support our problem, we need a dataset which (1) contains complex task and their sub-tasks; (2) encodes dependencies between sub-tasks; and (3) the sub-tasks come from a variety of webpages. Note that our goal is to create a dataset that enables model generalization, rather than constructing a comprehensive knowledge base. Therefore, rather than exhaustively annotating sub-tasks and their dependencies, we seek only to gather labels for the most important ones. In what follows we will describe the step-by-step construction of our dataset, and how these steps encode our three fundamental desiderata for task-sub-task relationships (see Section 2).
The dataset we collect, which we call MSComplexTasks is being released freely to the research community.
<sup>4</sup>We assume complex tasks as input, since the focus of our work is on their understanding and decomposition. We leave the problem of distinguishing complex tasks from simple ones to future work.
5While edge prediction is technically a structured prediction problem, we demonstrate in this paper that even a simple approach works well; we leave more complex modelling solutions to future work.
https://github.com/microsoft/MSComplexTasks
Collecting Complex Tasks and How-to Articles We begin with logs from the popular – now defunct – task management application Wunderlist<sup>7</sup>. These logs are privately and respectfully handled by passing them through an enterprise grade, legal- and trust-approved pipeline, which anonymizes, aggregates and scrubs all personally identifiable information. This yields a collection of task strings, some of which have associated sub-task metadata (not sub-tasks themselves). We retain those tasks which have at least one sub-task, and contain at least one verb<sup>8</sup> (to avoid issues with disambiguating task intent) as a candidate seed pool of complex tasks. It may be noted that while the logs are not publicly available, they play a minimal role in our end-to-end solution. Their only purpose is to seed the initial set of complex how-to queries.
In order to find relevant articles for each complex task we trawl through a month's worth of logs from a commercial search engine using the 'how-to' query expansion described in Section 2.1. Further, in order to protect user privacy we discard queries that were issued by fewer than 5 distinct users. To each remaining complex task query, we associate the top-10 clicked URLs across all users for the entire month. Text in these webpages satisfy our notion of relevance.
Finding Candidate Sentences for Sub-tasks Next, we create a pool of candidate sentences that we hypothesize might contain sub-tasks. Specifically, we use an in-house webpage parser to extract section headings and list items from the set of URLs previously collected. These types of text fragments often represent very short summaries that are then elaborated on in 'how-to' articles; we thus attempt to restrict our candidate pool to sub-tasks that capture the notion of abstraction.
Finally, we also care about consensus across articles, since this allows us to retain only those sub-task which are cited in different sources and are therefore more reliably important. Because the same sub-task can be expressed differently in text we perform clustering on the BERT (Devlin et al., 2018) embeddings of candidate sentences and discard those clusters that only contain a single source URL. The remaining set of sentences form our pool of candidate sub-tasks.
Labeling Sub-tasks and Dependencies Given the set of candidate complex task queries and their associated sub-task sentences, we ask crowd-workers to label them. Specifically, we guide annotators through a series of questions: (1) Is the candidate query about a task? A complex task? (2) If it is a complex task, which candidate sentences represent sub-tasks? (3) Does the ordering of sub-tasks matter? If so, assign pairwise temporal dependency labels to sub-tasks. We ask three workers to label each HIT, aggregating annotations by majority vote. Table 1 shows some examples of aggregate judgments from our annotation study.
Recall from Section 2 that we model complex task decomposition as a graph induction problem over unstructured text. In what follows we will first describe our approach for sub-task node construction, followed by our method for sub-task temporal dependency inference.
4.1 Sub-task Construction
As described in Section 2, we treat sub-task finding as a text generation problem. While we could ostensibly frame it as a span prediction problem, this is unsuitable for our modeling paradigm. First, our multi-source setting means that we might potentially (and in fact want to) extract more than one text span referring to the same sub-task. Thus resolving identical sub-tasks and picking the best among them would require additional logic, as well as a sub-task coreference module. Moreover, while we could use a webpage parser to build a candidate pool of sub-task text spans (see Section 3), such a parser might be brittle and error prone – or even non-existent. While we have human annotators to refine this pool during dataset construction, no such remedy exists at automatic inference time.
Model Architecture Our model is based on the pre-trained text generation model BART (Lewis et al., 2019). This is a sequence-to-sequence neural summarizer, which consists of a bidirectional encoder over corrupted text and a left-to-right autoregressive decoder. To extend it to a multi-source setting, we encode each source article independently with BART's encoder (a bi-directional Transformer (Vaswani et al., 2017)), and then concatenate the output embeddings of the encoders. These are then fed to the decoder which generates the output sentences autoregressively. We treat all of
$^{7}$ https://en.wikipedia.org/wiki/Wunderlist
Detected using https://spacy.io/.
Task?	Complex Task?	Sub-tasks	Ordering Matters?	Dependencies
apply for a green card	Yes	Yes	find out if you re eligible prepare for your interview ...	Yes	prepare for your interview← find out if you re eligible
start a grocery store	Yes	Yes	form your LLC finding the best location marketing your grocery store ...	Yes	marketing your grocery store← finding the best location
Table 1: Examples of results from the annotation
Figure 2: The model architecture of our solution. We extend BART to handle multiple documents  $(d_{1}, d_{2}, d_{3})$  as input, design the relevance-aware cross attention layer and cluster encoding techniques to guide the generation process.
the subtasks of a given complex task as a single target document, and subtasks are thus generated as a sequence of text fragments. A diagram of our architecture is shown in Figure 2. We initialize our model using the parameters of the pre-trained BART-large model released with the HuggingFace Transformer library (Wolf et al., 2019). These parameters are then fine-tuned to our task, using the proposed architecture. In our work, the inputs are the textual content from URLs returned by a 'how-to' complex task query, and the outputs are the set of generated sub-tasks. Because we are learning mappings between 'how-to' articles and short text fragments that summarize their contents, we are implicitly learning the notion of abstraction.
Relevance-Aware Cross-Attention As discussed in Section 2, sub-tasks need to also be relevant to complex tasks. In order to encode relevance we design a cross-attention mechanism that explicitly captures textual relevance. Specifically, we score each sentence in a set of articles based on its relevance to the complex task, then propose a general mechanism to inject this information into the text generation model.
Given a complex task  $t$ , we denote the query expansion "how to do  $t$ " as  $q$ , and the collection of related articles as  $D_{k}(q)$ . To score the relevance of each sentence  $s \in D_{k}(t)$ , we consider two factors. The first is how relevant the sentence  $s$  is to the query  $q$ , and the other is how relevant the article  $d$  (with  $s \in d$ ) is to  $q$ . We denote the relevance of  $s$  as  $P(s|q)$ , and the relevance of  $d$  as  $P(d|q)$ . To compute  $P(d|q)$ , we first represent  $q$  and  $d$  as  $n$ -dimensional embedding vectors respectively using BART, denoted respectively as  $\vec{q}$  and  $\vec{d}$ . Then article-query relevance is computed as the softmax:
$$
P (d | q) = \frac {\exp f (d , q)}{\sum_ {d ^ {\prime} \in D _ {k} (q)} \exp f \left(d ^ {\prime} , q\right)} \tag {1}
$$
where  $f(d,q) = \vec{q}^T\cdot \vec{d}$
The sentence-query relevance  $P(s|q)$  is similarly computed using the softmax over embeddings  $\vec{q}$  and  $\vec{s}$ . Notably, these embeddings are generated from a BART model fine-tuned on an auxiliary sequence classification task, where positive samples are sub-tasks of complex task  $t_i$  and negative samples are randomly samples sub-tasks from other complex tasks  $t_{j(j\neq i)}$ .
Given  $P(s|q)$  and  $P(d|q)$ , we need a way of injecting them into our generation model. The BART decoder performs cross-attention over the final hidden layer of the encoder to determine how much focus to place on other parts of the input sentence, as words in specific positions are encoded. Our model should not only pay attention to the inputs, but should additionally pay more attention to those are more relevant. We therefore define our relevance-aware cross-attention as follows:
$$
\sigma \left(\frac {Q (x) K (w) ^ {T} + \alpha p (s | q) * p (d | q)}{\sqrt {d _ {k}}}\right) V \tag {2}
$$
where  $\sigma (\cdot)$  represents the softmax function,  $x$  is a token in the output sub-task,  $w\in s\in d$  is a token
in the input article,  $Q,K$  and  $V$  are query, key and value representations,  $d_{k}$  is the dimension of the key vector, and  $\alpha$  is a learnable parameter which controls the importance of our relevance injection.
Cluster Encoding The final desiderata for our model is consensus, or the ability to be able to recognize and reward sub-tasks that are mentioned across multiple articles. To encode this signal we create a cluster embedding, which identically represents sentences that refer to the same sub-task across sources.
To generate the new cluster embeddings, we first embed the query  $q$  as well as the set of sentences  $s \in D_k(t)$  into  $n$ -dimension vectors using BART. Then we cluster the embeddings of all sentences  $s$  with KMeans, ranking the clusters by the proximity of their centroid to the embedding of  $q$ . Finally, using a formulation similar to the positional encodings from Vaswani et al. (2017) we define cluster encoding as:
$$
P E _ {(s, 2 j)} ^ {c} = \sin \left(\frac {r _ {i}}{1 0 0 0 0 ^ {2 j / d _ {\text {m o d e l}}}}\right)
$$
$$
P E _ {(s, 2 j + 1)} ^ {c} = \cos \left(\frac {r _ {i}}{1 0 0 0 0 ^ {2 j / d _ {\text {m o d e l}}}}\right) \tag {3}
$$
where  $r_i$  is the rank of the cluster that  $s$  belongs to, and the symbols  $j$  s.t.  $1 \leq j \leq n$  are dimensional indices; specifically  $2j$ , and  $2j + 1$  represent indices into the clusters' embeddings. For instance, if the cluster embedding were defined as a vector of length 512, then  $2j$  (resp.  $2j + 1$ ) represent the 0th, 2nd, 4th, ..., 510th (resp. 1st, 3rd, 5th, ..., 511th) index positions of the vector. These notations are identical to the ones used in the original by Vaswani et al. (2017) in their definition of positional encoding.
This formulation allows our model to identify tokens that belong to similar sentences, and are injected into the extended BART model as an additional input.
4.2 Dependency Inference
In our work, we treat inferring dependencies between sub-tasks as a binary classification problem. Specifically, we learn a classifier capable of predicting the existence (or not) of a temporal dependence  $s_i \gets s_j$  between all possible ordered pairs of generated sub-tasks  $(s_i, s_j) \in E$ .
We use the same extended BART architecture previously proposed in Section 4.1, and concatenate the intermediate representations for  $s_i$  and  $s_j$
to yield an output representation for a pair of subtasks. Then we add a linear layer on top of this output to predict a binary label for the existence of a temporal dependency edge.
Similar to sub-task generation, we hypothesize that finding consensus across articles may also be helpful in determining the dependencies between sub-tasks. Therefore, we also use cluster encodings in inferring edge dependencies.
RQ1 Can we accurately and automatically generate sub-tasks given an input complex task?
RQ2 Can we correctly identify the temporal dependencies between sub-tasks?