SlowGuess/NAACL_2022_OCR huggingface dataset:
Naoki Otani†, Michael Gamon‡, Sujay Kumar Jauhar‡, Mei Yang‡, Sri Raghu Malireddi‡, Oriana Riva‡
†Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA  
‡Microsoft Research, Redmond, WA, USA
†notani@cs.cmu.edu
$\ddagger$  {mgamon,sjauhar,meiyang,sri.malireddi,oriana.riva}@microsoft.com
Users write to-dos as personal notes to themselves, about things they need to complete, remember or organize. To-do texts are usually short and under-specified, which poses a challenge for current text representation models. Yet, understanding and representing their meaning is the first step towards providing intelligent assistance for to-do management. We address this problem by proposing a neural multitask learning framework, LITE, which extracts representations of English to-do tasks with a multi-head attention mechanism on top of a pre-trained text encoder. To adapt representation models to to-do texts, we collect weak-supervision labels from semantically rich external resources (e.g., dynamic commonsense knowledge bases), following the principle that to-do tasks with similar intents have similar labels. We then train the model on multiple generative/predictive training objectives jointly. We evaluate our representation model on four downstream tasks and show that our approach consistently improves performance over baseline models, achieving error reduction of up to  $38.7\%$ .
Task management tools are widely used to organize tasks and keep track of progress in work and daily lives. Examples include Microsoft To-do, Todoist, Trello, and digital assistants such as Amazon Alexa and Google Assistant. Machine learning techniques can automate various aspects of task management such as task creation (Mukherjee et al., 2020), organization (Landes and Di Eugenio, 2018), prioritization, and decomposition of complex tasks (Nouri et al., 2020; Zhang et al., 2021).
The goal of this work is to develop a single, general-purpose encoding system that converts todo task texts into real-valued vector representa
Figure 1: Our aim is to encode user-generated to-do texts (list names and descriptions) into vector representations so that machine learning systems can provide various kinds of intelligent assistance.
tions (Fig. 1). Using one encoding system for multiple task functions (task detection, organization, recommendation, etc.) as opposed to having multiple dedicated encoders saves the computational costs of updating models regularly and encoding texts from millions of users.
Representation learning has been extensively studied in natural language processing (Camacho-Collados and Pilehvar, 2018). Adapting models pre-trained on massive amounts of raw texts to a target domain or a task has become common practice (Qiu et al., 2020), with many publicly available pre-trained models such as BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2018), and sentence encoders (Cer et al., 2018; Reimers and Gurevych, 2019). Leveraging word context is one of the key strengths of these pre-trained models. However, to-do texts exhibit unique characteristics that make this context-based modeling less effective ( $\S 2$ ).
Our analysis on a dataset of 6.5 million entries shows that to-do texts are short and often lack an action verb. While similar to web search queries, they are not written to be understood by a search engine but instead are personal notes to the users themselves and assume rich personal context. On the other hand, some task management applications allow users to organize their to-dos under user-defined lists, which, our analysis shows, can sometimes convey important information about their
This work was done while the first author was an intern at Microsoft Research.
Human Language Technologies, pages 2410 - 2424
meaning (e.g., a "grocery" list vs. a "today" list).
Our hypothesis is that we can effectively fine-tune contextualized representation models for under-specified texts using multiple weakly-supervised prediction/generation tasks that focus on knowledge about to-do tasks. We induce supervision signals semi-automatically from existing resources so that to-do tasks that have similar intents share similar target labels. To this end, we propose LITE, a framework for training to-do task representation models using the following auxiliary tasks: (1) autocompletion of to-do descriptions, (2) pre-action and goal generation based on COMET (Bosselut et al., 2019; Hwang et al., 2021), and (3) action attribute predictions based on FrameNet (Ruppenhofer et al., 2016). We implement LITE on top of existing pre-trained language models and evaluate its performance through downstream tasks on two proprietary and two publicly available datasets (Jauhar et al., 2021; Landes, 2018): urgent and important to-do detection, actionable to-do classification, co-located and co-timed to-do pair detection, and intent detection.
Overall, we make the following contributions: (1) A neural multi-task learning framework to fine-tune embeddings of to-do texts based on intents. (2) A methodology to collect weak supervision signals from various resources without costly manual annotations. (3) An empirical comparison of contextual embeddings models on real to-do texts, where LITE outperforms various baseline models including BERT, RoBERTa, SentenceBERT/RoBERTa, achieving error reduction of 4.8-38.7%.
2 User-generated To-do Data
For training and data analysis, we use a dataset based on the now-retired Wunderlist task management app. The app was available on multiple platforms and had more than 13 million users in 2015. The dataset (henceforth WL) contains 6.5 million English to-do texts. Each to-do text includes a description (e.g., "call mom") and associated list name (e.g., "today"). See Appendix A for more details on how the dataset was anonymized.
We performed a basic linguistic analysis on the WL data. As observed by Landes and Di Euge
nio (2018), general-purpose analyzers often fail to analyze to-do texts correctly due to the writing style and the lack of context words. To alleviate this problem, we use frequency information obtained from a large corpus to correct automatically assigned POS tags, through the following 3-step process. First, we run the spaCy tagger (Honnibal et al., 2020)<sup>3</sup> to assign POS tags. Then, as proposed by Keyaki and Miyazaki (2017), we correct the POS tags based on frequency information derived from 3 billion sentences from the DepCC corpus (Panchenko et al., 2018).<sup>4</sup> Finally, we apply the spaCy dependency parser to the texts with the corrected POS tags and identified main verbs and arguments.
2.2 Observations
To-do descriptions are very short: The mean number of tokens per to-do description is 2.4, which is similar to that of search engine queries (Taghavi et al., 2012), but with two key differences: (1) many search queries are intended for information seeking (Broder, 2002), while to-dostypeically express things to perform or to remember, and (2) people write search queries with the capabilities of a search engine in mind, but to-do descriptions are personal notes to the users themselves.
Most to-do descriptions have no action verb: We observed that  $87.8\%$  of to-do descriptions do not have action verbs. If an action verb is present,  $75.1\%$  and  $12.7\%$  have a direct object and a prepositional phrase, respectively. The degree of underspecification depends on a to-do's list name. An action verb is more frequently used in to-do descriptions that appear in generic lists, such as "inbox"  ${}^{5}$ $(29.7\%)$ , "to do"  $(28.4\%)$ , and "today"  $(22.1\%)$ . When list names already imply a specific action, the action verb is more likely to be omitted such as in the "shopping"  $(3.3\%)$  and "movies to watch"  $(4.7\%)$  lists.
List names can be indicative of task intents: For example, a to-do text (description = "avocados",
3We use the English model en_core_web_lg v3.0.0
4We extracted the first 100 files from DepCC and re-tagged the sentences using spaCy. We counted the frequencies of 1-3 grams of token-XPOS pairs and replaced tokens that appeared fewer than 100 times with an out-of-vocabulary token. The frequencies were used to score the sequences of the POS tokens obtained in the previous step, and replace them with more frequent ones, if found. One of the authors manually evaluated the 100 frequent to-do descriptions with tags changed by post-processing and found 17/57 errors were corrected.
5"Inbox" was the default list name in the Wunderlist app.
<sup>1</sup>Short for Latent Intent Task Embedding
2The code is available at github.com/microsoft/Intent-based-Task-Representation-Learning
Figure 2: t-SNE (van der Maaten and Hinton, 2008) visualization of the embeddings of to-do texts generated by (1) BERT-base, (2) BERT-base domain-adapted by masked language modeling on WL, and (3) LITE.
list name = "to buy") signifies the intent "to buy avocados", but the same description can appear also in generic lists, such as "to do" or "reminders". When a list name is generic, a task description needs to be weighted more to accurately capture the intent of a task. Fig. 2 shows that this is a nontrivial problem for pre-trained language models like BERT. The figure visualizes the distribution of the embeddings of the "buy  $<$  grocery>" and "call  $<$  person>" to-do texts $^6$  expressed in two ways: (1) the descriptions "buy  $<$  grocery>" and "call  $<$  person>" are paired with generic list names ("to do" and "reminders"); or (2) the descriptions "<grocery>" and "<person>" are paired with specific list names (indicating the actions "to buy" and "to call"). To produce embeddings, we concatenated descriptions and list names in the input and extracted their pooled output from the encoders ( $\S 3.1$ ). We can see that a BERT model cannot capture the similarity within the buy nor call intent groups even after domain adaptation (DA) to to-do texts (see  $\S 4.3$  for more details on DA). Our model, LITE, can successfully ignore the generic lists and group similar tasks together.
3 Method: Multi-task Learning (MTL)
We propose a multi-task learning (MTL) framework to represent to-do descriptions along with their list names (Fig. 3). Our model first encodes text using off-the-shelf encoders ( $\S 3.1$ ). The token representations along with information about their types are merged by an intent extractor with multi-head attention ( $\S 3.2$ ). We train the encoder and extractor on three auxiliary tasks ( $\S 3.3, 3.4$ ).
3.1 Off-the-shelf Text Encoder
We encode input texts using off-the-shelf pretrained transformer-based language models, BERT (Devlin et al., 2019) and RoBERTa (Liu
et al., 2019b). Our model takes as input the concatenation of two types of texts, descriptions and list names, separated by the token [SEP]:  $
$  desc. [SEP] list name  $
$ , where  $
$  and  $
$  are beginning-of-sentence and end-of-sentence tokens pre-defined for the encoder. The encoder converts a sequence of  $N$  input tokens  $w_{1}, w_{2}, \cdots, w_{N}$  into real-valued vector representations using multiple layers of attention mechanism and fully-connected networks. We use the last hidden states  $H = \{h_{i}\}_{i=1,2,\cdots,N}$  as the contextual token representations of the input.
3.2 Intent Extraction with Attention
List names are often—but not always—indicative of task intents (§2). For example, a “shopping” list tends to have items that a user wishes to purchase and is useful for identifying intents, but some list names merely express time (e.g., “today”), topics/targets (e.g., “family”), or nothing specific (e.g., “things to do”). In these cases, the model should “pay more attention” to the to-do description.
To handle this, we use a multi-head attention mechanism (Vaswani et al., 2017; Chaudhari et al., 2021) to extract a vector representing the intent of a to-do task and introduce token type embeddings to explicitly inform a model of text types.
Multi-head attention: An attention mechanism is suitable to model the variable nature of token importance. We use a multi-head, scaled dot-product attention mechanism (Vaswani et al., 2017) and aggregate  $H$  based on token importance into the intent embedding  $z$ .
Suppose we have  $T$  attention heads. For each head, we convert a token representation  $\pmb{h}_i\in H$  into vectors  $\pmb{u}_i^t,\pmb{v}_i^t\in \mathbf{R}^{d'}$  by trainable transformation matrices,  $W_{u}^{t},W_{v}^{t}\in \mathbf{R}^{d^{\prime}\times d}$ . We set  $d^{\prime}$  to  $d / T$ .
$$
\boldsymbol {u} _ {i} ^ {t} = W _ {u} ^ {t} \tanh  \left(\boldsymbol {h} _ {i}\right) \tag {1}
$$
$$
\boldsymbol {v} _ {i} ^ {t} = W _ {v} ^ {r t} \tanh  \left(\boldsymbol {h} _ {i}\right) \tag {2}
$$
$6<$  grocery> stands for grocery items, and  $<$ person> stands for person names taken from the following web pages: vegetablesfruitsgrains.com and ssa.gov/oact/babynames
Note that our method is applicable to other encoder types.
Figure 3: LITE model overview. We encode input tokens with an off-the-shelf text encoder and feed the hidden states and type embeddings to an intent extractor to obtain the representation of the to-do task. We train LITE over three training objectives (1-3) jointly.
We then compute attention scores  $\alpha^t\in (0,1)$  and an output vector  $\pmb{o}^{t}\in \mathbf{R}^{d^{\prime}}$  ..
$$
\alpha_ {i} ^ {t} = \frac {\exp \left(\boldsymbol {q} ^ {t T} \boldsymbol {u} _ {i} ^ {t} / \sqrt {d ^ {\prime}}\right)}{\sum_ {j = 1} ^ {N} \exp \left(\boldsymbol {q} ^ {t T} \boldsymbol {u} _ {j} ^ {t} / \sqrt {d ^ {\prime}}\right)} \tag {3}
$$
$$
\boldsymbol {o} ^ {t} = \sum_ {i = 1} ^ {N} \alpha_ {i} ^ {t} \boldsymbol {v} _ {i} ^ {t}, \tag {4}
$$
where  $\pmb{q}^{t}\in \mathbf{R}^{d^{\prime}}$  is a trainable vector. Finally, we obtain an intent vector  $\textit{\textbf{z}}$  by concatenating the output vectors of the  $T$  attention heads:
$$
z = \operatorname {C o n c a t e n a t e} \left(\boldsymbol {o} ^ {1}, \boldsymbol {o} ^ {2}, \dots , \boldsymbol {o} ^ {T}\right) \tag {5}
$$
Token type embedding: We introduce token type embeddings,  $e_{\mathrm{task}}, e_{\mathrm{list}}, e_{\mathrm{other}} \in \mathbf{R}^d$ , to inform a model of the source of each token. BERT injects token type embeddings in the lowest layer, the embedding layer, and we train them during pretraining (Devlin et al., 2019), but other models do not (Radford et al., 2018; Liu et al., 2019b; Raffel et al., 2020). To avoid breaking the pre-trained parameters of those models, we add type embeddings to  $H$  and feed it to the multi-head attention module:
$$
\boldsymbol {h} _ {i} ^ {\prime} = \tanh  \left(\boldsymbol {h} _ {i}\right) + \tanh  \left(\boldsymbol {e} _ {\text {t y p e} (i)}\right) \tag {6}
$$
where type  $(i)$  is the type of the  $i$ -th token.
3.3 Auxiliary Tasks for MTL
One straightforward way to train the extractor is to directly optimize it to predict the intent of a given to-do task. However, task intents are often obscure and hard to discretize into a fixed number of categories. As a result, manual collection of such categories can be costly and subjective. For example, "buy milk" and "buy a car" are both purchase action, but they differ in many aspects: different locations, different prerequisite events, and different motives.
Input (desc., list)	Output
(milk, groceries)	buy milk
(buy milk, things to do)	buy milk
(eggs, Costco)	buy eggs at Costco*
(Chris, today)	call Chris today*
Table 1: Examples of texts for the autocompletion objective (§3.3.1). Suppose to-do descriptions "buy milk", "buy eggs" and "call Chris" exist in the WL dataset. We use list-based templates to generate the last two examples denoted by  $*$ .
Instead, we propose to train the extractor on multiple auxiliary tasks with weak supervision that provide semantic augmentation to under-specified to-do texts. The underlying assumption is that tasks with similar intents have similar target labels/texts in the auxiliary tasks. Below, we present our three auxiliary training tasks.
3.3.1 Autocompletion
Motivation: Inspired by Lewis et al. (2020), our first task focuses on surface-level information of to-do texts, namely prediction of missing tokens based on context tokens. Specifically, we feed a to-do text (the combination of a description and a list name) to a model, convert it into an intent embedding, and generate the maximal form of a to-do description that is inferable from the input. We call this auxiliary task autocompletion objective. We automatically collect such forms for under-specified to-do descriptions from the WL dataset.
Data collection: As previously observed, to-do descriptions under generic lists (e.g., "today") tend to be more specified than those under lists whose names imply specific action. For each to-do description in our WL dataset, we collect their longer descriptions (i.e., super-strings) up to five. We also use several templates for lists that represent locations and times to further expand descriptions (see
Input (desc.)	Output
buy milk	go to store	get milk for breakfast
call Chris	find number	talk to someone
subscribe Netflix	go to website	watch a movie
Relation used:	xNeed	xIntent
details in Appendix B). Table 1 shows examples, two of which were generated with templates. The resulting dataset contains 1,487,161 pairs of short and long to-do descriptions. We combine them with specified to-do descriptions, which already have action verbs and do not have longer counterparts, and sample 2M examples (50% of examples are under-specified.) During training, one generation target is picked at random for each instance.
3.3.2 Pre-action and Goal Generation
Motivation: This task aims to represent to-do tasks based on their prerequisite actions (what we must do beforehand) and goal events (what we want to achieve), assuming that tasks with similar intents have similar prerequisites and goals. Here, a model is trained to generate prerequisite and goal actions for a given to-do item (a task description and a list name). We call this objective pre-action and goal generation objective.
Data collection: We leverage COMET (Hwang et al., 2021), a BART model (Lewis et al., 2020) fine-tuned on  $\mathrm{ATOMIC}_{20}^{20}$ , to collect weak supervision signals about to-do tasks' prerequisites and goals. Specifically, we feed a long description of a to-do task generated in the previous step ( $\S 3.3.1$ ) to the BART model as a prompt followed by a relation token: (1) xNeed (prerequisite) token to generate the task's prerequisite or (2) xIntent (goal) token to generate the task's goal. We use beam search with width of 3 and collect the top-3 results for each relation. Table 2 shows generation results for three example to-dos.
3.3.3 Action Arguments Prediction
Motivation: Different to-do tasks have different domain-specific arguments. For example, a pur
Table 2: Texts generated for the pre-action and goal objective (§3.3.2) by COMET (Bosselut et al., 2019; Hwang et al., 2021).
Input (desc.)	Output
buy milk call Chris	Buyer, Goods Addressee, Topic,···	Money,··· Medium,···
FEs used:	Core	Non-core
Table 3: Labels collected from FrameNet (Ruppenhofer et al., 2016) for the action arguments prediction task (§3.3.3).
chase task must have a purchase target, and possibly a price argument. In contrast, contact tasks usually have a receiver and a topic of communication argument. We design a multi-label training task called action arguments prediction, where, given a description and a list name, a model predicts all the action arguments associated with the to-do task.
Data collection: We use FrameNet (Ruppenhofer et al., 2016), a manually-created database on the meaning and usage of English words/phrases. Semantic representations are defined for concepts and events (called frames) and for their semantic elements (called frame elements, FEs); example texts that trigger frames and FEs are also provided. FEs can be core FEs (essential information for a frame), or non-core (optional). Table 3 shows examples.
Using the "long" to-do descriptions collected for the autocompletion task (§3.3.1), we identify frames in them using an off-the-shelf frame identifier (Swayamdipta et al., 2017). As our focus is on to-do tasks, we discard frames whose root frame is not Event. We then collect FEs for each frame from FrameNet. If a to-do description has two or more frames, we take the union of their FEs. For non-core FEs, we calculate importance weights by TF-IDF over the whole FrameNet repository so that common FEs appearing in many frames (e.g., Manner) have low weight. We normalize the weights into  $(0,1]$  by dividing them by the maximum weight.
For the autocompletion as well as the pre-action and goal generation tasks, we employ a two-layer GRU (Cho et al., 2014) decoder with a cross-attention mechanism (Luong et al., 2015). We use the embedding layer of the encoder also in the decoder. We train the model to minimize the following cross-entropy loss for each instance:
$$
\mathcal {L} _ {\text {g e n}} = - \sum_ {j = 1} ^ {M} \log P \left(y _ {j} \mid y _ {<   j}, z, H\right), \tag {7}
$$
<sup>8</sup>We can retrieve prerequisites and goals of some to-do tasks from knowledge bases such as ATOMIC<sup>20</sup> (Hwang et al., 2021) and ConceptNet (Speer et al., 2017) without relying on language generation, but it is not always the case that we can find the action of interest in the existing resources. The use of COMET is advantageous in handling unseen actions as shown by several studies (Bosselut et al., 2019; Hwang et al., 2021).
where  $M$  is the length of the output text. We apply label smoothing with a smoothing factor of 0.1 (Pereyra et al., 2017).
For the action arguments prediction task (multi-label classification), we use GILE as a label-embedding approach (Pappas and Henderson, 2019). Given an intent embedding and label embedding, GILE projects them into a joint vector space and computes an association score from their element-wise product. Concretely, for each label  $l$  we calculate its score  $P(l) \in (0,1)$  as follows:
$$
\boldsymbol {e} _ {\text {i n}} = \operatorname {A c t} \left(W _ {\text {i n}} z\right) \tag {8}
$$
$$
\boldsymbol {e} _ {\text {l a b e l}} ^ {(l)} = \operatorname {A c t} \left(W _ {\text {l a b e l}} \boldsymbol {v} ^ {(l)}\right) \tag {9}
$$
$$
P (l) = \operatorname {S i g m o i d} \left(W _ {\text {o u t}} \left(\boldsymbol {e} _ {\text {i n}} \odot \boldsymbol {e} _ {\text {l a b e l}} ^ {(l)}\right)\right), \tag {10}
$$
where  $\pmb{v}^{(l)}\in \mathbf{R}^d$  is a pre-computed label embedding (constant), Act is an activation function and  $W_{\mathrm{in}},W_{\mathrm{label}}\in \mathbf{R}^{d\times d}$  and  $W_{\mathrm{out}}\in \mathbf{R}^{1\times d}$  are model parameters. To compute the label embeddings for FEs (Eq(9)), we encode the definitions of FEs in FrameNet with pre-trained transformer models.
We define the loss function to be:
$$
\mathcal {L} _ {\mathrm {c l f}} = \frac {1}{C} \sum_ {c = 1} ^ {C} (c \log P (c) + (1 - c) \log (1 - P (c))), \tag {11}
$$
where  $C$  is the number of classes.
We optimize a model to minimize the following weighted loss across three MTL objectives:
$$
\mathcal {L} = \sum_ {\text {t a s k}} \frac {\mathcal {L} _ {\text {t a s k}}}{\log N _ {\text {t a s k}}}, \tag {12}
$$
where  $N_{\mathrm{task}}$  is the number of target labels in a subtask (Aghajanyan et al., 2021).
Our aim is to obtain a single, general-purpose representation model that is effective on various downstream applications. We run LITE on top of BERT<sub>base</sub>, BERT<sub>large</sub>, and RoBERTa and evaluate its performance.
4.1 Evaluation Tasks
We evaluate LITE on four downstream tasks (Table 4): (1) urgent and important to-do detection (UIT), (2) actionable to-do classification (AT), (3) co-located and co-timed to-do pair detection (CoLoc and CoTim), and (4) intent detection (ID). Urgent and Important To-do Detection (UIT): The goal of this task is to detect urgent or important
tasks, an essential step for to-do prioritization in real applications. To evaluate this task we use a proprietary dataset (derived from WL) containing 2,254 human-labeled to-do descriptions. Each description is categorized into urgent and not-urgent classes based on the majority vote of 3 annotators. This dataset does not provide list names, hence we use a dummy list name "inbox" for LITE.