SlowGuess/NAACL_2022_OCR huggingface dataset:
Actionable To-do Classification (AT): This task aims to identify to-do tasks that require a concrete, individual action to accomplish (ActionableTask) (e.g., "Sign up for dance class"). We evaluate this task using a proprietary dataset (derived from WL) containing 12,189 to-dos. Each instance consists of a description and a list name, and is manually categorized into ActionableTask, Note, and ActionableCollection. A Note is a list item that users add for future use, without the need for immediate action (e.g., "baby names"). Tasks that are labeled as ActionableCollection are not performed individually but rather as part of a collection of items in a larger task: "tomatoes" in the "groceries" list, for example, are part of the larger task "do groceries" where all the individual to-dos are addressed at the same time and location. Each example was annotated by 3 annotators, the majority label is the gold label. Tasks where one or more annotators were unsure about the correct label were eliminated.
Co-located and Co-timed To-do Pair Detection (CoLoc/CoTim): This task focuses on the location and time where to-do tasks are accomplished. Time and location are particularly powerful cues for task recommendations and reminders (Graus et al., 2016). In this task, given a pair of to-do items, the model predicts whether the two to-do tasks can be completed in the same location (CoLoc) or at the same time (CoTim). To evaluate this task we use the MS-LaTTE (Jauhar et al., 2021) dataset (derived from WL), which contains 25,000 pairs of to-do tasks (description + list name), of which 398 are labeled as CoLoc and 401 as CoTim.
Intent Detection (ID): This task focuses on predicting the intent associated with a to-do description. We use Landes and Di Eugenio (2018)'s dataset, which contains 253 to-do instances, each one labeled with one of nine intent classes ("calendar", "find-service", "buy", etc.). No list name is provided in this dataset, so we use the generic list name "inbox" for LITE.
Task	Size	Example: (description, list name) [class]
Urgent and Important To-do Detection (UIT)	2,254	(pick up packages at FedEx, n/a) [urgent],
(sign up for HBO, n/a) [non-urgent]
Actionable To-do Classification (AT)	12,189	(Sign up for dance class, inbox) [Actionable],
(tomatoes, groceries) [ActionableCollection],
(StarWars, movies to watch) [Notes]
Co-located To-do Pair Detection (CoLoc)	25,000	(fix tv, inbox)-(clear sink, today) [+],
(fix tv, inbox)-(refill medicines, today) [-]
Co-timed To-do Pair Detection (CoTim)	25,000	(get breakfast, daily)-(check news, inbox) [+],
(get breakfast, daily)-(pickup drycleaner, inbox) [-]
Intent Detection (ID)	253	(schedule appointments with site managers, n/a) [calendar],
(fix the CD ROM drive on my computer, n/a) [find-service]
Table 4: Evaluation tasks. Note that the UIT and ID datasets do not have list names.
In all tasks, we first generate vector representations of instances in the dataset with a pre-trained encoder and train a simple classifier on them. The quality of the embeddings is measured by the performance of the classifier. We use a logistic regression classifier implemented in scikit-learn (Pedregosa et al., 2011), with or without a penalty term. To train a classifier for CoLoc and CoTim, which provide two to-do descriptions as input (see section 4.1), we concatenate the vector representations of the two items along with their element-wise product and difference vectors (Mou et al., 2016).
We generate 20 sets of training, validation, and test splits at random (Gorman and Bedrick, 2019), and, in each trial, we use a validation split to tune hyperparameters by grid search (a regularization  $\in$  {None, L1, L2} and a regularization coefficient  $\in$ $\{2^{-5}, 2^{-4}, 2^{-3}, 2^{-2}, 2^{-1}, 1\}$ ).
We implemented our MTL framework using PyTorch v1.10.0 (Paszke et al., 2019) and ran experiments on NVIDIA GeForce GTX TITAN X and RTX A6000 (for  $\mathrm{BERT}_{\mathrm{large}}$ ). We use uncased  $\mathrm{BERT}_{\mathrm{base}}$ , uncased  $\mathrm{BERT}_{\mathrm{large}}$ , and cased RoBERTa, in the transformers library v4.6.1 (Wolf et al., 2020) with the default parameters for dimensions, activation functions, and dropout. We set the number of attention heads in the extractor and the dimension of hidden states based on the choice of a text encoder, namely  $(T,d) = (12,768)$  for  $\mathrm{BERT}_{\mathrm{base}}$ -LITE and RoBERTa-LITE, and  $(T,d) = (16,1024)$  for  $\mathrm{BERT}_{\mathrm{large}}$ -LITE. We applied dropout of 0.1 to our modules except for the output layers. We optimized the model parameters using AdamW (Loshchilov
and Hutter, 2019) with batch size of 2,048, learning rate of 5e-5, L2 weight decay of 0.01, and linear learning rate decay with warm-up steps of  $2\%$  of the total steps. We also apply gradient norm clipping of 1. We train our models for 15 epochs, and freeze the transformer encoder for the first 5 epochs. We sampled 3,459 examples as validation data, on which we evaluate a model every epoch, and terminate training if the validation loss does not improve for three consecutive epochs. We tuned hyperparameters and architectural choices (§3.2) based on the average validation scores over 20 random trials on all the datasets (more details in Appendix C).
We compare the following encoders as baselines. $^{10}$
BERT (Devlin et al., 2019): We take the embedding of the first token, [CLS], to represent a to-do text. [CLS] embeddings are trained to represent the whole input sequence by next sentence prediction (NSP). We compare the base (12 layers, 768D) and large (24 layers, 1024D) models.
RoBERTa (Liu et al., 2019b): We take the average of the last hidden states to represent an input sequence as RoBERTa is not trained with NSP. We use RoBERTa  $\mathrm{base}$  (12 layers, 768D).
Motivated by Gururangan et al. (2020), we also compare the domain-adapted ("DA") version of BERT and RoBERTa. We perform additional pretraining to  $\mathrm{BERT}_{\mathrm{base}}$  and RoBERTa on the 6M raw to-do texts ( $
$  description [SEP] list name  $
$ ) from WL.
Sentence-Transformer: We also test off-the-shelf general-purpose sentence encoders based on Transformers. These encoders are pre-trained to induce sentence embeddings with siamese and
<sup>10</sup>We evaluate additional baselines in Appendix E. The implementation details can be found in Appendix F.
<sup>9</sup>We split data into 6:2:2 for UIT, AT, and CoLoc/CoTim, and 8:1:1 for ID.
UIT	AT	CoLoc	CoTim	ID
Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1	Prec.	Rec.	F1	Acc.
BERT	.826	.798	.811	.906	.800	.917	.855	.511	.362	.423	.628
BERT-DA	.862	.821	.840	.928	.801	.921	.857	.510	.386	.439	.614
Sentence-BERT	.821	.787	.803	.901	.817	.892	.853	.499	.396	.442	.542
BERT-LITE	.871	.855	.863	.932	.826	.901	.862	.511	.409	.454	.670
RoBERTa	.805	.763	.783	.868	.777	.923	.844	.492	.335	.398	.506
RoBERTa-DA	.819	.745	.779	.913	.787	.922	.849	.488	.360	.414	.500
Sentence-RoBERTa	.831	.789	.809	.897	.820	.893	.855	.493	.386	.433	.572
RoBERTa-LITE	.871	.847	.859	.919	.826	.905	.864	.509	.402	.449	.674
BERTlarge	.817	.795	.805	.896	.805	.910	.854	.488	.404	.442	.636
BERTlarge-LITE	.863	.849	.855	.936	.830	.907	.867	.516	.441	.475	.718
triplet network on top of pre-trained Transformer models (Reimers and Gurevych, 2019). We use the pre-trained encoder based on  $\mathrm{BERT}_{\mathrm{base}}$  and RoBERTa  $\mathrm{base}$ . The encoders are trained on about  $286\mathrm{k}$  of natural language inference and textual similarity instances.
Table 5 shows our main results. LITE consistently achieves the best performance on all tasks for all three encoders, demonstrating the generality of the learned representations. DA brings in performance improvements but only marginally on most tasks<sup>11</sup>. This is probably because to-do texts are too short to perform effective language model training.
Sentence-Transformers have proven effective in various sentence-level tasks (Reimers and Gurevych, 2019), but it is not the case in this experiment. The vanilla BERT and RoBERTa encoders perform on par with their Sentence-Transformer counterparts and in some cases outperform them. We conjecture that those sentence encoders cannot leverage contextual information effectively as they are pre-trained on sentences that are quite different from to-do texts. Our training framework can also fine-tune Sentence Transformers to adapt them to short and under-specified to-do texts, which we leave for future work.
Our goal is to train a general-purpose encoder. However, the interested reader can find an evaluation of task-specific fine-tuning in Appendix G.
Table 5: Results on downstream applications. The best scores in each text encoder are denoted in bold, and the overall best scores are underlined. The results of statistical significance tests can be found in Appendix D.
\( \mathrm{UIT}_{\mathrm{F}1} \)	\( \mathrm{AT}_{\mathrm{Acc}} \)	\( CoLoc_{F1} \)	\( CoTim_{F1} \)	\( ID_{Acc} \)
Full	.863	.932	.862	.454	.670
-Ac	.855	.931	.861	.448	.656
-PG	.859	.923	.860	.449	.726
-Aa	.857	.928	.860	.440	.702
Table 6: Ablation study on BERT-LITE demonstrating the effect on F1 and accuracy scores of removing (A)uto(c)ompletion, (P)re-action and (G)oal generation, or (A)ction (A)rguments prediction.
List	Attn. × Norm
errands	.036±.004
to do list	.040±.003
things to do	.040±.003
movies to watch	.040±.004
house to do	.041±.004
:
trip	.093±.005
target	.094±.006
cleaning	.097±.005
bring	.097±.006
movies	.098±.007
Table 7: To-do lists that are assigned low and high attention weight  $\times$  vector norm scores by LITE. Generic lists are denoted in **bold**, and specific lists in italic.
Table 6 shows the contribution of our auxiliary tasks to the overall performance. The full model performs the best in all the tasks except ID.
As discussed earlier, our model needs to combine information from descriptions and list names to infer the meaning of to-dos. We show that our model successfully learns when to attend lists. We extract list names from the AT dataset that appear with more than 17 different to-do descriptions (90% percentile) and analyze the product of attention weights and vector norms of descriptions and list
<sup>11</sup>It is also possible to combine domain adaptation by language modeling and LITE, however, it underperformed LITE overall. With BERT<sub>base</sub>, the performance were UIT 0.873, AT 0.931, CoLoc 0.863, CoTim 0.447, and ID 0.656.
tokens (Kobayashi et al., 2020). Table 7 shows list names with the highest and lowest average scores assigned by BERT-LITE. Generic list names have low scores ('to do list', "house to do") while specific (action-related) list names have much higher scores ("bring", "cleaning").
However, we believe it would not be prudent to just ignore generic lists as they can still convey semantic/pragmatic clues. For example, a list named "wishlist" typically has to-dos which a user does not need to act on immediately. Hence, this list is a strong indicator of a non-actionable task (in AT).
To-do Management: Intelligent systems can assist users with task management in many ways (Gil et al., 2012). To-do tasks can be inferred automatically from emails (Mukherjee et al., 2020). Systems can detect types of to-do items and suggest relevant applications or resources to users (Landes and Di Eugenio, 2018; Gil et al., 2012; Shah and White, 2021). Once to-do tasks have been created, a system can help users manage the completion progress, e.g., by sending reminders (Graus et al., 2016). Complex tasks can be decomposed automatically into more manageable sub-steps (Nouri et al., 2020; Zhang et al., 2021). In all these use cases, a common step is to represent the input language as computational vector representations, but none of the existing studies has produced general-purpose representations of to-do tasks.
Short-text Representations: Multiple NLP areas involve very short texts with some unique characteristics. Several methods have been developed for tweets (e.g., Nguyen et al., 2020). Tweets pose the added challenge of containing many non-standard colloquial expressions and contain non-language text like URLs. Still, Wang et al. (2020) present a similar finding to ours: massively pre-trained encoders do not always perform well. Search queries are also short, with an average of three terms (Taghavi et al., 2012). Unlike to-dos, information such as click logs (Zhang et al., 2019) can be used as an indicator of user intent. Another key difference is that search queries are written with the goal of having a machine interpret them.
Multi-task Learning: Multi-task learning improves the performance of pre-trained language models in various NLP tasks (Liu et al., 2019a; Shuster et al., 2020; Aghajanyan et al., 2021). The common perception in the research community is
that auxiliary training tasks are effective when they are similar to the target domain/task (Shui et al., 2019). However, there are few relevant tasks and datasets for the to-do domain. Our study is the first work to propose a time- and cost-efficient way to harvest weak-supervision for MTL in that domain.
We discussed how to produce general-purpose representations of short and under-specified to-do texts for performing various kinds of intelligent task assistance with a single encoder. Our method, LITE, uses a multi-head attention mechanism with token type embeddings on top of an off-the-shelf contextual text encoder to effectively induce semantic information from the combination of to-do descriptions and list names. The model is trained using three auxiliary tasks: autocompletion, pre-action and goal generation, and action arguments prediction.
We applied LITE to BERTbase, BERTlarge, and RoBERTa and compared them with various baseline models on four downstream tasks. LITE consistently outperformed the baselines, demonstrating the effectiveness of our method.
We thank the anonymous reviewers, Chirag Shah, Subho Mukherjee, and the members of the Knowledge Technologies and Intelligent Experiences (KTX) team at Microsoft Research Redmond for their valuable feedback.
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. arXiv.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Florence, Italy. Association for Computational Linguistics.
Andrei Broder. 2002. A taxonomy of web search. SI-GIR Forum, 36(2):3-10.
Jose Camacho-Collados and Mohammad Taher Pilehvar. 2018. From word to sense embeddings: A survey on vector representations of meaning. Journal of Artificial Intelligence Research, 63(1):743-788.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Sneha Chaudhari, Varun Mithal, Gungor Polatkan, and Rohan Ramanath. 2021. An attentive survey of attention models. ACM Transactions on Intelligent Systems and Technology, 12(5).
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1724-1734. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Rotem Dror, Gili Baumer, Marina Bogomolov, and Roi Reichart. 2017. Replicability analysis for natural language processing: Testing significance with multiple datasets. Transactions of the Association for Computational Linguistics, 5:471-486.
Yolanda Gil, Varun Ratnakar, Timothy Chklovski, Paul Groth, and Denny Vrandecic. 2012. Capturing common knowledge about tasks: Intelligent assistance for to-do lists. ACM Transactions on Information Systems, 2(3).
Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786-2791, Florence, Italy. Association for Computational Linguistics.
David Graus, Paul N Bennett, Ryen W White, and Eric Horvitz. 2016. Analyzing and predicting task reminders. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization, pages 7-15, Halifax, NS, Canada. ACM.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining:
Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength natural language processing in Python.
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. COMET-ATOMIC 2020: On symbolic and neural commonsense knowledge graphs. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, pages 6384–6392, Online.
Sujay Kumar Jauhar, Nirupama Chandrasekaran, Michael Gamon, and Ryen W. White. 2021. MS-LaTTE: A dataset of where and when to-do tasks are completed. arXiv.
Atsushi Keyaki and Jun Miyazaki. 2017. Part-of-speech tagging for web search queries using a large-scale web corpus. In Proceedings of the Symposium on Applied Computing, pages 931-937, New York, NY, USA. ACM.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7057-7075, Online. Association for Computational Linguistics.
Paul Landes. 2018. Supervised approach imperative todo list categorization. https://github.com/plandes/todo-task. Accessed: 2022-01-14.
Paul Landes and Barbara Di Eugenio. 2018. A supervised approach to the interpretation of imperative to-do lists. arXiv.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487-4496, Florence, Italy. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. The Seventh International Conference on Learning Representations.
Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111-3119, Stateline, Nevada, USA. MIT Press.
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 130-136, Berlin, Germany. Association for Computational Linguistics.
Sudipto Mukherjee, Subhabrata Mukherjee, Marcello Hasegawa, Ahmed Hassan Awadallah, and Ryen White. 2020. Smart To-Do: Automatic generation of to-do items from emails. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8680–8689, Online. Association for Computational Linguistics.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14, Online. Association for Computational Linguistics.
Elnaz Nouri, Robert Sim, Adam Fourney, and Ryen W White. 2020. Step-wise recommendation for complex task support. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, pages 203-212, New York, NY, USA. ACM.
Alexander Panchenko, Eugen Ruppert, Stefano Farali, Simone P Ponzetto, and Chris Biemann. 2018. Building a web-scale dependency-parsed corpus from CommonCrawl. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, Miyazaki, Japan. European Language Resources Association.
Nikolaos Pappas and James Henderson. 2019. GILE: A generalized input-label embedding for text classification. Transactions of the Association for Computational Linguistics, 7:139-155.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang,