SlowGuess/NAACL_2022_OCR huggingface dataset:
Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Ajay Patel, Alexander Sands, Chris Callison-Burch, and Marianna Apidianaki. 2018. Magnitude: A fast, efficient universal vector embedding utility package. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 120-126, Brussels, Belgium. Association for Computational Linguistics.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. The Fifth International Conference on Learning Representations.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872-1897.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners. Technical report.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text Transformer. Journal of Machine Learning Research, 21(140):1-67.
Nils Reimers. 2021. Pretrained models — Sentence-Transformers documentation. https://www.sbert.net/docs/pretrained_models.html. Accessed: 2022-01-14.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
J Ruppenhofer, M Ellsworth, M R L Petruck, C R Johnson, and J Scheffczyk. 2016. FrameNet II: Extended theory and practice.
Chirag Shah and Ryen W White. 2021. Bridging task expressions and search queries. In Proceedings of the 2021 Conference on Human Information Interaction
and Retrieval, pages 319-323, New York, NY, USA. Association for Computing Machinery.
Changjian Shui, Mahdieh Abbasi, Louis-Émile Robitaille, Boyu Wang, and Christian Gagné. 2019. A principled approach for learning task similarity in multitask learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 3446-3452. International Joint Conferences on Artificial Intelligence Organization.
Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, and Jason Weston. 2020. The Dialogue Dodecathlon: Open-domain knowledge and image grounded conversational agents. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2453-2470, Online. Association for Computational Linguistics.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2020. MPNet: Masked and permuted pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 33, pages 16857-16867.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 4444-4451, San Francisco, California, USA. AAAI Press.
Swabha Swayamdipta, Sam Thomson, Chris Dyer, and Noah A. Smith. 2017. Frame-semantic parsing with softmax-margin segmental RNNs and a syntactic scaffold. arXiv.
Mona Taghavi, Ahmed Patel, Nikita Schmidt, Christopher Wills, and Yiqi Tew. 2012. An analysis of web proxy logs with query distribution pattern approach for search engines. Computer Standards & Interfaces, 34(1):162-170.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(Nov):2579-2605.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I Guyon, U V Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Long Beach, California, USA.
Lili Wang, Chongyang Gao, Jason Wei, Weicheng Ma, Ruibo Liu, and Soroush Vosoughi. 2020. An empirical survey of unsupervised text representation methods on Twitter data. In Proceedings of the Sixth Workshop on Noisy User-generated Text, pages 209-214, Online. Association for Computational Linguistics.
Hongfei Zhang, Xia Song, Chenyan Xiong, Corby Rosset, Paul N. Bennett, Nick Craswell, and Saurabh Tiwary. 2019. Generic intent representation in web search. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, page 65-74, New York, NY, USA. Association for Computing Machinery.
Yi Zhang, Sujay Kumar Jauhar, Julia Kiseleva, Ryen White, and Dan Roth. 2021. Learning to decompose and organize complex tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2726-2735, Online. Association for Computational Linguistics.
The proprietary Wonderlist data was anonymized and personally identifiable information was scrubbed. Names were replaced by random names. In addition, k-anonymization was performed on the data so that tasks that were created by fewer than five users or fewer than 100 times in total were automatically discarded. The result is an aggregate view of the logs, devoid of any identifiers, private information or infrequent tasks that can be correlated back to a user. The data cleaning process was approved by an internal legal review board before the data was cleared for internal use. None of the data is exposed in this paper. Example texts presented in this paper are made up by the authors, and no text is taken verbatim from the original data.
As LITE is essentially built on pre-trained language models, biases existing in the original language models can still remain in the final model (e.g., biased associations between gender and actions). We did not observe any undesired associations caused by the models in our experiments, but it may be required to monitor biases and apply debiasing techniques before deploying the model to production systems.
Although LITE is not specifically designed for English, it will require significant cost to deliver the
list name	Expansion
<date>	(description) on <date>
<shop>	(description w/ verb) at <shop>
buy (description w/o verb) at <shop>
netflix	watch (description w/ verb) on netflix
mom	(description) for mom
Table 8: Examples of the templates used to generate autocompletion data. Tokens added by the templates are denoted in **bold** (<date>=monday,tuesday,weekend,…, <shop>=costco,wholefoods,…, etc.)
UITF1	ATAcc	CoLF1	CoTF1	IDAcc
None	.860	.900	.860	.443	.676
Input	.851	.923	.856	.448	.670
Intent ext.	.856	.929	.863	.453	.714
outcome to other languages due to the dependence on English resources (knowledge bases used for training COMET and English FrameNet).
B Templates for Autocompletion Data
We used 312 hand-crafted templates for collecting the autocompletion data. We first created templates for common nouns used in list names such as "today", "monday", "mom", and "home". We then used a publicly available dataset<sup>12</sup> to mine list names that represent company names such as "costco" and "target". We show some examples in Table 8.
C Architecture Search
We present the validation scores with different architectural choices in Table 9 (how to inject type embeddings) and Table 10 (number of attention heads in the intent extractor). We used  $\mathrm{BERT}_{\mathrm{base}}$  as a base text encoder and trained BERT-LITE on 500k samples of our dataset.
D Statistical Significance Test
Following the recommendation of Gorman and Bedrick (2019), we performed a permutation test with 5,000 trials between vanilla Transformer vs. DA, vanilla Transformer vs. LITE, and DA vs.
Table 9: Best validation scores with different type embedding settings: no token-type embeddings, injection into the input layer, and injection into the intent extractor. The models were trained on a 500k subset of the Wunderlist dataset.
\( \mathrm{U}\mathrm{T}_{\mathrm{F}1} \)	\( \mathrm{A}\mathrm{T}_{\mathrm{A}\mathrm{c}\mathrm{c}} \)	\( \mathrm{C}\mathrm{o}\mathrm{L}_{\mathrm{F}1} \)	\( \mathrm{C}\mathrm{o}\mathrm{T}_{\mathrm{F}1} \)	\( \mathrm{I}\mathrm{D}_{\mathrm{A}\mathrm{c}\mathrm{c}} \)
1	.857	.893	.856	.446	.650
4	.862	.925	.862	.450	.700
8	.860	.923	.862	.447	.740
12	.856	.929	.863	.453	.714
Table 10: Best validation scores with different numbers of attention heads  $(T)$ . The models were trained on a 500k subset of the Wunderlist dataset.
UIT	AT	CoL	CoT	ID
BERTbase
vanilla < DA	0	20	0	1	0
vanilla < LITE	7	20	4	8	0
DA < LITE	1	0	1	2	0
Sent. < LITE	14	20	5	0	0
RoBERTa
vanilla < DA	0	20	3	2	0
vanilla < LITE	20	20	20	19	1
DA < LITE	13	1	20	10	1
Sent. < LITE	6	20	6	0	0
BERTlarge
vanilla < LITE	6	20	3	9	1
Table 11: The number of random trials (out of 20) where the test score of the model on the right side is significantly better than the model on the left side after Bonferroni correction  $(\alpha = 0.05)$ .
LITE for each of twenty trials. We applied Bonferroni correction to the obtained p-values (Dror et al., 2017) to avoid over-estimate statistical significance. Table 11 reports the number of random trials where one model's score is significantly higher than that of the other model  $(\alpha = 0.05)$ . We can see that LITE performs significantly better than the vanilla counterpart more often than DA does. The results show that RoBERTa-LITE's score is even significantly higher than that of RoBERTa-DA in some tasks (UIT, CoLoc and CoTim).
E Additional Baseline Results
In this section, we present experimental results with the following additional baselines:
GPT-2 (Radford et al., 2018): We take the average of the last hidden states to represent an input sequence as we do for RoBERTa. Unlike BERT and RoBERTa, GPT-2 is a unidirectional encoder.
Sentence-MPNet: MPNet is a Transformer-based pre-trained language model that is reported to outperform BERT and RoBERTa (Song et al., 2020). Sentence-Transformer (Reimers and Gurevych, 2019) based on MPNet (Sentence-MPNet) is trained on 1.2B sentences from vari
$^{12}$ kaggle.com/peopledataabssf/free-7-million-company-dataset/version/1
UIT	AT	CoL	CoT	ID
Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1	Prec.	Rec.	F1	Acc.
GPT-2	.845	.803	.823	.908	.805	.907	.853	.501	.353	.414	.544
Sentence-MPNet	.865	.834	.849	.919	.798	.924	.856	.499	.386	.435	.654
word2vec	.856	.804	.829	.789	.780	.925	.846	.493	.284	.360	.628
word2vec-DA	.857	.816	.835	.805	.798	.896	.844	.506	.279	.360	.604
fastText	.856	.816	.835	.797	.780	.923	.845	.492	.282	.358	.678
BERT-LITE	.871	.855	.863	.932	.826	.901	.862	.511	.409	.454	.670
RoBERTa-LITE	.871	.847	.859	.919	.826	.905	.864	.509	.402	.449	.674
BERTlarge-LITE	.863	.849	.855	.936	.830	.907	.867	.516	.441	.475	.718
Table 12: Performance of additional baseline models and LITE (from Table 5) on downstream applications. The overall best scores are denoted in underlines.
UIT	AT	CoLoc	CoTim	ID
Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1	Prec.	Rec.	F1	Acc.
BERT	.828	.840	.833	.938	.898	.942	.919	.542	.608	.563	.320
RoBERTa	.849	.859	.853	.940	.864	.952	.905	.411	.432	.384	.288
Table 13: Result of in-dataset fine-tuning.
ous tasks and is considered to be the best-quality general-purpose encoder (Reimers, 2021).
word2vec and fastText: Unlike the other baseline encoders, word2vec (Mikolov et al., 2013) and fastText (Bojanowski et al., 2017) do not contextualize embeddings. We use a 300D word2vec model trained on Google News 100B and extend it by Magnitude (Patel et al., 2018) for OOV words. For fastText, we use a 300D model trained on Common-Crawl 2M. We also train a word2vec model from scratch on the same texts without special tokens as the domain-adapted version (DA).
Results (Table 12): GPT-2 performed worse than BERT and RoBERTa. Sentence-MPNet is trained with a huge amount of additional training data but still under-performs LITE. word2vec and fastText performed similarly and outperform vanilla BERT and RoBERTa on UIT and ID. The two datasets do not provide list names as input and have fewer data points than the other datasets. Thus, we conjecture that (1) there is not enough word context that vanilla BERT and RoBERTa can leverage and (2) the dimension of embeddings is too high for a classifier to find generalizable patterns from a small amount of data.
F Implementation Details of Baselines
We implemented the baseline encoders with the following libraries.
Transformers: We used Huggingface's transformers library (Wolf et al., 2020) to run
pre-trained Transformer models.
Sentence Transformers: We use the SentenceBERT library (Reimers and Gurevych, 2019)<sup>13</sup> to run pre-trained sentence encoders. We used the following pre-trained models:
BERT: roberta-base-nli-stsb-mean-tokens14
RoBERTa: roberta-base-nli-stsb-mean-token\*
MPNet: all-mpnet-base-v2<sup>16</sup>
G Fine-tuning BERT and RoBERTa
We present the performance of BERT and RoBERTa fine-tuned on downstream datasets. Note that our main goal is to train a general-purpose encoder that does not need to be re-trained for each downstream task as we describe in §1. We aim to answer the following two hypothetical questions.
Q1 (In-dataset fine-tuning): How well could BERT and RoBERTa perform if they were fine-tuned on the target dataset? This approach is commonly practiced for task-specific representations (Devlin et al., 2019).
13 www.sbert.net/   
14huggingface.co/sentence-transformers/bert-base-nli- stsb-mean-tokens   
15huggingface.co/sentence-transformers/roberta-base-nli- stsb-mean-tokens   
16huggingface.co/sentence-transformers/all-mpnet-basev2
→Test
↓Train	UIT	AT	CoLoc	CoTim	ID
UIT	.833	.638	.793	.394	.110
AT	.604	.938	.801	.405	.134
CoLoc	.497	.560	.919	.394	.116
CoTim	.325	.512	.814	.563	.148
ID	.362	.541	.782	.394	.320
LITE	.863	.932	.862	.454	.670
(a) BERTbase
→Test
↓Train	UIT	AT	CoLoc	CoTim	ID
UIT	.853	.645	.793	.373	.112
AT	.536	.940	.798	.372	.110
CoLoc	.412	.570	.905	.328	.104
CoTim	.276	.513	.823	.384	.106
ID	.359	.509	.796	.360	.288
LITE	.859	.919	.864	.449	.674
(b) RoBERTabase
Table 14: Test performance of fine-tuned BERT and RoBERTa. The diagonal cells show the performance of the models trained with the in-dataset fine-tuning setting.
Q2 (Cross-dataset fine-tuning): How well could BERT and RoBERTa perform on the target dataset if they were fine-tuned on another dataset? (Were the fine-tuned encoders generalizable to multiple to-do datasets?)
Setup: We fine-tune and evaluate  $\mathrm{BERT}_{\mathrm{base}}$  and  $\mathrm{RoBERTa}_{\mathrm{base}}$  models on the 20 random splits used in the main experiments. We follow Devlin et al. (2019) and add a linear classification layer that takes in the final hidden state of the first token ([CLS] token). For fine-tuning, the encoder and classifier are trained to optimize a binary cross entropy loss (UIT, CoLoc, and CoTim) or a cross entropy loss (ID and AT). We use the same optimization configurations described in §4.2.1. We continue training for 5 epochs and take the checkpoint that achieves the best validation score. For the cross-dataset experiment, we initialize the encoder with the fine-tuned parameters and freeze it during training. We use the same optimization settings except that we set a learning rate to 0.001.
A1 (Table 13): As expected, the fine-tuned models perform better than LITE on several datasets (AT, CoLoc, and CoTim with BERT, and AT with RoBERTa). When the main goal is to build task-specific representations, and there is a sufficiently large training dataset, task-specific fine-tuning will be a better solution than LITE. However, the result
shows the fine-tuned models do not always outperform LITE. We conjecture that for datasets without a sufficient number of training instances like UIT and AT, a fine-tuning strategy is not very effective.
A2 (Table 14): Performance consistently drops when the encoders are trained on another dataset, and all the scores are far below those of BERT/RoBERTa-LITE. This result indicates that LITE is more effective for training generalizable encoders than fine-tuning on a single dataset.