SlowGuess/NAACL_2022_OCR huggingface dataset:
Belinda Z. Li  
MIT  
bzl@mit.edu
Jane Yu Meta AI janeyu@fb.com
Madian Khabsa Meta AI mkhabsa@fb.com
Luke Zettlemoyer Meta AI lsz@fb.com
Alon Halevy Meta AI ayh@fb.com
Jacob Andreas MIT jda@mit.edu
When a neural language model (LM) is adapted to perform a new task, what aspects of the task predict the eventual performance of the model? In NLP, systematic features of LM generalization to individual examples are well characterized, but systematic aspects of LM adaptability to new tasks are not nearly as well understood. We present a large-scale empirical study of the features and limits of LM adaptability using a new benchmark, TASKBENCH500, built from 500 procedurally generated sequence modeling tasks. These tasks combine core aspects of language processing, including lexical semantics, sequence processing, memorization, logical reasoning, and world knowledge. Using TASKBENCH500, we evaluate three facets of adaptability, finding that: (1) adaptation procedures differ dramatically in their ability to memorize small datasets; (2) within a subset of task types, adaptation procedures exhibit compositional adaptability to complex tasks; and (3) failure to match training label distributions is explained by mismatches in the intrinsic difficulty of predicting individual labels. Our experiments show that adaptability to new tasks, like generalization to new examples, can be systematically described and understood, and we conclude with a discussion of additional aspects of adaptability that could be studied using the new benchmark.
Much of the recent research effort in NLP has shifted from training task-specific models to adapting pre-trained language models (LMs) by finetuning their parameters or input prompts for downstream tasks (Devlin et al., 2019; Raffel et al., 2020; Li and Liang, 2021; Lester et al., 2021). This paradigm is general, in the sense that a large number of distinct NLP tasks benefit from pretraining (Peters et al., 2018; Devlin et al., 2019; Raffel et al., 2020). But many questions about the
nature and limits of LM adaptation remain unanswered. For example: given a new task, can we predict how quickly (and how effectively) pre-trained LMs can be adapted to perform it? From among the variety of different adaptation techniques (e.g. finetuning or prompt-tuning), can we predict which one will be most effective? Today, new pre-training and adaptation schemes are evaluated using small suites of curated tasks, typically featuring classification, textual inference, and question answering (Wang et al., 2018, 2019). These benchmarks have been extremely successful in identifying new tools for adaptation, but they are poorly suited for answering larger, structural questions like the ones above.
We present a large-scale study of LM adaptability using a new suite of benchmark tasks called TASKBENCH500.1 TASKBENCH500 consists of 500 procedurally generated tasks involving lexical semantics, factual information, memorization of random relations, list processing, and logical composition (Fig. 1). Analogous to past work that uses synthetic data to characterize LM performance on single examples (Weston et al., 2016; Lake and Baroni, 2018; Saxton et al., 2019; Kim and Linzen, 2020; Keysers et al., 2020; Liu et al., 2021a), TASKBENCH500 enables systematic study of LM adaptability at the task level. In this paper, we use it to study three aspects of adaptability:
Memorization: When can adaptation successfully memorize new functions (e.g., to update factual knowledge about entities, or learn arbitrary new token correspondences)? We find that LMs' ability to memorize new input-output mappings is strongly influenced by task type. Datasets of lexical relations (like antonym pairs) are easier to memorize than factual information (like name-occupation pairs). Both are easier to memorize than lists of random word pairs. These findings are particularly striking in the case of prompt tun
$^{1}$ Data and code available at: https://github.com/facebookresearch/task_bench
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4696 - 4715 July 10-15, 2022 ©2022 Association for Computational Linguistics
1) Define tasks
2) Make datasets for each task
3) Adapt model to each task
Figure 1: Overview of our task creation process. We begin by defining a set of atomic tasks that all synthetic tasks are built upon. These include lexical tasks (blue text/outline), random tasks (green text/outline), and factual tasks (orange text/outline). They also include both predicates and relations. These tasks are combined using composition functions to form more complex, compositional tasks. Given a particular task specification, we synthetically create a dataset for each task. Finally, we fine-tune or prompt-tune a pre-trained language model on each task dataset.
ing, which in standard configurations struggles to memorize even small random word pair lists.
Composition: Is LM performance on simple tasks predictive of their performance on compositions of those tasks? (If the father and occupation relations are easy to learn via adaptation, does this imply that the father's occupation relation is also easy to learn?) We find a nuanced answer. LMs exhibit compositional adaptation to lexical and factual relations (like father's occupation), with success on composed tasks strongly correlated ( $r^2 > 0.5$ ) with success on atomic tasks. However, when composing these relations with sequence processing operations, success on the base task does not predict success on the composed task.
Distribution matching: In models fine-tuned on datasets exhibiting a distribution of acceptable answers (e.g., translating ungendered pronouns into gendered ones), do model predictions match these distributions? We find that LMs are often unable to match label distributions in datasets used for adaptation. In particular, when labels in the fine-tuning dataset are drawn from a uniform mixture of labels from two tasks (e.g., labeling half of the words with their antonym and half with their synonym), models disproportionately assign mass to labels from the task that is easier to learn.
Each of these forms of adaptability corresponds to a central challenge in NLP: reliable updating of deployed models, composition of previously learned skills, and fair and predictable output from models trained on curated data. Our study of mem
orization, composition, and distribution matching have direct analogs to previous studies of sample expressivity (Zhang et al., 2017), compositional generalization (Lake and Baroni, 2018; Kim and Linzen, 2020; Keysers et al., 2020), and calibration (Guo et al., 2017). However, we study these phenomena at the task level, rather than the example level. Our experiments highlight important qualitative differences between current adaptation paradigms; identify several novel challenges for LM adaptation, and offer a new benchmark for approaches aimed at meeting these challenges.
Fine-tuning and prompt search In languages for which large digitized corpora are available, most NLP system development today involves adaptation of a pre-trained model to a downstream task of interest. Pre-training typically involves reconstruction of masked or corrupted text sampled from a large corpus (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020). Adaptation to a new task typically involves one of three approaches: (1) fine-tuning of all of a pre-trained model's parameters (possibly in conjunction with a specialized decoder) on a task-specific training set (Devlin et al., 2019); (2) manual prompt engineering of an input template that induces pre-trained model predictions to perform the task of interest (Brown et al., 2020; Petroni et al., 2019); or (3) automated prompt tuning of these templates, in either the discrete space of tokens (Shin et al., 2020) or con
tinuous space of token embeddings (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021b). The latter two approaches have grown more popular as pre-trained models have grown larger. The performance of both prompt-search approaches still lags fine-tuning (Raffel et al., 2020; Brown et al., 2020; Lester et al., 2021), though the difference between approaches appears to shrink as model size increases (Lester et al., 2021).
Measuring generalization and adaptability The success of the training paradigm described above stems from its generality—a large number of NLP tasks appear to benefit from some combination of pre-training and adaptation. Previous attempts to quantify this generality have typically relied on benchmarks like GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019), each of which aggregates ten natural language processing tasks designed to probe different aspects of language understanding. Similar benchmarks have also been built for non-English languages (Xu et al., 2020; Kakwani et al., 2020; Park et al., 2021; Hu et al., 2020). However, the heterogeneity and small number of distinct tasks represented in existing benchmarks makes it difficult to make finer-grained predictions, e.g. by identifying specific features of tasks that contribute to the success or failure of adaptation.
This challenge has a direct analog to the problem of characterizing generalization at the example level in models trained for a single task. Model performance on natural test sets is often loosely correlated with accuracy on individual examples featuring rare syntactic constructions or word collocations (McCoy et al., 2019). A great deal of past work has focused on improving evaluation using synthetic evaluation sets (Jia and Liang, 2017; Naik et al., 2018; Lake and Baroni, 2018; Richardson et al., 2020). These datasets have been used to study long-range agreement (Marvin and Linzen, 2018), compositional generalization (Lake and Baroni, 2018; Ruis et al., 2020; Keysers et al., 2020), and mathematical reasoning (Saxton et al., 2019). But no analogous notion of systematicity, or tool for studying it, currently exists for tasks rather than examples.
Thus, building on this past work, we describe how to construct synthetic data distributions that enable systematic study of adaptation to new tasks rather than generalization to new examples. Like previous research that uses procedural data generation procedures to study models in NLP, we focus
on coverage rather than naturalness, using datasets designed to complement, rather than replace, existing naturalistic benchmarks.
3 A 500-task benchmark
Our goal is to study the generalizability of task adaptation paradigms. In particular, we would like to identify which attributes of a task make it easy or difficult to learn, across different models and training schemes. While this work shares many of its high-level goals with existing benchmarks built from collections of real-world datasets, the makeup and difficulty of these datasets is often difficult to characterize precisely: differences in annotation standards, annotation quality, and dataset size mean that models often exhibit very different performance on datasets designed to evaluate model performance on the same abstract task. In addition, existing datasets cover an exceedingly small subset of the space of all tasks that future NLP practitioners might wish to perform. To account for all these limitations, we propose to generate tasks synthetically as described below.
The space of tasks TASKBENCH500 is constructed compositionally: we begin by defining a space of atomic tasks, which are combined using a set of composition operators to produce more complex tasks. Every task takes as input a word or word sequence, and outputs either a boolean value or a set of words/word sequences. We refer to any task that outputs booleans as a predicate task, and any task that outputs sets of words or word sequences as a relation task. A subset of relation tasks involve modeling relations between single words at the input and output; we refer to these as word-level tasks and the remaining relation tasks (that take sequences as input or output) as sequential tasks.
The choice of atomic tasks and composition functions aims to capture aspects of real language processing tasks. Accordingly, the set of atomic tasks comprises of:
1. Lexical tasks, which test knowledge of lexical semantics. These include lexical relations like synonym, or lexical predicates like is-noun. These tasks are constructed from WordNet relations (Fellbaum, 1998).
2. Factual tasks, which test factual knowledge. These include factual relations like father-of, or factual predicates like
is-human. These tasks are constructed from Wikidata properties (Vrandecic and Krötzsch, 2014).
3. Random relation tasks, which test memorization ability. These are created by mapping a word in the vocabulary to a singleton set containing a random other word. We create 4 random relations with different random seeds.
To recursively create arbitrarily complex tasks, we define a set of composition functions. These take tasks as arguments and return other tasks. These functions fall into two categories:
1. Word-level compositions, which test ability to combine word-level information in different ways, such as through set or logical operations. These functions take word-level tasks and return other word-level tasks. Examples include intersection and chaining.
2. Sequential compositions, which test ability to operate on sequences. These functions convert word-level tasks to sequence-level tasks. There are two functions in this category: map takes a word-level relation task and returns a task that maps a sequence of  $n$  words to a set of all possible sequences resulting from applying  $f_{W}$  to each input word.2 filter takes word-level predicate tasks and returns a sequence consisting only of words for which the task returns true, preserving the original ordering of those words.
The full list of atomic tasks and composition function can be found in Appendix Tables 4 and 5. We surmise that typical NLP tasks may require some combination of lexical knowledge, factual knowledge, sequential processing, and other task-specific reasoning; our data distribution lets us evaluate all these aspects separately and in combination.
Datasets for tasks We create datasets  $\mathcal{D}(f) = \{(x_i,y_i):x\sim \mathcal{X}_f,y\sim \mathrm{Unif}(f(x_i))\}$  for each task  $f$  , where  $\mathcal{X}_f$  is the input distribution for the task, and recalling that  $f(x_{i})$  returns a set of possible outputs associated with the input  $x_{i}$  . For all tasks, we randomly partition the dataset into  $\mathcal{D}_{\mathrm{train}}(f)$  and  $\mathcal{D}_{\mathrm{test}}(f)$  splits.
For lexical atomic tasks and their compositions, we directly use the most common words in the task's input language for  $\mathcal{X}_f$ . We create tasks in English and Spanish. For factual atomic tasks and their compositions, we sample the entities from Wikidata that participate in the relation or predicate defined by the task (e.g. for the child-of task, we sample only entities that have children). For sequential tasks, we use a random sampler, which samples  $n$  random words from the vocabulary and concatenates them.
Figure 1 shows examples of tasks and associated datasets. More details on dataset construction can be found in Appendix A.
Model & Training For all experiments, we adapt a pre-trained T5-base model (Raffel et al., 2020). We examine two types of training paradigms: fine-tuning and prompt-tuning. During fine-tuning, we update all model parameters on the training set. During prompt-tuning, we follow Lester et al. (2021) and introduce a new set of prompt-tokens  $\{p_1,\dots ,p_n\}$  to the vocabulary, which will be prepended to every sample from the task during inference, i.e., each sample input  $x$  becomes  $p_1p_2\dots p_n x$ . Let  $\theta$  denote the parameters of the original pretrained LM. During training, the entire model is frozen and only the word embeddings of the new tokens  $\{\theta_{p_1},\dots ,\theta_{p_n}\} \subset \theta$  are updated. We use prompts of length  $n = 100$  for all experiments. We also study each paradigm on various quantities of training data, and separately evaluate their memorization and generalization adaptabilities. In particular, for word-level tasks the test-set words are disjoint from the train-set words, so evaluating on the test set will strictly measure generalization capacity. We optimize all models using AdamW. See Appendix B for full hyperparameters.
Evaluation For each task  $f$  and model  $\mathcal{M}[\theta]$  (with parameters  $\theta$ ), we measure the model's average per-token accuracy on both training and test splits of the dataset  $\mathcal{D}(f)$ . As each task defines multiple acceptable outputs for each input, we credit models for producing any acceptable output. Letting  $y' = \mathcal{M}(x)$ , we measure the fraction of positions  $i$  at which any valid answer  $y_i$  matches the
2Note word-level relations return sets of words—we turn a sequence of sets of words into a set of sequences by considering all combinations of words in each set.
Figure 2: Left: Overview of the memorization experiment, which evaluates how accurately models adapted via fine-tuning and prompt-tuning can memorize training data. Right: Memorization and generalization curves for fine-tuning and prompt-tuning on 1000 training examples. Memorization curves are shown by solid lines, while generalization curves are dashed. We average over all atomic tasks from each task category: lexical tasks, factual tasks, and random tasks. The shaded region shows the standard error of the mean. Transparent lines are each individual task, colored by task category. In both paradigms, lexical tasks are easiest to memorize, followed by factual tasks, then random tasks. However, prompt-tuning has overall much less memorization capacity than fine-tuning, which can perfectly memorize even completely random relations.
predicted  $y_{i}^{\prime}$  ..
$$
\operatorname {a c c} (\mathcal {M}, \mathcal {D} (f)) =
$$
$$
\max  _ {y \in f (x)} \frac {1}{| \mathcal {D} |} \sum_ {(x, y) \in \mathcal {D}} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} [   [ y _ {i} = y _ {i} ^ {\prime} ]   ]\right) \tag {1}
$$
Further details can be found in Appendix B.
Given a pretrained model  $\mathcal{M}[\theta_{\mathrm{pretrain}}]$ , an adaptation procedure  $\mathcal{T}$ , and a task suite  $f$ , let  $\mathcal{M}[\theta_{\mathcal{T},\mathcal{D}(f)}]$  denote the model trained using  $\mathcal{T}$  on training data  $\mathcal{D}(f)$ . We then define the adaptability of a (pretrained model, adaptation paradigm, task suite) as:
$$
\begin{array}{l} \operatorname {a d a p t} \left(\mathcal {M} \left[ \theta_ {\text {p r e t r a i n}} \right], \mathcal {T}, f\right) \\ = \operatorname {a c c} \left(\mathcal {M} \left[ \theta_ {\mathcal {T}, \mathcal {D} _ {\text {t r a i n}} (f)} \right], \mathcal {D} _ {\text {e v a l}} (f)\right) \tag {2} \\ \end{array}
$$
We denote by  $\text{adapt}_{\text{mem}}$  the value of this metric over training data  $(\mathcal{D}_{\text{eval}} = \mathcal{D}_{\text{train}})$ , and by  $\text{adapt}_{\text{gen}}$  the metric over test data  $(\mathcal{D}_{\text{eval}} = \mathcal{D}_{\text{test}})$ .
5 Memorizing datasets
Our first experiment investigates the extent to which task adaptation paradigms can memorize different types of tasks. We are interested in memorization because many real NLP tasks involve some degree of memorization. For example, translation builds on memorizing lexical associations between words in various languages, and semantic similarity and paraphrasing require memorizing word meanings and/or groupings of semantically similar words.
Method We use training-set adaptability (adapt_mem) as an indicator of a model's memorization ability (Fig. 2). We train on a set of 1000 examples, and plot the value of Eq. (2) on each atomic task as models are adapted via fine-tuning or prompt-tuning. This allows us to visualize both the final training-set performance, as well as the time it took to achieve that performance, both of which we use to quantify memorization ability.
Results Figure 2 shows the training curves for fine-tuning (left) and prompt-tuning (right), on different types of tasks. Solid lines show  $\mathsf{adapt}_{\mathsf{mem}}$  while dashed lines show  $\mathsf{adapt}_{\mathsf{gen}}$ .
Under both adaptation paradigms, we find that lexical tasks are easier to memorize than factual tasks, while random tasks are the hardest to memorize. However, for fine-tuning, we find that models can (eventually) learn to perfectly memorize all types of tasks—even entirely random word associations. However, different types of tasks converge at different rates—lexical tasks converge first, followed by factual tasks, followed by random tasks.
Prompt-tuning, with many fewer parameters than fine-tuning, is much less expressive. As shown in Fig. 2, none of the tasks types converge to  $100\%$  accuracy across tasks. Prompt-tuning overall also takes significantly longer to converge; in particular, on random tasks, the finetuned model generally converges at  $\sim 20k$  updates, while the prompt-tuned model takes over  $200k$  updates to even begin performing nontrivially.
Atomic	Word-level Comp	Seq Comp
FFT	46.9±4.0	39.5±2.1	21.5±1.9
FPT	42.6±4.3	28.1±2.4	11.5±1.4
32FT	33.6±3.8	22.2±1.8	5.7±0.9
32PT	32.4±3.6	21.7±1.7	6.9±1.1
Table 1: Model (generalization) adaptabilities to atomic, word-level compositional, and sequential compositional tasks, under full fine-tuning (FFT), full prompt-tuning (FPT), 32-shot fine-tuning (32FT) and 32-shot prompt-tuning (32PT). Prompt-tuned models are comparable to fine-tuned models for atomic tasks, but not for compositional tasks. However, this distinction disappears under few-shot learning.
However, despite being much worse at memorization, prompt-tuned models still generalize almost as well as fully fine-tuned models, at least on atomic tasks. This suggests that the inability to memorize arbitrary functions is not necessarily a problem for prompt-tuning in general, and more broadly that overfitting the training set—at least during fine-tuning—may not be necessary for generalization.
In Appendix E, we run a version of this experiment on permuted task labels in order to better disentangle the effect of learning novel tasks vs. retrieving existing ones. We find that, for both prompt-tuning and fine-tuning, pre-trained models can more easily adapt to existing relations than to novel (permuted) ones, but they are still able to adapt to new tasks, especially compared to non-pretrained models.
6 Composing tasks
In the previous section, we found that while prompt-tuning cannot memorize arbitrary tasks like fine-tuning, it can still generalize well on simple atomic tasks, almost comparably to fine-tuning. In this section we investigate whether this finding extends to more complex tasks. Specifically, we examine the behavior of prompt-tuned and fine-tuned models when adapted to compositions of atomic tasks.
Many prior studies of compositionality focus on instance-level compositionality (Lake and Baroni, 2018; Keysers et al., 2020): they test whether models can generalize to new instances by combining information from previously-seen instances within the same task. For example, Lake and Baroni (2018) study whether models can learn to jump left, after learning to jump, run, and run left. In our work, we instead focus on task-level compo
sitionality, studying whether models can adapt to new tasks that are compositions of simpler tasks on which they are known to perform well. Thus, while a model exhibiting compositional generalization will correctly compose fragments of previously observed training examples, a training procedure exhibiting compositional adaptability will perform well on tasks involving compositions of previously learned skills.
Method We study adaptation to complex tasks by relating performance on atomic tasks with performance on depth-2 compositional tasks. We also study each paradigm under few-shot learning, by creating a random 32-sample subset of each training dataset, and training on that subset. To mitigate the effect of the random seed, we report average performance over 4 different subsets.