SlowGuess/NAACL_2022_OCR huggingface dataset:
Sequential compositions Sequential composition functions convert word-wise tasks to sequence-level tasks. We specifically consider only two sequential functions: map and filter. Note that compositions of multiple maps or multiple filters can instead be expressed as compositions of multiple word-level functions. For example,
$$
\begin{array}{l} \text {m a p} \{\lambda x. \text {o c c u p a t i o n} (x) \} (\text {m a p} \{\lambda x. \\ \text {f a t h e r} (x) \} (S)) \end{array}
$$
(for an input sequence S) is equivalent to
$$
\operatorname {m a p} \{\lambda x. \text {o c c u p a t i o n (f a t h e r (x))} \} (\mathrm {S})
$$
Specifically, we define the following top-level sequential operator
$$
\begin{array}{l} \operatorname {m a p - f i l t e r} \left\{f _ {M}, f _ {F} \right\} \tag {5} \\ = \operatorname {m a p} \left\{f _ {M} \right\} (\text {f i l t e r} \left\{f _ {F} \right\}) \\ \end{array}
$$
where  $f_{M}$  is a word-wise relation and  $f_{F}$  is a wordwise predicate. All recursively-defined sequential operators follow this form. The following are the recursive rules for mapping nested maps and filters into a function of this form: in the base cases,
$$
\begin{array}{r} \operatorname {m a p} \left\{f _ {M} \right\} = \operatorname {m a p - f i l t e r} \left\{f _ {M}, \lambda x. \text {t r u e} \right\} \\ \operatorname {f i l t e r} \left\{f _ {F} \right\} = \operatorname {m a p - f i l t e r} \left\{\lambda x. x, f _ {F} \right\}; \end{array} \tag {6}
$$
in the recursive cases,
$$
\begin{array}{l} \operatorname {m a p} \left\{f _ {M} ^ {\prime} \right\} (\operatorname {m a p - f i l t e r} \left\{f _ {M}, f _ {F} \right\}) \\ = \operatorname {m a p - f i l t e r} \left\{f _ {M} ^ {\prime} \left(f _ {M}\right), f _ {F} \right\} \\ \end{array}
$$
$$
\begin{array}{l} f i l t e r \left\{f _ {F} ^ {\prime} \right\} (\text {m a p - f i l t e r} \left\{f _ {M}, f _ {F} \right\}) \\ = \operatorname {m a p - f i l t e r} \left\{f _ {M}, f _ {F} \wedge f _ {F} ^ {\prime} \left(f _ {M}\right) \right\}. \tag {7} \\ \end{array}
$$
A.2 Dataset creation details
Note that many tasks created through composition will be degenerate or identical to other tasks, even with our heuristic filters. We do a preliminary filter for degenerate tasks by removing tasks for which we have less than 100 samples. We also manually inspect all depth-2 word-level lexical compositions to ensure they are nontrivial and unique.
Word-level lexical tasks For English lexical tasks, we use words that appeared more than 5 times in the Brown corpus (Francis and Kucera, 1979) as our inputs  $x$ . For Spanish lexical tasks, we in use words that appeared at least once in the CESS Spanish Treebank (Martí et al., 2007) as our inputs. This results in a total of 9143 English words and 5298 Spanish words. We then construct outputs for each input word using either WordNet or SentiWordNet. From each task, we filter out samples for which the relations map to an empty set—thus, for a task like intersection(synonym(x), antonym(x)), most samples will be filtered out as the set of synonyms are usually disjoint from the set of antonyms. (This task ends up getting filtered out entirely, as the final number of samples is under 100.)
Word-level factual tasks We use a dump of Wikidata from 2017, taken from Sorokin and Gurevych (2018).<sup>5</sup> We convert each word-level factual task into SPARQL queries which returns a set of input-output data pairs from Wikidata.
For factual relations  $R$ , we create two queries: a sample query which gives us a set of entities that participate in the relation, from which the inputs  $x$  are derived, and a function query that maps specific inputs  $x$  to its set of output entities  $R(x)$ . For factual predicates  $P$ , we create three queries: a positive sample query which gives samples  $x$  for which  $P(x) = \text{true}$ , a negative sample query which gives samples  $x$  for which  $P(x) = \text{false}$ ,
<sup>5</sup>https://public.ukp.informatik.tu-darmstadt. wikidata-dump/wikidata-virtuoso-dump-2017.zip
Task (T)	SPARQL fragment (sparql(T, y))
A(x)	?xA ?y .
union(T1(x),T2(x))	{ sparql(T1(x), y)} UNION { sparql(T2(x), y)}
intersection(T1(x),T2(x))	sparql(T1(x), y) sparql(T2(x), y)
lor(T1(x),T2(x))	BIND(y1 || y2 AS y) sparql(T1(x), y1) sparql(T2(x), y2)
land(T1(x),T2(x))	BIND(y1 && y2 AS y) sparql(T1(x), y1) sparql(T2(x), y2)
Table 2: Rules for mapping word-level factual tasks to SPARQL conditional statements. Blue substrings represent recursive calls to this set of rules, which are to be replaced with their output SPARQL fragments. Note the second argument to the sparql function represents the variable name to output to.
and a function query that maps specific inputs  $x$  to its output boolean value  $P(x)$ .
The SPARQL query is generated recursively given the specification of the task. We define a function task2sparql(T(x), y) which converts tasks T(x) to SPARQL fragments (where the second argument to the function is the variable name we define for the output). We then convert the output of this function into a well-formed query using:
SELECT ?x
WHERE <task2sparql(T(x),y)>
for sample queries and
SELECT ?y
for function queries. Note for function queries that the input  $x$  is provided to us (and is not a variable).
The general rules specifying the task2sparql function are given in Table 2.
Sequential tasks In practice, naively concatenating outputs from a random word sampler to create sequences will return degenerate or trivial sequences for many functions (for example,  $\mathsf{map}\{\lambda \mathbf{x}$  child(x)} is not meaningful for sequences consisting of words that don't refer to humans). Thus, we define a sequence sampler in Algorithm 1 that takes in a sequential function (given in the form from Eq. (5)), an input length  $n$  and an output length  $m\leq n$  , which will always sample sequences with length  $n$  such that the output, when the function is applied to the sequence, is of length  $m$
At a high level, this algorithm samples  $n$  input words which are in the domain of the map relation, and for which the filter predicate returns true, and  $m - n$  input words for which the filter predicate returns false, then permutes and concatenates them.
B Experimental Setup Details
Hyperparameters We adapt a pre-trained T5-base model (24-layer, 220M parameters) to our
tasks. We use an AdamW optimizer with a learning rate of 1.0 for all prompt-tuning experiments, and learning rate of 1e-3 for all fine-tuning experiments. We use batch sizes of 64 for word-level tasks, and 32 for sequential tasks. We run all experiments up to 100 epochs, and run 3-4 trials for each few-shot experiment to estimate average performance over possible choices of few-shot training samples. These hyperparameters were chosen by trial and error on top of default configurations.
Infrastructure and Reproducibility For each task, we adapt our model using a single 32GB NVIDIA V100 GPU, or a single 40GB NVIDIA A100 GPU. Training time varies by training dataset size and maximum number of epochs, but on average (using the hyperparameters specified above) is less than a few hours per task. Prompt-tuning is also more efficient than fine-tuning, updating the parameters of only 100 prompt tokens vs. the full 220M parameters in the model.
Evaluation of Sequential Tasks When evaluating accuracies of sequential tasks (Eq. (1)), note that we must align words in the generated sequence  $y_{i}^{\prime}$  with words in the ground-truth sequence  $y_{i}$ . However, this can be nontrivial, especially under the setting where word and entity boundaries are
Figure 5: Compositionality of map function, when token separators are explicitly provided in the input and output. All adaptation paradigms demonstrate compositionality except for full fine-tuning, where there seems to be a large proportion of tasks for which the model can adapt to sequentially but not atomically.
not explicitly generated by the model. We cannot rely on whitespaces to segment words as a single word can span multiple white-spaces; for example, an entity Will Smith constitutes a single word. Instead, given a ground-truth sequence of  $n$  words (note ground-truth segmentations are present in the dataset), we optimize over all possible length- $n$  segmentations of the generated sequence.
C Compositionality Experiment: Additional Results
Additional results for the compositionality experiment, including all composition functions, and the formula for the best-fit regression line in each case, are reported in Table 3. Furthermore, the map task with explicit segmentation (+separators) is plotted in isolation in Fig. 5.
D Prediction distribution experiment: Additional details
We adapt the model to the task  $\cup (f_e, f_h)$ , constructing the training dataset for  $\cup (f_e, f_h)$  to be balanced — such that the model sees an equal number of examples of form  $(x, f_e(x))$  as  $(x, f_h(x))$ .
Let  $\mathcal{M}_{\cup (f_e,f_h)}$  denote a model adapted to this task. Note that the domains of either function are not always identical, for example the set of entities in the domain of political-party-of(x) (mostly politicians) is different from the set of entities in the domain of position-played-on-sports-team(x) (mostly
athletes). We create a balanced training set by first taking all items in the intersection of both domains, then sampling an equal number number of items in either domain. Furthermore, to minimize the effect of the order seen during training, we shuffle the entire dataset after creating all example-label pairs.
During test-time, we give  $\mathcal{M}_{\cup (f_e,f_h)}$  a novel input  $x^{\prime}$  and record the average probability mass it assigned to all  $y_{e}^{i}\in f_{e}(x^{\prime})$  vs. all  $y_{h}^{i}\in f_{h}(x^{\prime})$ . Note we evaluate only on inputs  $x^{\prime}$  which are in the domain of both  $f_{e}$  and  $f_{h}$ . Under the rare scenario that a prediction is in both target tasks for a particular word (i.e.  $y$  is in both  $f_{e}(x^{\prime})$  and  $f_{h}(x^{\prime}))$ , we count that towards both tasks, and increment the probability mass on both tasks by the probability the model assigned to  $y$ .
Instead of averaging across outputs in either set  $f_{e}(x^{\prime}),f_{h}(x^{\prime})$ , we also looked at the probabilities assigned to highest-scoring predictions from each set. The overall trends were similar: the model tends to assign greater mass to the highest-scoring prediction from the easier task compared to highest-scoring prediction from the harder task.
E Permuting task labels: disentangling effect of “learning” vs. “retrieval”
We hypothesize two ways that pre-trained models might adapt to new tasks: (1) through learning the underlying rules and patterns governing the task, or (2) through learning how to "retrieve" the correct label from memorized pre-training data. These hy
Figure 6: Memorization experiments on permuted vs. non-permuted versions of tasks, using pre-trained vs. non-pretrained models. Left figure shows an averaged memorization curve for a non-pretrained model on permuted tasks. Middle figure shows a pre-trained model on permuted tasks. Right figure shows a pre-trained model on non-permuted tasks. Pre-training enables models to adapt to novel tasks, but adapting to existing, non-permuted tasks is easier than adapting to novel, permuted tasks.
potheses, respectively, suggest two different roles for pre-training: (1) providing a "generally good" initialization from which many different tasks can be learned, or (2) imbuing the LM with memorized knowledge that can later be retrieved.
To determine which effect is at play (for which types of tasks), for each atomic task, we permute the labels associated with each input, then run each adaptation paradigm on the permuted version of the task. Notably, permuted labels differ from random tasks as the input and label distributions are restricted to be identical to original task. Because the model would be unable to generalize to permuted labels, we only look at memorization ability. The setting is similar to Section 5. We compare the rate of adaptation for (A) a non-pretrained model to a permuted task, (B) a pre-trained model to a permuted task, and (C) a pre-trained model to non-permutated task. If a pre-trained model is better able to adapt to a task than the non-pretrained model (B > A), this indicates that pre-training helps models learn new tasks on the fly, supporting hypothesis 1. If a pre-trained model can better adapt to a non-permutated task than it can to a permuted task (C > B), this indicates that adaptation requires some
thing learned during pre-training, supporting hypothesis 2.
Results are shown in Fig. 6 (which, from left to right, shows settings A-C respectively). We find that for fine-tuning and prompt-tuning, both hypotheses are supported. For both lexical and factual tasks, pre-trained models can memorize novel word relations faster than non-pre-trained models. However, pre-trained models can still adapt to non-permuted tasks faster than permuted ones. Furthermore, note that for fine-tuning, the order of convergence of the three task types is reversed when going from permuted tasks to non-permuted tasks. In particular, random relations are easier to learn than permuted lexical or factual tasks. This suggests that models can more easily to adapt to random labels than labels that are known to be false.
Function type	Training type	Avg. adaptability	Optimal formula	r2value
Chainingf2(f1)	Full Fine-tuning	37.43±3.18	1.27x + 0.14	0.56
Full Prompt-tuning	22.37±3.03	1.32x + 0.05	0.65
32-shot Fine-tuning	18.59±2.21	1.34x + 0.07	0.57
32-shot Prompt-tuning	18.19±2.21	1.32x + 0.07	0.6
Unionf2∪f1	Full Fine-tuning	31.18±2.02	1.24x + 0.02	0.73
Full Prompt-tuning	25.05±2.11	1.4x - 0.01	0.83
32-shot Fine-tuning	17.28±1.52	1.37x + 0.02	0.8
32-shot Prompt-tuning	18.43±1.55	1.35x + 0.02	0.8
Intersectionf2∩f1	Full Fine-tuning	43.31±22.42	2.25x - 0.12*	0.97*
Full Prompt-tuning	16.68±8.78	1.64x - 0.04*	0.98*
32-shot Fine-tuning	22.77±17.03	5.93x - 0.12*	0.91*
32-shot Prompt-tuning	25.91±19.38	6.81x - 0.12*	0.94*
Logical Andf1 ∧ f2	Full Fine-tuning	78.39±2.53	2.15x - 0.85*	0.8*
Full Prompt-tuning	79.25±2.57	1.27x - 0.18*	0.58*
32-shot Fine-tuning	66.49±2.55	4.75x - 2.13*	0.88*
32-shot Prompt-tuning	55.86±1.22	0.48x + 0.3*	0.05*
Logical Orf1 ∨ f2	Full Fine-tuning	72.41±1.97	1.39x - 0.37*	0.54*
Full Prompt-tuning	74.71±2.01	1.15x - 0.18*	0.48*
32-shot Fine-tuning	58.04±1.11	1.52x - 0.35*	0.63*
32-shot Prompt-tuning	53.91±0.48	0.8x + 0.1*	0.33*
Mapmap{λx.fM(x)}	Full Fine-tuning	13.44±1.73	0.15x + 0.09	0.03
Full Prompt-tuning	5.39±0.93	0.13x + 0.03	0.07
32-shot Fine-tuning	3.59±0.70	0.21x + 0.0	0.2
32-shot Prompt-tuning	3.77±0.85	0.3x - 0.01	0.29
Map(+separators)map{λx.fM(x)}	Full Fine-tuning	67.40±2.51	0.49x + 0.52	0.17
Full Prompt-tuning	18.02±1.96	0.83x + 0.02	0.64
32-shot Fine-tuning	10.66±1.34	0.79x - 0.01	0.86
32-shot Prompt-tuning	5.22±1.14	0.57x - 0.04	0.64
Filterfilter{λx.fF(x)}	Full Fine-tuning	82.08±5.92	1.59x - 0.58*	0.95*
Full Prompt-tuning	78.58±5.43	1.38x - 0.43*	0.95*
32-shot Fine-tuning	38.39±3.27	0.81x - 0.24*	0.87*
32-shot Prompt-tuning	51.58±4.99	1.19x - 0.43*	0.87*
Table 3: We study the correlation between the atomic word-level functions and their compositions, under various training paradigms. We train a linear regressor to predict a model's generalization adaptability on a composite function based on its adaptabilities on the atomic constituents. Finally, we report the average generalization adaptability of composite tasks, for each training paradigm, under each type of composition.
* indicates composition function has less than 20 tasks, thus reported numbers may not be significant.
Category	Predicates	Relations
Lexical	is-POS-noun[eng]	synonyms[eng]	synonyms[spa]
is-POS-verb[eng]	antonyms[eng]	antonyms[spa]
is-POS-adjective[eng]	hyponyms[eng]	hyponyms[spa]
is-POS-adverb[eng]	entailments[eng]	entailments[spa]
is-sentiment-positive[eng]	translate[eng->spa]	translate[spa->eng]
is-sentiment-negative[eng]
is-sentiment-neutral[eng]
Factual	is-instance-human	child	location[inv]
is-instance-film	child[inv]	manufacturer
is-instance-book	continent	member of political party
is-instance-city	country of citizenship	member of sports team
is-instance-taxon	country of origin	mother
is-occupation-actor	country	mother[inv]
is-occupation-politician	creator	named after
is-occupation-writer	creator[inv]	native language
is-occupation-journalist	developer	occupation
is-occupation-teacher	diplomatic relation	official language
is-occupation-composer	father	original language of film or TV show
is-birthplace-london	father[inv]	owned by
is-birthplace-nyc	genre	performer
is-birthplace-la	has part	place of birth
is-birthplace-buenosaires	head of state	place of death
head of state[inv]	position held
influenced by	position played on team
languages spoken written or signed	record label
location	sex or gender
Random	random-seed@[eng]	random-seed2[eng]
random-seed1[eng]	random-seed3[eng]
Table 4: Full list of atomic tasks in TASKBENCH500. The content inside brackets specifies task input and output languages (eng for English and spa for Spanish). {inv} indicates the task is inverted, e.g. creator takes creations as input and returns their creators, while creator{inv} takes creators as input and returns their creations.
Category	Function	Example Tasks	Example Data
Chaining	chain	mother(head of state)	Russia → {Maria Ivanovna Putina}
Set Operations	union	union(mother, father)	Elizabeth I of England → {Anne Boleyn, Henry VIII of England}
intersection	intersection(entailments[eng], synonyms[eng])	live → {be, exist}
Logical Operations	land	land(is-occupation-actor, is-birthplace-nyc)	Anne Hathaway → true
Brad Pitt → false
Franklin Delano Roosevelt → false
lor	lor(is-birthplace-london, is-birthplace-nyc)	Franklin Delano Roosevelt → true
David Bekham → true
Mao Zedong → false
Sequential Operations	map	map{λx. synonyms[eng]}(S)	criticality pillow delinquent culture eternity cling sane sentry → {···, criticalness rest neglectful acculturation timelessness cohere reasonable spotter,···}
filter	λx. filter{is-POS-noun[eng]} (S)	expect inexpensive direct bones sullen breed switching eight → {bones breed switching eight}
Table 5: Full list of composition functions used in TASKBENCH500, with examples.
Table 6: Full list of word-level compositional tasks in TASKBENCH500, organized by composition type.
Table 7: Full list of sequential compositional tasks in TASKBENCH500, organized by composition type.
Table 8: Full list of sequential compositional tasks in TASKBENCH500, organized by composition type. (Continued from previous page.)