Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners
Authors: Seonghyeon Ye
Doyoung Kim
Joel Jang
Joongbo Shin
Minjoon Seo
Venue: ICLR 2023 poster (2023)
Keywords: natural language processing
zeroshot language models
large language models

Abstract: Meta-training, which fine-tunes the language model (LM) on various downstream tasks by maximizing the likelihood of the target label given the task instruction and input instance, has improved the zero-shot task generalization performance. However, meta-trained LMs still struggle to generalize to challenging tasks containing novel labels unseen during meta-training. In this paper, we propose Flipped Learning, an alternative method of meta-training which trains the LM to generate the task instruction given the input instance and label. During inference, the LM trained with Flipped Learning, referred to as FLIPPED, selects the label option that is most likely to generate the task instruction. On 14 tasks of the BIG-bench benchmark, the 11B-sized FLIPPED outperforms zero-shot T0-11B (Sanh et al, 2021) and even a 16 times larger 3-shot GPT-3 (175B) (Brown et al, 2020) on average by 8.4% and 9.7% points, respectively. FLIPPED gives particularly large improvements on tasks with unseen labels, outperforming T0-11B by up to +20% average F1 score. This indicates that the strong task generalization of FLIPPED comes from improved generalization to novel labels. We release our code at github.com/seonghyeonye/Flipped-Learning.

GUESS THE INSTRUCTION!
FLIPPED LEARNING MAKES LANGUAGE MODELS STRONGER ZERO-SHOT LEARNERS
Seonghyeon Ye
Doyoung Kim!
Joel Jang"*
Joongbo Shin²
Minjoon Sco!
KAIST LG AI Research
(seonghyeon ye, ikevin98, joel min miag kr jb.
ABSTRACT
Meta-training, which fine-tunes the language model (LM) on various downstream tasks by maximizing the likelihood of the target label given the task instruc- tion and input instance, has improved the zero-shot task generalization perfor- mance. However, meta-trained LMs still struggle to generalize to challenging tasks containing novel labels unseen during meta-training. In this paper, we pro- pose FLIPPED LEARNING, alternative method of traiiiiriinnng which trains the LM to generate the task instruction given the input instance and label. During in- ference, the LMM trained with FLIPPED LEARNING, referred to as FLIPPEDD selects the label option that most likely to generate the task instruction. On 14 tasks of the BIG-bench benchmark, the 11B-sized FLIPPED outperforms zero-shot TO- 11B (Sanh et al., [02I] and even 16 times larger 3-shot GPT-3 (175B) (Brown) al. 2020) on average by 8.4% and 9.7% points, respectively. FLIPPED gives par- ticularly large improvements on tasks with unseen labels, outperforming TO-1 by up to +20% average FI score. This indicates that the strong task generalization of FLIPPED comes from improved generalization to novel labels. We release our code at gittb..com/sennggooogg///lleeeep
INTRODUCTION
Large Language Models (LMs) pretrained downstream tasks through instructions (task prompts) concatenated with the input instances without
amount of corpora capable of solving various
any task-specific fine-tuning (Brown et al. 2020 Rac al. 2021 Chowdhery et al., 2022] Zhang et al. 2022). Previous work has shown that fine-tuning the LM on various downstream tasks by generating the correct answer given prompted input (instruction and input), also referred to meta-training, leads to significant improvement in zero-shot task generalization (Sanh et al. 2021 Wei al., 2021; Wang et al. 2022). However, Webson & Pavlick (2021): }); Min al. (2022c) show that LMs meta- treenet through this standard approach are sensitive to different label words, implying that standard meta-traines LMS often fail generalize to tasks that contain novel labels.
In this paper, we introduce an alternative meta- teeta-training method called FLIPPED LEARNING that flips the task instruction and label space, training the underlying L. to generate the instruction when given the input instance and label. This differs from the standard meta-training methods which train the LM to generate the label given instruction and input instance (DIRECT) or generate instruction and input instance given the label (CHANNEL). Also, add unlikelihood loss for FLIPPED LEARNING, making the LM not generate the task instruction for an incorrect label option. During inference, the LM trained via FLIPPED LEARNING, referred to FLIPPED, selects the label option that is most likely to generate the task instruction, as shown in Figure]] I
To compare with existing meta-trained LM TO (annh et al. 2021) trained by the DIRECT ap- proach, we implement FLIPPED by meta-training the T5 (Raffel et al. 2019) model on 20 different
Work done while interning at 1.G AI Research.

---

Direct
Channel
Flipped
sentence positive? What great day!
Is this sentence positive?
Plyl4.))
P(yl4.x)
Pu.xly
Pu.xly)
x))
x()(x,y)
f
t
sentence positive? What a great day!
great day! Yes
great day!
No
(Ix.y) = (Is this sentence positive?, What great day!, Yes)
Figure 1: Inference of DIRECT, CHANNEl and FLIPPED to select an appropriate label (Yes) from label options (Yes/No). DIRECT, which is the standard LM inference, computes the conditional probability of label given instruction+input. CHANNEL, which is noisy channel inference, computes the conditional probability of instruction+innut given label. Our FLIPPED computes the conditional probability of instruction given lnput+label.
70
Flipped O-shot
60
Channel O-shot
Mean Acc
TO 0-shot
GPT-3 0-shot
50
GPT-3 1-shot
GPT-3 3-shot
PaLM 0-shot
40
PaLM 1-shot
PaLM 2-shot
30
1010
10
Model Parameters
Figure 2: Mean Accuracy on 14 datasets from the BIG-Bench benchmark. FLIPPED shows the best performance among zero-shot LMs and even better performance than GPT-3 175B 3-shot.
datasets (around half of the datasets used to train TO) with only ~5% of training compute com- pared to TO. Evaluation on 14 datasets from BIG-Bench (Srivastava al., (022) demonstrate that FLIPPED is effective (Figure not only showing state-of-the-art performance compared to all LMs regardless of size the zero-shot setting, but also outperforming much larger GPT-3 175B (3-shot) by significant margin, even without any demonstrations of the task (zero-shot). We also compare FLIPPED with baseline models on 14 additional common English NLP tasks, further amplifying its effectiveness compared to previous methods and models.
We hypothesize that FLIPPED shows strong zero-shot generalization ability on unseen tasks be- cause of the improved generalization capability unseen labels. To test this hypothesis, eval- uate on various label pairs with different surface forms but with the same meaning (e.g. yes/no vs agree/disagree). Results show FLIPPED has up +20% average F1 score performance gap with TO- 11B, indicating that FL,IPPED LEARNING indeed significantly improves label generalization capa- bility. This hypothesis further bolstered by the fact that the tasks that show significant performance improvement from the baselines among the 28 evaluation datasets are datasets with unseen labels during meta-training training. Because FLIPPED LEARNING conditions on the label instead of generating it, FLIPPEd LEARNING is likely to avoid label overfitting, resulting in improved label generalization, which consequently leads better task generalization.
In summary, our contributions are as follows:
We propose FLIPPED LEARNING, a novel meta-trant meta-training method that computes the likeli- hood of the task instruction given the concatenation of input instance and label. By adding
2

---

unlikelihood loss, make LMs generate the task instruction depending the input instance-label correspondence.
On 14 datasets from the BIG-Bench benchmark, we show that 11B-sized FLIPPED (LM trained through FLIPPED LEARNING) outperforms not only meta-trained TO-11B by 8.4% points on average, but also 16x larger 3-shot GPT-3 by 9.7% points. When evaluating on 14 additional English NLP tasks, FLIPPED outperforms all baseline models on average, further demonstrating the effectiveness of our proposed method.
We show that FLIPPED is particularly effective generalization to labels that unseen during meta-training, outperforming TO-11B by up to 20% average FI score for novel label pairs.
RELATED WORK
2.1 META-TRAINING
Prior work has shown that meta-training. multitask fine-tuning on various downstream tasks with task instructions included, enables zero-shot task generalization (Sanh al. 2021 Wei al. 2021 Wang et al. 2022 Mishra et al., 2022). Specifically, [Sanh et al. (2021) Wang et al. (2022) have shown that moderate-sized LMs can also generalize to unseen tasks through meta-training and the
generalization performance improves by scaling the number of training tasks, the number of prompts per task, and the size of the LM. Based on this method, Ouyang al. (2022) apply reinforcement learning with human feedback after meta-training to make better instruction-followin LMs. To improve the task generalization performance of meta-trained LMs. [in et al. (2022) ; suggest using retrieval-based framework. Min et al. (2022b); [Chan et al. (2022) Chen et al. 2021)
Ye et al. (2022)
apply meta-training by using input-label pairs instead task instructions.
2.2 NOISY CHANNEL PROMPTING
When performing classification tasks, zero-shot LMs (Brown et al. 2020 Chowdhery et al. 2022) compute the conditional probability the labels given input instances concatenated with instruc- tions demonstrations, referred to direct prompting. On the other hand, noisy channel prompting reverts the input and the output space, making LMs generate every word in the input instances when conditioned on the label (Min et al. 2022a Lazaridou et al.) 2022). Specifically, Min et al. (2022h) apply noisy channel prompting during meta-training, optimizing the model to generate the input in- stance given the concatenation of demonstrations and the label. Motivated from Min al. (2022b). we optimize the model to generate only the task instruction while conditioning on the input and label (example shown in Figure [].. While Honovvic al. (2022); Gupta et al. (2022) have similar intuition of guessing the instruction given input and label, they only do flipping either training or inference, not both.
2.3 LABEL GENERALIZATION
Previous work has shown that LMs very sensitive to different label surface forms, indicating poor robustness. Zhao et al. (02T)) show that even 175B-sized GPT-3 suffers from high sensitivity and propose contextual calibration define this problem as surface form competition and propose Domain Conditional Pointwise Mutual Information scoring or fuzzy verbalizers to mitigate this problem. For meta-training, Webson & Pavlick (202I) analyze the effect of various label surface forms for a meta-trained LM and find that meta-trained LMs are more sensitive label surface forms than different wordings of the prompt, which suggests that the meta-trainet LMs overfit to the label space provided during meta veta--tainiig. This shows that meta-traines LMs cannot generalize to unseen label space, indicating poor label generalization.
solve this issue. Holtzman et al. (2021); Shi et al. (2022)
3 FLIPPED LEARNING
In this section, we introduce FLIPPED LEARNING, which trains the LM to compute the conditional probability of the task instruction given input instance and label (Figure I] We first introduce
3

---

notations and compare the difference between previous approaches (DIRECT and CHANNEL) and our proposed method during inference (Section 3.1). Next, we provide detailed explanation of the training objective of FLIPPED LEARNING during meta-training and explain the intuition for including an unlikelihood training loss in addition to the original loss (Section 3.2).
3.1 INFERENCE OF PROBABILISTIC LMS
In this work, we focus on tasks with label options such classification and multi-choice tasks for both meta- training and evaluation. For given task T = {z, Y) where x is the input instance and Y {811 M/k } label option set, we convert the data instance into a prompted version {[4, .2). L}. From {14, x L) [4, æ] denotes the prompted input instance including natural language instruction and I, (71,-... 1+) denotes the natural language label option set where = 8y (3k) and by the
verbalizer corresponding to I. The goal during inference is to select the correct 1, from I. {
given / and x.
DIRECT method computes the conditional probability of the label given task instruction and input instance. During inference, it selects the label that leads to the highest conditional probability:
arg max [v)
This is the most common approach used for zero-shot inference of LMs (Brown et al. 2020 Chowd- hery et
2022 Sanh et al. 2021 Wei et al. 202T).
CHANNEL method (Min et al. 2022a) computes the conditional probability of instruction and input instance given label. Using Bayes rule, the probability can be reparameterized follows:
P(I, P((.)) arg max P(T,x(14)
arg max P(L]T, = arg max
P(I,x)
since P(I,x) x)) is independent from and P(L) μ we assume the prior be an uniform distri- bution for tasks with label options.
FLIPPED LEARNING, our proposed method, computes the conditional probability of the task in- struction given an input instance and a label. Different from previous approaches, we separate [T, 2] into / and x and use Bayes' rule to reparameterize the conditional probability follows:
arg max P(L]I, x) arg max P(I ar 4) P().
arg max P(I arg P(I
2(),z)
4
where we assume P(L[z) R + for simplicity. By considering P(J[2, 4:), we allow the LM to put more focus on the task instruction. The intuition of FLIPPED LEARNING can be considered to be similar generative question answering (Lewis & Fan 2018) which generates the question given context and answer, but FLIPPED LEARNING generates the task instruction for task generalization.
To compute P(] 4,) for FLIPPED LEARNING, the prompted input [4, should be separated into task instruction / and input instance æ. However, the prompted input might be sometimes an inter- mix of I and rather than their sequential concatenation. To handle this, we follow Raffel et al. 2019)'s 's denoising objective using sentinel tokens where the portions representing task instruction are replaced by sentinel tokens and the LM learns to denoise the sentinel tokens by generating |
3.2 TRAINING USING FLIPPED LEARNING
Next, we explain how optimize the LM to utilize P([ 4:) which requires adding in an unlike- lihood loss during meta-training. Given the sequence of task instruction / = (J,,.. IT), we denote the LM loss function follows:
T LLM = Σ log P(I,,, Ier)
(4)
where le corresponds to the correct label option. By minimizing this loss function, the LM learns to generate I when given the correct label option and the input instance.
We illustrate the denoising objective example in Appendix
4

---

Unlikelihood Loss However, from preliminary experiments, observe that meta-training the LM only on LLM results in ignoring the correspondence between the input instance and label: meta-trained LM generates task instruction I regardless of the correspondence of the label option. We conjecture that this is a result of shortcut learning of large LMs (Du et al. 2022; Min et al.) 2022c). To amplify the correspondence signal between the input instance and the correct label, add an unlikelihood loss (Tam et al. 2021 Liu et
2022 Welleck et al. 2019) during meta- trainir
which can be denoted follows:
T
LUL =
log(1 11 - P(Iel,, Ict))
(5)
where le corresponds an incorrect label option randomly sampled from the incorrect label option set LC- = {2]l E L,1 * le). This unlikelihood loss term allows the LM to nol generate the task instruction if the label option does not correspond to the input instances. The final training objective of FLIPPED LEARNING is the weighted sum of LLM and Lu
I. LLM + ALUE
where À is hyperparameter. By optimizing both likelihood and unlikelihood objectives, the LM is optimized to generate the instruction when given the correct label and not generate the instruction when given the incorrect label, strengthening the correspondence between the input instance and the correct label.
EXPERIMENTAL SETUP
Training For meta-training, we utilize the subset of TO (Sanh et al. 2021) meta-training datasets: task clusters (sentiment classification, paraphrase detection, topic classification, multi-choice QA), which are 20 datasets in total. We only train on tasks with label options and exclude tasks such as free-form generation because FLIPPED LEARNING requires label options for unlikelihood training on incorrect label options. We provide detailed training configurations in Appendix D and the full list of training datasets in Appendix E.1
Evaluation Following the evaluation setting of [Snnh et al. 2021 generalization performance on 14 tasks of BIG-bench which contain challenging and various tasks that are unseen during meta-training. For each of the BIG-bench tasks, we report the accuracy of a single instruction for each task following the convention of past work (Sanh et al.] 2021 Lin al. 2022). Furthermore, we additionally evaluate on 14 English NLP unseen tasks, consisting of 7 classification and 7 multi-choice datasets, also following the setting of [Sanh et al.] (2021); Lin al. (2022). For evaluation metric, we use Macro-F for classification and accuracy for multi- choice tasks, following Min et al. (2022b)c). We also report the average standard deviation among different evaluation instructions, indicating the robustness of different wordings of the evaluation instruction (the lower, the better). For analysis of label generalization of classification tasks, evaluate on 5 datasets: seen datasets during meta-training (IMDB, PAWS) and unseen datasets (RTE, CB, WiC). We provide the full list of evaluation datasets in Appendix E.2 and more details on the evaluation setting are specified in Appendix F
first measure unseen task
Baselines We evaluate several baselines observe the effectiveness of FLIPPED LEARNING: (1) TO-3B, a 3B-sized meta-trained LM by |aanh et al. (2021), (2) DIRECT, a 3B-sized meta-trained LM using the same language modeling objective (standard method) of TO-3B, but with training configurations, (3) CHANNEL, a 3B-sized meta-trained LM using noisy channel language modeling objective, (4) FLIPPED-3B. 3B-sized LM meta-trained through our proposed FLIPPED LEARNING, (5) TO-1 B, a larger meta-trained LM Sanh et al. (2021). (6) FL.IPPED- 11B, larger meta-trained LM trained through FLIpped LEARNING, (7) GPT-3 (Brown et al., 2020). 175B sized pretrained LMM, (8) PaLM (Chowdhery et al. (022), 540B sized pretrained LM. Note that DIRECT, CHANNEL,
and FLIPPED is meta-trained on the same number of datasets and training steps.
Macro-F1 is more appropriate for imbalanced classification than accuracy.
5

---

Zero-shot
Few-shot
T0
DIR.
CHAN.
FLIP.
TO
FLIP.
GPT-3
PALM
GPT-3 (3)
PALM (1)
Dataset (metric)
3B
3B
3B
11B
LIB
175B
540B
175B
540B
Known Un.
47.83
63.04
52.17
71.74
58.70
86.96
60.87
56.52
50.00
67.39
Lugic Grid
41.10
35,90
30,90
41.70
38,30
42,50
31,20
32.10
31.10
42.20
Strallgy.
52.79
53.01
53.19
52.75
53.23
$2.30
64.00
57.10
69.00
Hindu Kn.
25.71
30.29
16.57
47.43
29.71
52.57
32.57
56,00
58,29
94.86
Movie
52.85
47.15
51.06
47.93
53.69
48.49
51.40
49.10
49.40
57.20
Code D.
46.67
33.33
71.67
45.00
43.33
60.00
31.67
25.00
31.67
61.67
Concept
45.52
58.14
35.67
61.64
69.29
64.93
26.78
59.26
35.75
80.02
Language
14.84
22.01
11.55
19.01
20.20
26.87
15,90
20.10
10.90
37.30
Vitamin
58.89
63.83
15.73
$7.07
64.73
65.57
12.30
14.10
52.70
70.40
Syllogism
52.94
49.85
50.43
50.56
51.81
50.39
50.50
49,90
52.80
$2.20
Misconcept.
50.23
50.23
47.79
46.58
$0.00
54.34
47.95
47.49
60.27
77.63
Logical
46.64
38.06
25.73
59.82
54.86
64.56
23,42
24.22
33.93
34.42
Winowhy
44,29
44,33
55.36
53.33
$2,11
55.08
51,50
45,30
56.50
47.50
Novel Con.
15.63
3.13
15.63
25.00
15.63
46.88
46.88
46.88
56.25
59.38
BIG-bench AVG
42.56
43.75
38.07
48.57
46.79
55.17
38.23
42.14
45.48
60.80
Table 1: Task generalization performance on BIG-bench tasks. DIR. denotes DIRECT, CHAN. denotes CHANNEL, and FLIP. denotes FLIPPED. Parentheses in the Few-shot column denote the number of shots. FLIPPED performs the best on average for zero-shot setting.
5 EXPERIMENTAL RESULTS
In this section, evaluate the effectiveness of FLIPPED compared to various baselines. For task generalization performance, we evaluate on 14 tasks of BIG-bench and 14 common English NLP tasks (Section [5.Ip. For analysis of FLIPPED, evaluate on multiple label options with the same meaning but with different surface forms (Section 5.2
5.1
MAIN RESULTS
DIRECT outperforms TO-3B. Our implementation of DIRECT outperforms TO-3B significantly despite using only half of the datasets used to train TO (20 out of 38), indicating competitive perfor- mance of baselines. Note that TO-3B and DIRECT with the only difference in training configurations, showing that our training setting is more optimal evaluate the task generalization performance of meta-trained LMs. We conjecture the significant performance improvement to come from 3 potential factors: (1) We train for about 5% token updates compared Sanh et al. (202I), implying that DIRECT avoids overfitting to the training dataset (2) DIRECT includes the EOS (end-of-sequence) token during training and evaluation unlike TO-3B, (3) We do not use sequence-packingg during meta-training. Our DIRECT baseline shows that the task generalization performance from Sanh et al. 20221) might have been underestimated
trained the
training objective,
CHANNEL is not effective for task generalization. CHANNEL method largely underperforms DIRECT and TO, showing close to random guessing performance for many unseen tasks. This result consistent with that of Min et al. (2022b) in that instructions zero-shot (not few-shot) setting worsen the task generalization performance for CHANNEL method unlike the DIRECT method. We conjecture that because prompted inputs mostly question-answer formats, it is unnatural for CHANNEL to generate a question-like instruction given only an answer even after meta-training.
FLIPPED outperforms baselines. For the 14 BIG-bench tasks of Table I FLIPPED-3B signifi- cantly outperforms all mnta-trained models with the same model size; +6.01% 01% mean accuracy com- pared to TO-3B and +4.82% mean accuracy compared to DIRECT. FLIPPED-3B also outperforms 4x
Although one might think that the other 18 training datasets of TO might have had negative impact on unseen tasks, we ohserve that training whole datasets also shows similar results to DIRECT from preliminary experiments, implying that the performance improvement from the 3 potential factors mentioned above.

---

TO
DIR.
CHAN.
FLP
TO
FLIP.
GPT-3
Dataset (metric)
3B
3B
3B
11B
11B
175B
RTE (FI)
61.89
72.83
36.62
71.03
80.91
72.20
40.68
CB (F1)
30.94
49.81
22.35
52.27
53.82
61.51
29.72
ANLI (F1)
24.39
30.17
21.30
33.92
34.72
34.93
20.90
ANLI R2 (F1)
23.73
28.23
21.44
32.62
31.25
32.59
22.50
ANLI (FI)
23.45
30.41
22.50
34.65
33.84
34.77
23.77
WSC (FI)
54.64
50.35
46.38
52.82
58.36
49.88
26.24
WiC (FI)
38.53
36.42
38.69
37.36
51.64
39.26
45.36
COPA
75.88
89.63
50.13
89.88
91.50
90.75
91.00 91.00
Hellaswag
27.43
31.61
20.82
41.64
33.05
41.97
78.90
StoryCloze
84.03
94.24
57.84
88
92.40
96.12
83.20
Winogramde
50.97
55.96
50.99
58.56
59.94
66.57
70.20
PIQA
56.63
62.60
47.08
67.32
67.67
71.65
81.00
ARC-Chall
51,10
49.30
29.23
49.63
56.99
64.62
51,40
OpenbookQA QA
42.66
54.00
38.57
59.11
72.54
NLP AVG
46.16
52.54
36.00
55.69
57.51
59.24
52.41
En NLP STD (1)
4.74
4.36
4.58
3,29
5.24
3,11
Table Zero-shot task generalization performance on 14 English NLP tasks consisted of 7 classifi- cation and 7 multi-choice tasks. 11B-sized FLIPPED (FLIP.) shows the best performance on average and also shows the best robustness to different evaluation instructions (lower STD).
times larger meta-trained TO-11B on average by +1.78% points. This result is significant consider- ing that the effect of scaling law is strong for zero-shot generalization of meta-trained models (Wei (ct al., 2021] Sanh et al.) 2021 Wei ct al. 2022). FLIPPED-IIB even shows better performance, outperforming T0-1IB on average +8.38% points. Compared to even larger pretrained LMs eval- uated a few-shot setting, FLIPPED- 1B outperforms 3-shot GPT-3 which is 16x larger by 9.69% points average. When compared to 1-shot PaLM which is 50x larger, FLIPPED outperforms tasks out of the 14 tasks. This shows that FLIPPED is effective for generalizing to unseen tasks that challenging, resulting in the best performance the zero-shot setting even when compared to LMs with much larger sizes.
For the 14 common English NLP tasks which are consisted of 7 classification and 7 multi-choice tasks shown in T FLIPPED-3B outperforms baseline models with the same model size (TO-3B, DIRECT, CHANNEL) on task generalization performance by significant margin, largely reducing the gap between TO-1 1B. FLIPPED-1 1B shows the best performance average, outperforming TO- 11B by 1.73% points. Also, FLIPPED shows the lowest standard deviation among multiple different evaluation instructions compared to other meta-trained baseline models, including TO-1 This indicates that FLIPPED not only effective for zero-shot task generalization but also robust to different surface forms of the instruction.
5.2 ANALYSIS OF FLIPPED
FLIPPED significantly outperforms baselines for tasks with unseen labels. We first analyze the tasks that FLIPPED outperforms baseline models and identify clear correlation: FLIPPED especially shows strong performance on tasks that contain many label options unseen during meta- tratt-ttriiing.. For RTE, WSC. WiC datasets, which are consisted of seen label options (yes/no), direct meta-trained trained LMs (DIRECT, TO) show strong performance as shown in Table 2 On the other hand, FLIPPED shows strong performance on CB and ANLI datasets that contain an unseen label (e.g. maybe). This can be seen result of effeetive label generalization of FLIPPED; the prediction of TO and DIRECT is largely biased to label options that are seen during training (yes/no) while FLIPPED makes balanced predictions (yes/no/maaybe). Although the calibration method of Zhao et al. (2021) known to mitigate the prediction bias LMs, we find that calibration worsens the performance of meta-trained LMs (Appendix G).

---

(a) RTB
(b) CB
(c) WSC
(d) IMDB
(e) PAWS
Direct (3B)
Channel (3B)
TO-11B
Flipped (3B)
Figure 3: Label generalization performance on 3 unseen and seen datasets during meta-training. We evaluate on 20 different label pairs including many unseen labels. Result shows that FLIPPED significantly outperforms other baseline models.
For multi-choice tasks in Table which are more likely to contain many unseen labels because they have different label options for every data instance, FLIPPED outperforms meta- veea-ttainee LMs with the same size for most tasks. Moreover, FLIPPED outperforms TO- IB on BIG-bench (Table which is mostly consisted of unseen labels. From these findings, we can hypothesize that the strong zero-shot task generalization of FLIPPED likely to come from its strong label generalization capability.
FLIPPED generalizes to unseen labels that are semantically the same. To further test the above hypothesis, we analyze the label generalization performance FLIPPED compared to other baseline models by varying the surface form the label options (e.g. yes/no agree/disagree) for 5 classi- fication datasets: 3 datasets (RTE, CB, WSC) for unseen tasks, and 2 datasets (IMDB, PAWS) for seen tasks during meta-training. We vary the label options 20 different pairs that have the same meaning but different surface forms including the original labels [
Figure 3 shows the label generalization performance of TO-3B, DIRECT, TO-11B and FLIPPED-3B. For unseen tasks, FLIPPED outperforms T0-3B by (+23.37%, +18. +18.78% +10.92%), outperforms DI- RECT by (+16.42%, +13.46%. +7.6 +7.82%). and outperforms CHANNEL by (+21.88 888, 24.84%, 9.93% average F1 score on (RTE, CB, WSC) respectively. Even when compared with a 4x times larger meta-trained LM (TO- IB), FLIPPED outperforms by (+ (+19.72% +12.32%, +10.81% 1)% average score for (RTE, CB, WSC) respectively. This shows that FLIPPED can generalize to various novel labels, which is what even larger meta-trained LMs trained through direct prompting cannot do. Al- though baseline models outperform FLIPPED for best accuracy among different label pairs, this is mostly when the label is seen during meta-training training (c.g. yes/no). The result of Figure a alsso indicates that the classification tasks evaluation setting of [aanh al. (2021) overestimates the true general- ization ability LMs because Sanh et al.) (2021) mostly evaluate unseen target tasks on labels that seen during meta-training (yes/no), which is not guaranteed for zero-shot generalization scenario.
Aligned with the experiments the unseen tasks, FLIPPED further outperforms baselines the seen tasks during meta-training by a significant margin: (+25.55%, +46.68%) ) for TO-3B, (+20.74%, +34.66%) for DIRECT, (+34. 12%, +43.74%) for CHANNEL, and (+26.63%. +31.91%) for TO-1 on (IMDB, PAWS). This further bolsters the hypothesis that standard meta-training leads to label overfitting, especially for seen tasks and FLIPPED LEARNING avoids this by conditioning on the label option instead of generating it.
'We collected synonyms from https://ww..cessarruss
We provide the full list of 20 label
options in Appendix H
8

---

MMLU Accuracy
MMLU Accuracy
40
Flipped
35
TO
FLAN-TS
Dalasets
Number
Figure 4: Zero-shot MMLU accuracy when scaling the number of datasets during meta-training. FLAN-T5 (Chung al. 2022 is trained 473 datasets (1,836 tasks). Although FLIPPED trained with 52 datasets (FLIPPED+) uses about 10% of training datasets compared to FLAN-T5, it largely reduces the performance gap. Left: Average accuracy of 3B-sized models on MMLU benchmark. Right: Average accuracy of IB-sizeed models on MMLU benchmark.
ADDITIONAL EXPERIMENTS
Concurrent work of Chung et al. 2022) show that scaling the number of training datasets (up 473 datasets) during meta-training results in state-of-the-art performance on challenging tasks such as the MMLU benchmark (Hendrycks et al. 2020). From the findings of Appendix A.2 we also expect that scaling up the number of datasets during meta-training can improve the performance further. Similar to the approach of Chung et al. (2022), we scale up the number of datasets during meta- training by adding generation tasks that are used to train the T0++ model (Sanh et al., [022]) For generation tasks, we train with the same training objective as classification tasks. For unlikelihood training of generation tasks, we sample an incorrect label option from a different training instance of the same dataset which different from the correct label option. The number of training datasets in total is 52 and we refer to the model trained with FLIPPED LEARNING with these datasets as FLIPPED+. We evaluate FLIPPED+ on the zero-shot setting of the MMLU benchmark and compare the performance with TO models (Sanh al. 2021) and FLAN-T5 (Chung et al.) (022) the same model size, shown in Figure 4
Consistent with the result of Appendix A.2 FLIPPED LEARNING additionally benefits from the scale of the number of datasets: FLIPPED+ outperforms FLIPPED for both 3B and 1BB sized mod- els. Compared to TO models, which do not always benefit from scaling the number of datasets, FLIPPED+ shows significant improvement. Moreover, while only using about 10% of the number of training datasets compared FLAN-T5, FLIPPED+ largely reduces the performance gap between FLAN-T5. We suggest that using less number of training datasets during meta taaa-traiiing but resulting in strong zero-shot performance important because closer to a true zero-shot setting.
CONCLUSION
In this paper, we propose FLIPPED LEARNING, which is meta-training method that flips the in- struction and label space, training the LM to compute the conditional probability of the task in- struction given input instance and label. Our findings show that by conditioning on the label space instead of generating it, FLIPPED LEARNING avoids label overfitting, leading to better zero-shot unseen task generalization capabilities especially for tasks that contain various novel labels. To this end, we suggest the research community consider FLIPPED LEARNING for making efficient LMs that can generalize to challenging unseen tasks.
9

---

ACKNOWLEDGMENTS
We thank Sewon Min, Sungdong Kim, Miyoung Ko, Seungone Kim, Yujin Kim, and Yejin Cho for helpful discussions. This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022- 0-00113, Developing a Sustainable Collaborative Multi-modal Lifelong Learning Framework, 80%; Artificial Intelligence Innovation Hub, 10%; N0.2019-,,,0075, Artificial Intelli- gence Graduate School Program (KAIST), 10%)
No.
REFERENCES
Stephen Bach, Victor Sanh, Zheng Xin Yong. Albert Webson, Colin Raffel, Nihal V. Nayak, Ab- heesht Sharma, Taewoon Kim, Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, An- drea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. PromptSource: An integrated de- velopment environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. As- sociation for Computational Linguistics, 2022.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Jun Shern Chan, Michael Pieler, Jonathan Jao, Jérémy Scheurer, and Ethan Perez. Few-shot adapta- tion works with unpredictable data. arXiv preprint 2xXi::208..01099 2022.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning. arXiv preprint arXiv:21 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint S.Xix:2224.023111 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finttune language mod- els. arXiv preprint arXiv:2200. 14166, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.0547, 2018.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop. 2005.
Marie-Catherine De Mamneffe, Mandy Simons, and Judith Tonhauser. The commitmentankk: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, 2019.
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop Paraphrasing (ZWP2005), 2005.
Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding: A survey. arXiv preprint arXiv: 11857, 2022.
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey P Bigham. Improving zero and few-shot generalization in dialogue through instruction tuning. arXiv preprint arXiv:2205. 0055522675 12673, 2022.
10

---

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 00XY::0000.03300,, 2020.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form compe- tition: Why the highest probability answer isn't 't always right. arXiv preprint arXiv:2104.08315, 2021.
Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205. 10782, 2022.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine read- ing comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguisties, 2019.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qase: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05112 055, 2022.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, D. Kontokostas, Pablo N. Mendes, Sebas- tian Hellmann, M. Morsey, Patrick van Kleef, S. Auer, and C. Bizer. Dbpedia a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 2015.
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Pro- cessing. Association for Computational Linguistics, November 2021.
Hector Levesque, Emestt Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning. 2012.
Mike Lewis and Angela Fan. Generative question answering: Learning to answer the whole ques- tion. In International Conference on Learning Representations, 2018.
Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. Unsupervised cross-task generalization via retrieval augmentation. arXiv preprint vrXiv:2204.079371 2022.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learn- ing, arXiv preprint 2XXx:2205.00633 2022.
Andrew I. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2011.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Seventh ACM Conference on Recommender Systems, RecSys '13, Hong Kong, China, October 12-16, 2013. ACM, 2013.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct clectricity? new dataset for open book question answering, arXiv preprint arXiv:1809.022 02789, 2018.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). Association for Compu- tational Linguisties, 2022a.
11

---

Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies. Association for Computa- tional Linguistics, 2022b.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettemmyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 2022c.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generaliza- tion via natural language crowdsoureing instructions. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers). Association for Com- putational Linguistics, 2022.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruy Batra, Lucy Vander- wende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Association for Computational Linguistics, 2016.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint urXiv:2203.02/555 2022.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL '05). Association for Computational Linguistics, 2005.
Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-context dataset for eval- uating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers). Association for Computational Linguistics, 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 12Xiv:212..11444 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Lin- guistics, 2019.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formaliza- tions of commonsense reasoning. 2011.
Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky, Getting closer to ai complete question answering: A set of prerequisite real tasks. Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint 1 2021.
12

---

Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-JCCILP) Association for Computational Linguisties, 2019.
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. Nearest neighbor zero-shot inference. arXiv preprint arXiv:2205.13792, 2022.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0460 04611, 2022.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. DREAM: A challenge data set and models for dialogue- based reading comprehension. Transactions of the Association for Computational Linguistics, 2019.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. Quarel: dataset and models for answering questions about qualitative relationships. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019a.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. QuaRTz: An open-domain dataset of qualitaaive relationship questions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP) Association for Computational Linguistics, 2019b.
Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. arXiv preprint arXiv:2 1955, 2021.
Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. WIQA: A dataset for "what if..." reasoning over procedural text. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, 2018.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2022.
Albert Webson and Ellie Pavlick. Do prompt- based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247 2021.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language arXiv:2109.01652 01652, 2021.
languaeedlnnguage models are zero-shot learners. arXiv preprint
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language 2022.
models. arXiv preprint arXiv:2206.0766
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text. Association for Computational Linguistics, 2017.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint arviv: 1908 1908.043/9, 2019.
13

---

Seonghyeon Ye, Joel Jang, Doyoung Kim, Yongrae Jo, and Minjoon Seo. Retrieval of soft prompt enhances zero-shot task generalization. arXiv preprint arXiv:2210.03029, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting the Association for Computational Linguistics. Association for Computational Linguistics, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00c0 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc., 2015a.
Xiang Zhang. Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, 2015b.
Yuan Zhang. Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scram- bling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Pa- pers). Association for Computational Linguistics, 2019.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, 2021.
14

---

45
Mean F1
40
Accuracy
56
Flipped
30
Diroct+UL
TO
20
40
20
40
Number Datasets
Number of Datasets
Figure 5: FLIPPED trained on varying numbers of datasets. FLIPPED w/o UL indicates ablation of FLIPPED without unlikelihood training. We also analyze the effect unlikelihood training on DIRECT (DIRECT+UL) UL.. Left: Average Fl score of 7 classification tasks. Right: Average accuracy 7 multi-choice tasks. All models are 3B-sized meta-trained LMs.
A ABLATION STUDIES
In this section, we analyze the effect of unlikelihood training. Also, we vary the number of meta- training datasets of FLIPPED analyze the effect of the number of datasets task generalization. We evaluate on 14 English NLP tasks and report average Fl score on 7 classification tasks and mean accuracy on multi-choices tasks respectively.
A.1 EFFECT OF UNLIKELIHOOD TRAINING
As mentioned in Section 3.2 and shown in Figure we observe that FLIPPED LEARNING ignores the input instance-label correspondence if unlikelihood loss is not added, hurting the performance significantly. We additionally analyze if the strong task generalization of FLIPPED solely coming from unlikelihood training by applying unlikelihood training on DIRECT, which is our strong base- line. As shown the performance of DIRECT+UL in Figure 5 unlikelihood training worsens the task generalization performance especially for classification tasks while giving marginal improve- ment on multi-chott tasks, underperforming FLIPPED for both types of tasks. This shows that the effectiveness of FLIPPED LEARNING is not coming from unlikelihood training itself; both factors of FLIPPED LEARNING, flipping the label and instruction space and unlikelihood training, are needed generalize effectively unseen target tasks.
A.2 NUMBER OF DATASETS
Meta-trained LMs via direct prompting shows improved performance when the number of datasets increases (Sanh et al. 2021 Wang et al. 2022; Wei et al. 2021). We also analyze if this phenomenon holds for FLIPPED LEARNING by varying the number of datasets per task cluster; we increase the tal number of datasets by 4, 8, and 20. As shown in Figure 5 the performance of FLIPPED increases as the number of datasets increases, similar LMs trained through direct prompting. Interestingly, using only 8 datasets to meta-train FLIPPED also shows strong performance, outperforming DIRECT model trained with 20 datasets on multi-choice tasks. Also, this efficient but effective model signif- icantly outperforms TO-3B, while only using 20% of the number of datasets and 5% token updates. This shows that FLIPPED LEARNING can result in generalization to unseen tasks while using only a few number of datasets, making not only effective but also efficient zero-shot learners.
B LIMITATIONS
In this work, we do not explore FLIPPED for performing unseen tasks that do not have label op- tions such as free-form generation. However, we believe FLIPPED can be used for these tasks as well by obtaining the list of label options from a different LM, which we leave for future work. FLIPPED LEARNING also assumes that the task instruction and input instance be separated dur- ing zero-shot inference. However, although instruction- based benchmarks such as Natural Instruc- tions (Mishra et al. 2022 Wang et al. 2022] define the prompted input naive concatenation of
15

---

eextra 0> Using only the above description and what you know about the world, is "<extra id 1> definitely correct? Yes no?
p(
+
input: The girl was found in Drummondville.. <extra 0> Drummondville contains the girl. perl.eextraiicc output: Yes
Figure 6: Illustration of denoising objective of FLIPPED LEARNING. Given, an input instance with sentinel tokens, FLIPPED LEARNING makes the LM generate the task instruction corresponding to the sentinel tokens for correct label option.
task instruction and input instance, this is not guaranteed for prompt libraries such Promptsource (Bach et al. (022). Therefore, FLIPPED LEARNING needs additional techniques to separate the task instruction and the input instances shown in Section 3.1
C ILLUSTRATION OF DENOISING OBJECTIVE
shown in Figure 6 FLIPPED LEARNING uses a denoising objective while meta-training fectively separate the prompted input obtained through Promptsource (Bach et al. (022) into task instruction and the input instance. By replacing task instruction as sentinel tokens, FLIPPED LEARN- ING makes the LM generate the task description that corresponds to the sentinel tokens.
D TRAINING CONFIGURATIONS
For backbone LM of FLIPPED, we use 15.1.1 Raffel 2019) which pre-trained denoising objective while we use T5-LM adapted model (Lester et al. 2021 for DIRECT and CHANNEL which continually trained T5.1.1 model on language modeling objective for 100B additional tokens. We use different backbone LM for FLIPPED LEARNING because the meta-training objective is de- noising objective while DIRECT and CHANNEL language modeling objective. From preliminary experiments, we observe that the language modeling training objective of DIRECT and CHANNEL. on T5.1.1 model leads to poor performance. Also, denoising objective FLIPPED LEARNING on T5-LM adapted model leads to poor performance. Following Sanh al. (22T); Raffel et al. 2019), we limit the number data instances for each dataset to 500,000 to resolve data instance imbalance during meta-training. We train each model for 5K steps, with batch size of 240. We set input and output sequence lengths as 512 and 128 respectively for FLIPPED-3B. For FLIPPED-11BB we input and output sequence lengths 384 and 64 respectively for computational efficiency. For DIRECT and CHANNEL, we set the learning rate as le-4 and for FLIPPED, we set the learning rate as Se-5 because the training objective different (generation vs denoising). We set the weight hyper- parameter of likelihood and unlikelihood loss as À = 3. Note that our total training compute used during meta- trtinttaining around 5% that of the training compute used train the original TO: different from Sanh et al.] (202T) which uses the batch size of 1024, sequence length of 1024, training steps of 12,200, we use batch size 240, half of the sequence length, training steps 5,000 leading to 4.8% token updates compared to TO. For FLIPPED+, we almost keep the training configurations of FLIPPED with only a few variations. Unlike FLIPPED, we limit the number data instances for each dataset to 50,000 resolve data instance imbalance during meta-training. Also, for 3B-sized FLIPPED+, we train for 10K steps during meta-training due the increased number of datasets. For 11B-size FL.PPED++ keep the number of training steps 5K steps due computational costs.
16

---

Classification
Multi-choice
TO-3B
36.79
55.53
TO-3B Calibration
33.59
46.40
FLIPPED
44.95
66.43
Table Effect of calibration on TO-3B meta-trained LM. Results show that the performance worsens if calibration is applied especially for multi-choice tasks.
E TRAINING AND EVALUATION DATASETS
E.1 META-TRAINING DATASETS
We use 4 task clusters for meta-training of DIRECT, CHANNNE and FLIPPED: sentiment clas- sification, paraphrase, topic classification, which 20 datasets total. We use imdb (Maas et al. 201T). amazon-polarity (McAuley & Leskovee 2013), rotten_tomatoes (Pang & Leel yelp review full (Zhang et al. 2015b), app reviews for sentiment, glue/qqp (Wang et al, 2018), paws/labeled final (Zhang al. 20T9), glue/mrpe (Dolan & Brocketi 2005) for para- phrase, ag_news (Zhang et al.] 2015a dbpedia_11 1l1aa_144410 (Lehmann et al. 2015) for topic classification, cos_e/ vos_e/v1.11 (Rajani al. 2019). dream (Sun al. 2019), quail (Rogers et al. 2020)), quartz (Tafjord et al. 2019b). socialliqq (Sap et al. 2019), wiqa (Tandon et al. 2019), cosmos_qp (Huang et al.,
2005
2019 9
qasc (Khot al. 2020 quarel (Tafjord al. (019a), SCIq (Welbl et al. 2017) for multi-
choice QA.
E.2 EVALUATION DATASETS
We evaluate on 14 datasets of BIG-bench benchmark (Srivastava al. 2022): Known Unknown, Logic Grid, StrategyQA, Hindu Knowledge, Movie Dialog, Code Description, Conceptual, Lan- guage ID, Vitamin C. Syllogisms, Misconceptions, Logical Deduction, Winowhy, Novel Concepts, following Sanh et al. (2021). For English NLP tasks, addition to unseen evaluation datasets datasets from Lin al.
from Sanh et al. (2021). ) add unseen
(2022), result-
ing in classification (RTE (Dagan et al. 2005), CB(De Marneffe et al.] 2019), A. ANI. R1,22R33
(Nie al. 2020) WSC (Levesque | (012)). WiC (Pilehvar Gamacho-Colladee 2019) and
multi-choice datasets (COPA (Roemmele et al. 2011. Hellaswag (Zellers et al. 2019)) Storycloze
MMostafazadeh al. 2016), PIQA (Bisk al., 2020), ARC-Challenge (Clark et al. 2018). Open-
bookQA (Mihaylov et al. (2018). We exclude SQuAD2.0 which is included in evaluation setting of
Lin et al. (2022) because it does not have label options.
F EVALUATION SETTING
For the result of PaLM and GPT-3 of Table we use the performance reported in the paper. For the result of GPT-3 on zero-shot setting in Table 2 we use the performance reported in the paper for multi-choice tasks while rerun the experiments using OpenAI API for classification tasks to report F1 scores. We used the prompt named 'GPT-3 style' for every dataset of Promtpsource library. For experiments of Figure we randomly sample 1,000 data instances for seen task label generalization evaluation, for efficiency.
G CALIBRATION RESULTS
Previous work has used calibration methods to match the label distribution of the target task during inference of zero-shot setting (Zhao et al.) 2021 Holtzman et al. 2021). We also analyze if cali- bration effective for meta-tramn LMs by applying contextual calibration on TO-3B. Because we evaluate the zero-shot task generalization performance, empty string for calibration. As shown Table 3 applying calibration hurs the performance of meta-trained LMs.
the probability of the label given
17

---

yes
no
true false
positive negative
right
wrong
correct
incorrect
agree
disagree
good
bad impossible never contradicting not ever not all disagreeable cannot be never way nope
guaranteed
always
affirmative
exactly
undoubtedly
fine good enough definitely unquestionable yep
yea
nah refused unwilling
without doubt
willing
Table 4: List of 20 pairs of labels used to evaluate label generalization on binary classification datasets (RTE, WiC, IMDB, PAWS).
H LABEL PAIR VARIATIONS
We provide the list of variations of label pairs on Sable4and Table 5 Table shows the label pair variation of binary classification datasets (RTE, WiC, IMDB, PAWS) while Table 5 shows the label pair variation of CB, which consists of label options.
18

---

no
maybe
true false neither
positive
negative right wrong perhaps
inconclusive
correct
incorrect
might be
disagree bad impossible never contradicting not ever not at all disagreeable cannot be never no way yep nope yea without doubt willing
agree
could be
good
neutral possible sometimes feasible as may be doubtfully conceivable can be uncertain questionable iffy controversial not for sure
guaranteed
always
affirmative
exactly
undoubtedly
fine
good enough definitely unquestionable
nah refused unwilling
Table 5: List of 20 pairs of labels used to evaluate label generalization for CB, which has 3 label options.
19


Reviews:
[{"review_id": "zeXfj7NQLH", "reviewer": "Reviewer_55Bv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the zero-shot task generalization setting and proposes to learn to generate the instruction conditioned on the input and label. During inference, labels can be predicted by checking which label is the most likely to generate the given instruction. The authors also incorporate unlikelihood training loss to overcome the degeneration issue where the label becomes uncorrelated with the instruction generation. Such a reversed way of generation achieves very good results on multiple unseen tasks, outperforming the T0 baseline. Notably, a 3B T5 model trained with the proposed method even outperforms T0-11B on 14 BIG-bench tasks and much larger GPT3 and PaLM in zero-shot settings. Analysis shows that the proposed method is mainly beneficial for tasks where the label options are unseen during training.", "summary_of_the_review": "I recommend acceptance of this paper since the proposed method is novel and simple and achieves very strong results on impactful benchmarks. ", "strength_and_weaknesses": "### Strengths\n\n1. The proposed method is novel and neat\n2. The empirical results are strong demonstrated by a comprehensive set of experiments. As a new way of instruction tuning, FLIPPED could be potentially very impactful given that it outperforms much larger models with much smaller training costs.\n3. The analysis is interesting and insightful, revealing that T0-like methods have difficulties generalizing to unseen label options while FLIPPED does a pretty good job. \n4. The paper is well-written.\n\n### Weaknesses\n\nAs mentioned in the Limitation section, FLIPPED is only naturally applicable to classification tasks where the label space is limited, while DIRECT methods appear to be more flexible from this aspect. \n\n\n### Suggestions\nWhen presenting the main results, I think the authors should clarify FLIPPED outperforms DIRECT on X out of X datasets. The gains are actually not very consistent in the Tables and the mean accuracy does not tell the full story.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "l6hYVQHjAw9", "reviewer": "Reviewer_CFao", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new meta-training approach for language models. For an instance of task instruction, input and label, their method, called Flipped Learning, increases the likelihood of the task instruction given the input and correct label, while decreasing the likelihood of the task instruction given the input and an incorrect label. Training a T5-3B model with this method on 20 datasets leads to performance gains on various datasets compared to larger (size and compute) models.", "summary_of_the_review": "Overall, the approach of conditioning on the label space instead of generating it is interesting, but limited. The motivation for this method could have been discussed more. Some parts about the intuition of why the method increases label generalization are not clear.\n", "strength_and_weaknesses": "### Strength\nComputing the conditional probability of instruction given the concatenation of the input and label has led to strong zero-shot generalization performance. It is mentioned that this is due to the model’s better generalization to labels that it has not seen before (labels that have different surface form, but similar meaning). \n\n### Weaknesses\nAs the authors mention these as well, their proposed training method is limited to tasks which have label options (vs. free-form generation). This would limit the approach in both training and inference/evaluation. It is also not always the case that one could have a task instruction that is easily separable from the input instance. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KFm3nzXn0m", "reviewer": "Reviewer_dMHf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an approach to improve zero-shot prediction results by making a language model predict the instruction (or measure likelihood) given the label and the input, rather than having the model predict the label instead. The paper demonstrates the effectiveness on multiple datasets within the BIG-Bench benchmark. ", "summary_of_the_review": "Overall, this paper proposes a simple approach consisting of the flipped inference along with the unlikelihood meta training that seems to work well. The experiments are quite extensive. The performance gain is not too consistent so while this method might work in some domain, it might be challenging to be adopted as a universal method.", "strength_and_weaknesses": "Pros:\n\nThe simplicity of this approach is a plus. The avoidance of label overfitting due to FLIP is also a nice advantage with additional benefits on unseen labels.\n\n\nQuestions:\n- Any additional thoughts on why unlikelihood training is crucial for the success of FLIPPED?\n\nCons:\n\n- It is not surprising that the meta training done will make 3B model outperform larger models. The statement that the model performs better compared to 175 or 540B models do not mean much and such statement should be avoided or with caveats mentioned explicitly. To compare with the larger models, it would be good to see the few-shot results without meta training. Or is it the case that FLIPPED does not work at all without meta training? \n- In approach makes sense, however, I still question the universality of this approach. The free-form generation of label being one. \n- Another strong baseline for tasks with unseen labels is to add label explanation in the instructions. \n- Performance gain is observed for some datasets but not overwhelmingly consistent.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster