A Universal Discriminator for Zero-Shot Generalization
Year: 2023

Abstract:
Generative modeling has been the dominant approach for large-scale pretraining and zeroshot generalization. In this work, we challenge this convention by showing that discriminative approaches perform substantially better than generative ones on a large number of NLP tasks. Technically, we train a single discriminator to predict whether a text sample comes from the true data distribution, similar to GANs. Since many NLP tasks can be formulated as selecting from a few options, we use this discriminator to predict the concatenation of input and which option has the highest probability of coming from the true data distribution. This simple formulation achieves state-of-theart zero-shot results on the T0 benchmark, outperforming T0 by 16.0%, 7.8%, and 11.5% respectively on different scales. In the finetuning setting, our approach also achieves new stateof-the-art results on a wide range of NLP tasks, with only 1/4 parameters of previous methods. Meanwhile, our approach requires minimal prompting efforts, which largely improves robustness and is essential for real-world applications. Furthermore, we also jointly train a generalized UD in combination with generative tasks, which maintains its advantage on discriminative tasks and simultaneously works on generative tasks.

heading: 1 Introduction
text: Generative modeling has been the dominant approach for large-scale pretraining and zeroshot generalization (Brown et al., 2020; Artetxe et al., 2021; Rae et al., 2021). Combined with prompts (Brown et al., 2020), most of the natural language processing (NLP) tasks can be formulated into the fill-in-the-blank format and perform generative language modeling. Based on the unified generative formulation, pretrained models such as GPT-3 (Brown et al., 2020), BERT (Devlin et al.,
*Corresponding authors. †Our code is available at https://github.com/Rafa-zy/UD.
2019; Schick and Schütze, 2020), T5 (Raffel et al., 2019), can perform zero-shot inference on new tasks.
More recent work (Sanh et al., 2021) proposed to further pretrain a generative T5 (Raffel et al., 2019) with multitask prompted datasets and has substantially enhanced the performance of zeroshot generalization. In contrast, methods based on discriminative modeling (Devlin et al., 2019) have not been able to achieve state-of-the-art performance on zero-shot learning. The adoption of discriminative approaches for zero-shot learning has been limited in the literature.
In this work, we challenge the convention of zero-shot learning and propose to study and improve discriminative approaches. This is motivated by the fact that many NLP tasks can be framed as selecting from a few options; e.g., telling whether sentence A entails sentence B, or predicting which answer is correct for a given question. We call these tasks discriminative tasks. As we will discuss in later sections, a significant portion of NLP tasks is in fact discriminative tasks. We hypothesize that discriminative approaches perform better for discriminative tasks.
To verify the hypothesis, we propose the universal discriminator (UD), which substantially improves zero-shot generalization over the previous generative state-of-the-art (SOTA) (Sanh et al.,
10559
2021), as Figure 1 shows. The main idea is to train a single discriminator to predict whether a text sample comes from the true data distribution of natural language, similar to GANs (Goodfellow et al., 2014). Given a set of training tasks with labeled data, we construct a dataset with positive and negative examples, where positive ones are indistribution natural language samples and negative ones are out-of-distribution. There are two major types of discriminative tasks. The first type is tasks with multiple options, such as multi-choice question answering and news classification. We fill the options into the sentences and the ones with correct options are considered positive samples. The second type is tasks with yes/no options, which can be formulated as a binary discrimination problem itself. For example, natural language inference aims to predict whether a premise entails a hypothesis. In this case, we use a prompt to concatenate the premise A and the hypothesis B into a sentence “Premise: A. Hypothesis: B.” If entailment holds, this sample is treated as positive in-distribution samples and otherwise negative out-of-distribution ones.
For the performance of zero-shot generalization, our approach achieves new state-of-the-art on the T0 benchmark, outperforming T0 by 16.0%, 7.8%, and 11.5% respectively on different scales. UD also achieves state-of-the-art performance on a wide range of supervised NLP tasks, using only 1/4 parameters of previous methods. Compared with the previous generative prompt-based methods, our universal discriminator requires minimal prompting, which is simple, robust, and applicable in real-world scenarios.
In addition, we also generalize UD to a larger scope of tasks, such that UD can perform discriminative and generative tasks at the same time. Specifically, we extend UD to the encoder-decoder architecture for training on generative tasks, and restrict the model’s prediction on "yes"/"no" tokens for jointly training discriminative tasks. Results prove that generalized UD maintains UD’s advantages on discriminative tasks and achieves comparable results on generative tasks (See § 3.4).

heading: 2 Related Work

heading: 2.1 Zero-Shot Generalization Using PLMs
text: Pretrained language models (PLM) can transfer knowledge from training data to downstream tasks. Prompting methods further narrow the gap between
training data and downstream tasks. Schick and Schütze (2020) reformulate NLP tasks into cloze filling using prompts so that PLMs can conduct zero-shot inference by generating tokens given prompted inputs. Meng et al. (2022) use PLMs to generate class-conditioned texts with the guidance of prompts without seeing any task-specific data. Most recently, researchers have introduced natural language prompts to unify various kinds of tasks and propose a multi-task prompted training framework to achieve great zero-shot performance even faced with unseen downstream tasks (Wei et al. (2021); Sanh et al. (2021); Chung et al. (2022)). However, zero-shot learning has been dominated by generative approaches.

heading: 2.2 Prompt-based and Prompt-free Methods in NLP
text: Prompting is the method of reformatting NLP tasks using natural language templates to adapt to downstream tasks (Raffel et al., 2019; Schick and Schütze, 2020). To reduce the instability and labor costs brought by prompting, researchers have tried various approaches (Liu et al. (2021a); He et al. (2021a)) to learn continuous prompts.
Recently, prompt-free methods are also being explored. Mahabadi et al. (2022) adopts task-specific adapters to learn task descriptions implicitly for few-shot learning with PLMs. It has also been indicated that using null prompts without task-specific templates can achieve decent performance compared with manually-designed prompts on various tasks (Logan IV et al. (2021)).
Our work further shows that those widely used lengthy instructive prompts are not necessary for zero-shot learning. Actually, minimal prompting performs better with our discriminative formulation in the multi-task zero-shot learning setting.

heading: 2.3 Discriminative Models in NLP
text: PLMs trained with masked language modeling (MLM) (Devlin et al., 2019; Liu et al., 2019) can be finetuned in a discriminative manner for downstream tasks. ELECTRA (Clark et al., 2020) trains a discriminator to detect whether a token has been replaced. WKLM (Xiong et al., 2019) employs an entity-centric approach for pretraining and predicts whether an entity has been replaced. However, finetuning for these methods is usually based on one separate CLS head per task, which is not suitable for zero-shot generalization.
Recently, prompting has been combined with token-level discriminators based on ELECTRA for few-shot learning (Yao et al., 2022; Xia et al., 2022). While these are also discriminative approaches, there are a few key differences from our approach. The biggest difference between them and us is that: we unify all discriminative tasks into one single task with minimal prompting, showing extremely good zero-shot generalization. Moreover, these methods are specific to ELECTRA-like pretraining, while our approach accepts arbitrary pretrained encoders. In our experiments, we will also make a direct comparison with these approaches to demonstrate our effectiveness.

heading: 3 Approach
text: Previous works (Sanh et al., 2021; Wei et al., 2021) have shown that prompted multi-task training can greatly improve zero-shot performance on unseen tasks. One intuitive reason behind the validity of this improvement is that all the NLP tasks share a common ability that allows LMs to solve unseen tasks based on the data from other training tasks. To test this idea and even enhance zero-shot generalization, a direct way is explicitly defining what this "common ability" is. Here, we define this "common ability" by designing a new general task of “discriminating whether a text sample comes from the true data distribution of natural language”.
We will first formulate the learning problem (§ 3.1), and then define the concept discriminative tasks (§ 3.2), followed by describing how we transform discriminative tasks into our shared formulation. In § 3.3 and § 3.4, we will study our UD, respectively on discriminative tasks and on a generalized scope of both discriminative and generative tasks.

heading: 3.1 Multi-Task Training for Zero-Shot Generalization
text: Now we describe the learning problem we aim to solve in this work. We adopt the same setting as in Sanh et al. (2021). The input to our problem is a set of training tasks with labeled data, and the goal is to train a model that generalizes to unseen test tasks. The training and test tasks are constrained to have distinct task types for the evaluation of cross-tasktype generalization. A pre-trained model is jointly trained on the set of training tasks and directly evaluated on the set of test tasks in a zero-shot manner.

heading: 3.2 Discriminative Tasks
text: We use the term “discriminative tasks” to refer to tasks that can be framed as selecting from a few options.
More concretely, there are two types of discriminative tasks. The first type is tasks with multiple options, such as multi-choice question answering and news classification. The problem can be framed as selecting the right option from multiple ones, where the options are either customized for each sample (e.g., multi-choice question answering) or shared within the task (e.g., news classification). The second type is tasks with yes/no options, such as paraphrase identification and natural language inference. Given a sample of these tasks, a model is asked to predict a yes/no (or true/false) answer.
It is important to notice that discriminative tasks constitute a significantly large portion of modern NLP research tasks. For example, all of the test tasks of the T0 benchmark (Sanh et al., 2021), SuperGLUE (Wang et al., 2019a), GLUE (Wang et al., 2019b), and 85+% tasks in BBH benchmark (Suzgun et al., 2022) are discriminative tasks.
Also note that our definition of discriminative tasks has a larger scope compared to the conventional notion of “classification” which usually refers to tasks with a non-customized, fixed set of labels. In contrast, discriminative tasks might have sample-customized options, e.g., multi-choice question answering and coreference resolution.

heading: 3.3 A Universal Discriminator
text: Given a text sample x, let P (true|x) be the probability that x is sampled from the true data distribution of natural language. We train a universal discriminator (UD), denoted as D(x), to estimate the probability P (true|x) for each text sample x. From another perspective of contrastive learning (Oord et al., 2018), this problem can also be viewed as learning a partial order of the probability distribution. Specifically, for two text samples x1 and x2, if P (true|x1) > P (true|x2), the UD is expected to predict D(x1) > D(x2). This contrastive view is essential for tasks with multiple options, i.e., learning to select from a few options based on the partial order given by UD.
Figure 2 compares the multi-task prompted formulation of T0 and the formulation of our UD. In the following, we will show how we use this formulation of UD to unify and solve discriminative tasks.

heading: 3.3.1 Unifying Discriminative Tasks
text: We assume that for any task, the concatenation of input and the correct option follows the true data distribution of natural languages, while the concatenation of input and the other wrong options deviates much from the true data distribution.
Given this assumption, we claim that almost all discriminative tasks are equivalent to our defined task (i.e., estimating P (true|x)) above. Here, “equivalent” has bi-directional meanings: on one hand, there exists a reduction* from UD’s task (say, task U) to any discriminative task (say, task A): given a piece of labeled training data for task A, we can generate several pieces of labeled training data for task U.
On the other hand, there exists another reduction from any discriminative task A to UD’s task U: given a piece of testing data for task A, we can generate several pieces of testing data for task U such that by first predicting D(·) on them and then using a mapping from task U’s outputs to task A’s outputs, we can generate the answer for task A.
Based on the definition of discriminative tasks in § 3.2, there are two main categories, multi-choice tasks and yes/no tasks. We will discuss each category in detail as follows (also see Table 6 in appendix for specifics).
Multi-Choice Tasks For multi-choice tasks, we concatenate the text input xin with each choice {ci}Nci=1 to form samples. For example, for multichoice question answering, we concatenate the
*In complexity theory, a reduction is an algorithm transforming one problem A into another problem B such that a solution for problem B could also be used to solve problem A.
given paragraph and question with each answer candidate. See Table 6 for more task formulations. During training, the concatenated samples with the correct choice are given label 1 (true) for UD and the other incorrect ones are given label 0 (false). During testing, similarly, we concatenate the text input xin with each choice {ci}Nci=1 to form several samples {(xin, ci)}Nci=1 and ask UD for their D(·) scores. We then select the sample with the maximal D(·) score and output its corresponding choice.
Tasks with Yes/No Choices For yes/no tasks, we directly treat the text input xin as a sample and assign its 0/1 label based on its yes/no label. During training, we use xin with its assigned 0/1 label as UD’s training data. During testing, we first get the output of UD on xin, D(xin), and then output answer yes/no based on whether D(xin) > 0.5†.
Empirical experiments suggest that unifying tasks with Yes/No choices in such a new way can produce better zero-shot performance than using the same method for Multi-Choice Tasks. We provide two justifications here: First, the Yes/No answer tokens here don’t contain specific information and thus the model cannot benefit from concatenation. Second, the two tokens Yes/No are asymmetric in the training dataset which may result in the model uniformly assigning higher scores for one of them no matter what the task input is.
Minimal Prompting A key principle we follow for task formulation is minimal prompting. From Table 6, one can see that our prompts are min-
†We note that more delicate threshold search might be possible, but we find it performs well using a constant 0.5.
imal in the sense that they are mostly just concatenations of different elements from the raw input, discarding most of the previously instructive prompting words. This is very different from T0 (Sanh et al., 2021) and other generative approaches (Brown et al., 2020; Schick and Schütze, 2020) that add lengthy task descriptions with different wordings into the prompts.
We argue that there are two major benefits of minimal prompting. First, previous work (Liu et al., 2021b) has shown that zero-shot and few-shot performances are very sensitive to the prompts used for inference. Minimal prompting is more robust and requires less prompt engineering efforts at test time. This is especially important for true zero-shot real-world applications as there is no data available for choosing the right prompt. Second, as we will show in our experiments, UD performs much better with minimal prompts than lengthy descriptive prompts, while generative approaches do not work well with minimal prompts. This is also consistent with our motivation that all the NLP tasks share a common ability: “discriminating whether a text sample comes from the true data distribution” and UD is attempting to learn “what kind of concatenation between input and option makes it look like the true language?”, which does not rely much on the descriptions for each task. On the other hand, T0 attempts to generate the answer directly basing on all the information it gets, so prompts provide an extra source of information and are helpful. See § 4.4.1 for our ablation study on minimal prompts.
Note that it is also important to use minimal prompts to resolve ambiguity in some cases. For example, consider the natural language inference (NLI) task that predicts whether a premise A entails a hypothesis B. Simply concatenating A and B is ambiguous, because the model cannot tell which is the premise. The model also is not aware that this is an NLI task. To resolve this kind of ambiguity, we use a minimal prompt “Premise: A. Hypothesis: B.” instead, as shown in Table 6.

heading: 3.3.2 Architecture
text: UD can use any pre-trained encoder model as the backbone. In this work, we experiment with the T5 encoder and DeBERTa (He et al., 2021b). Since T5 is an encoder-decoder model, we only use the encoder part. For the T5 backbone, we perform mean pooling over the last-layer encoder features, followed by a dropout layer and a linear layer to predict a scalar logit. For the DeBERTa backbone,
we use the last-layer feature of the first token, followed by a two-layer perceptron with dropout to also output a scalar logit. We train UD with the binary cross entropy loss.

heading: 3.4 A Generalized Universal Discriminator
text: To further study how the discriminative approaches work in combination with generative tasks, we also propose to experiment with a generalized version of UD (denoted as generalized UD).
Different from the previous UD that only uses an encoder as the backbone model, the generalized UD employs an encoder-decoder architecture. In the following, we experiment with the T5 model. Generalized UD takes both discriminative and generative tasks into consideration, and is jointly trained over both types of tasks at the same time.
For discriminative tasks, they are reformulated into binary classification tasks through minimal prompting, as is described in § 3.3.1. Specifically, it takes the minimal prompted texts into the encoder and uses the decoder to predict over {“Yes”, “No”}. In such cases, generalized UD is optimized with the binary cross-entropy loss. For generative tasks, they take the form of “input-and-target” pairs. Generalized UD is fed with the textual inputs, and generates the targets through decoding. For generative tasks, generalized UD is trained to optimize the cross-entropy loss.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: We performed extensive experiments to validate the performance of the zero-shot generalization of our UD. We follow the same zero-shot setting as T0 (Sanh et al., 2021) by training on multi-task datasets and evaluating a held-out set of tasks that are never seen during training.
Datasets The original T0 training set consists of 38 tasks of 8 different types. There are in total 21/38 discriminative training tasks, with which we train the UD. The evaluation set covers four types of tasks, including natural language inference (RTE (Candela et al., 2006), CB (De Marneffe et al., 2019), ANLI/R1-R3 (Nie et al., 2020)), coreference resolution (WSC (Levesque et al., 2012), Winogrande (Sakaguchi et al., 2020)), sentence completion (COPA (Roemmele et al., 2011), StoryCloze (Mostafazadeh et al., 2017),
Hellaswag (Zellers et al., 2019)), and word sense disambiguation (WiC (Pilehvar and CamachoCollados, 2018)). Following T0, we use accuracy on the validation split as the evaluation metric. For prompt-based baselines, we report the average accuracy over multiple prompts for each test task. Besides, we also evaluate zero-shot performance on 13 BigBench (Srivastava et al., 2022) tasks, which are also adopted by T0 (Sanh et al., 2021), and 22 BBH tasks (Suzgun et al., 2022), which are adopted by Flan-T5 (Chung et al., 2022).
Baselines We primarily compare our method with T0 (Sanh et al., 2021), which is a generative approach. Another baseline is prompting ELECTRA (Xia et al., 2022) which is a recent work on
discriminative modeling. Since it was proposed in a different setting (i.e., a few-shot setting or direct zero-shot inference without any finetuning), we reproduced their method under our multitask zero-shot setting for comparison.
For a fair comparison, we follow T0 to use the T5-V1.1-LM-Adapted (Raffel et al., 2019) as the backbone model, and we experimented with three different scales, respectively 800M, 3B, and 11B. For UD, it only makes use of the encoder of T5v1.1 and additionally replaces the output layer with a classification head.
In addition, we provide reported zero-shot results of several large language models (with hundreds of billions of parameters) for reference, including GPT-3 (Brown et al., 2020), GLaM (Du
et al., 2022), PaLM (Chowdhery et al., 2022), and FLAN (Wei et al., 2021). We also reproduce zeroshot results of a recent work Flan-T5 (Chung et al., 2022) by evaluating their released checkpoints on BBH tasks‡. Note that Flan-T5’s training data sets are much broader than ours, so results for Flan-T5 here are only for reference but not a fair comparison.
Training During training, we truncate the input sequence to 256 tokens and use a batch size of 256. For optimization, we use the Adam optimizer with a fixed learning rate of 1e-5 and a dropout rate of 0.1. Each experiment is trained with 10, 8, and 5 epochs respectively for 800M, 3B, and 11B models.

heading: 4.2 Main Results on Zero-Shot Tasks
text: UD Zero-Shot Results The main results are presented in Table 1. We compare methods of similar scales. Results in Table 1(a) show that our UD substantially outperforms the T0 baseline on average by a large margin of around 9, 5, and 7 points respectively at Large, XL, and XXL scales. Comparing the results of UD-T5-Large, UD-DeBERTaV3, and prompting ELECTRA, both variants of UD also substantially outperform prompting ELECTRA by more than 6 points. On BIG-Bench datasets, results in Table 1(b) show that our UD
‡T0 test sets are included in Flan-T5’s training data sets, so we can’t test its zero-shot performance on those data sets.
outperforms the T0 baseline by a margin of around 4-8 points. Besides T0 benchmark, we also test UD on BBH datasets, which are very different from T0 training sets, results in Table 1(c) show that our UD constantly outperforms T0 and Flan-T5 by a margin of around 2-5 points, even though our UD is only trained on a small fraction of Flan-T5’s training sets. Overall, these results demonstrate the advantages of UD at every scale, and a broad range of tasks compared with baselines.
Another interesting finding is that the advantages of UD significantly increase along with scaling. When scaling from Large-scale to XL-scale (i.e., around 3.75x of the parameters), the average performance improves by around 2 points. However, when scaling from XL-scale to XXL-scale (i.e., 3.6x of the parameters), the improvements of average zero-shot performance enlarge to 8 points. Based on the observation, we hypothesize that UD can achieve even better performance of zero-shot generalization if further scaling to an even larger models, which we leave to future work.
To further boost the zero-shot performance, we also train a new variant of UD at 11B scale by scaling to more training tasks, including the discriminative English tasks used in Wang et al. (2022), and the discriminative English tasks used in Tay et al. (2022). The new model is denoted as UD+. UD+ achieves the highest average accuracy among all the zero-shot evaluation tests.
Generalized UD Zero-Shot Results The zeroshot results of generalized UD on 11 T0 discriminative test tasks and on 13 Big-Bench tasks are respectively reported in Table 7(a) and Table 7(b) We also select the top 15 uncommon generative tasks from BigBench basing on ascending order of data size, results are in Table 7(c). We assume that tasks with smaller data sizes are less common and more likely to be unrelated to our training data and more suitable for zero-shot tests.
Analyses are as follows. First, comparing the results of generalized UD and T0, generalized UD still holds significant improvements on discriminative tasks. Second, comparing generalized UD with our previous UD (in Table 1), we observe there is a slight decrease in average performance, proving that adding generative tasks into training could have impacted a little bit, in trade for capability for handling generative tasks. Third, on 15 generative tasks, both generalized UD and T0 show comparable results.

heading: 4.3 SOTA Results on Finetuned Tasks
text: To explore how UD performs on fully-supervised tasks, we finetuned UD for a wide range of downstream tasks and reported their results in Table 2. For each finetuning experiment, the maximum training epoch is set to be 10. We search a hyperparameter space with learning rate in {2e-5, 1e-5, 5e-6}, batch size in {32, 64, 128}. We select the best checkpoint using a validation set with early stopping.
From results in Table 2, we find that UD can achieve remarkable performance on most of the downstream tasks. We achieve state-of-the-art performance on 12 out of the 17 tasks we evaluated. The results also show that more challenging tasks (tasks that require more knowledge) will benefit more from the multi-task training period, especially some QA tasks.

heading: 4.4 Ablation Study
text: We have also conducted ablation studies to further explore how several factors affect the performance of zero-shot generalization. Please see appendix for further ablation studies on UD with different base models (§ C.1)

heading: 4.4.1 Instructive Prompts vs Minimal Prompts
text: UD employs minimal prompts that use simple concatenation, while previous approaches rely on lengthy instructive prompts to provide more detailed instructions (Sanh et al., 2021; Wei et al., 2021; Brown et al., 2020). Statistically, we count the average number of prompt words (excluding raw input) for both minimal and instructive prompts, and statistics are respectively 0.4 versus > 10. We compare these two types of prompts in the following experiment. We adopt the instructive prompts from T0 and apply them on UD without
changing the discriminator formulation. To construct minimal prompts for T0, we remove all the instructive words similar to UD.
Results are shown in Table 3. We observe that minimal prompts yield better performance for UD than instructive prompts. In contrast, for T0, instructive prompts perform much better than minimal prompts. These results are consistent with our motivation that UD tends to unify the tasks better with a shared discrimination formulation. As a result, task-specific instructions are not necessary and might hurt generalization performance. Generative approaches, on the other hand, rely on instructive prompts to better distinguish different tasks and generate specific answers directly.

heading: 4.5 How Well UD Generalizes to a Broader Domain?
text: Our discrimination problem formulation is in fact more general than solving supervised labeled tasks and can be applied to a broader domain of natural language. We conduct the following experiment to see how UD generalizes.
To test whether a model discriminates against the true data distribution, a straightforward way of verification is to compare the probability of real data with that of some generated, fake data. This form of verification is not specific to any downstream task and can be viewed as generalizing to a broader domain. Formally, given a text sample x, let D(x) be the output of UD, which estimates the probability that x is sampled from the true data distribution, i.e., P (true|x). Given a true data sample x and a generated data sample x′, we expect a well-trained UD to predict D(x) > D(x′).
Specifically, we randomly select 2,600 real data samples x from the validation set of the T0 training data and generate the data x′ in two different ways: model-based generation and manual generation.
For a model-based generation, we utilize the T0-
Large model with a paraphrase prefix “Paraphrase the sentence:” to generate data x′. It is expected that the generated samples x′ are similar to true samples x to some extent but demonstrate some flaws that are unique to generated data. For a manual generation, we manually create some conflict or contradiction in the real sample x. Specifically, we manually attach wrong answers to the original data and obtain x′ , which is similar to what we have done in constructing negative samples in our main framework.
We then use our universal discriminator based on T5-Encoder Large to compute the probability D(x) and D(x′) for both real and generated data. As displayed in Table 5, we find that the universal discriminator assigns a higher score for x than x′ 80% of the time for manually-generated data. When tested with model-generated data, UD assigns a high probability for real data in 74% of the cases. This is probably because manually generated data are more paradoxical and logically incoherent and thus are easier for UD to discriminate. Overall, these results demonstrate that the discrimination ability of UD is not limited to the downstream tasks on which it was trained, but is also generalizable to a broader domain of text data. This indicates a possibility of extending UD to other scenarios such as model pretraining and generation tasks.

heading: 5 Conclusions
text: Universal Discriminator is a discriminating model for predicting whether a sample comes from the true data distribution, which is a new formulation for all discriminative NLP tasks. Experiments show that UD sets the new state-of-the-art for zero-shot generalization on many benchmarks. UD is highperforming with minimal prompting, and thus is more robust and applicable in practice. A generalized UD can also solve generative tasks at the same time which keeps UD’s advantage on discriminative tasks and has comparable performance on generative tasks.

heading: 6 Limitation
text: Even though our generalized UD can get comparable performance on some generative tasks, generalized UD may not handle certain complex generation tasks very well (e.g., summarization) We leave expanding UD to solve a broader range of generative tasks and achieve greater performance advantage as our future work.

heading: A Examples of Minimal Prompt
text: Here we provide Table 6 for some examples of how to construct minimal prompted data according to § 3.3.1.

heading: B Full Experiment Results
text: B.1 Evaluation on Big-Bench Here we report the full results for 13 tasks in the Big-Bench Srivastava et al. (2022), which is also utilized in original T0 paper (Sanh et al., 2021). All the tasks from BIG-Bench are ensured unseen in our training set for the zero-shot setting. The results are displayed in Table 7, where UD outperforms T0 by 4-8 points on different scales.
B.2 Evaluation on BBH Here we report the full results for 22 discriminative tasks from BBH (Suzgun et al., 2022). For reference, we reproduce Flan-T5(Chung et al., 2022)’s zero-shot performance on BBH tasks by evaluating their
public checkpoints. All the tasks from BBH are ensured unseen in our training set for the zero-shot setting. The results are displayed in Table 8, where UD constantly performs better than T0 and Flan-T5 on all the scales even though Flan-T5 is trained on a much broader scope of tasks than UD is.

heading: C More Ablation Studies
text: C.1 Ablation on Base Models
We also study the effects of using different backbone pretrained models. We experiment with three backbone models of different types, respectively the encoder part of an encoder-decoder model, an encoder model, and a decoder model. Specifically, we use the T5 encoder, DeBERTa (He et al., 2021b), and GPT (Radford et al., 2018) respectively for these three types. It is noteworthy that though similar in architecture for both T5 encoder and DeBERTa, they are pretrained with different self-supervised language modeling tasks, which in fact leads to huge differences in zero-shot generalization, as we will show in Table 9.
Results of different backbone models are presented in Table 9. Among all three types of backbone models, the encoder backbone models appear to be the most suitable type of backbone, where both encoder models of two scales respectively achieve the best and the second best results, outperforming all the others by more than 5 points.
Using the same number of parameters (i.e., 1.5B), both DeBERTa-V2 and T5-Encoder significantly outperform GPT-XL, which demonstrates that a bidirectional architecture works better than the unidirectional architecture for the discriminator formulation. In addition, DeBERTa-V2 outperforms T5-Encoder by 7 points, implying that not only model architecture but also the self-supervised pretraining task determines
the ability of UD discrimination. Models pretrained with masked language modeling tasks are more suitable for UD.
The impacts of the architecture and pretraining tasks of backbone models are even larger than the influence of scale, as we also observe that an encoder model with 300M parameters (i.e., DeBERTaV3) achieves much better performance than the T5 encoder and GPT-XL with 1.5B parameters.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: , summarization) We leave expanding UD to solve a broader range of generative tasks and achieve greater performance advantage as our future work