Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes SuperGen, which targeting at automatic dataset generation for zero-shot learning of NLU tasks. The framework relies on manually designed prompts to "extract" task-specific knowledge from a pre-trained PLM, thus constructing a synthetic dataset. In the whole process, no human annotations are required. The experiments show that the proposed method can even outperform few-shot learning methods on some NLU tasks. 
strengths_and_weaknesses: Strength:

The idea of modifying an auto-regressive PLM decision boundary to a discriminative one by generating synthetic data is interesting.

The empirical evaluation shows promising results of SuperGen beating the performance of PLMs used for the few-shot setting.

Weakness:

Although it's encouraging to see that the zero-shot SuperGen outperforms the few-shot setting, I'm not fully convinced by the results. As been widely discussed, the data selection process under a few-shot setting can have a huge impact on evaluation results. Not to mention the impact brought by prompts, decoding strategies, etc. I understand that some of these factors are difficult to control, so I suggest the following baselines for evaluation:

PLMs fine-tuned on the original train set for each task. While I understand that these results will outperform the results of SuperGen, I think it is still relevant and important to know what is the gap in performance between fine-tuning the PLMs and the zero-shot evaluations as shown in the Table.
A knowledge distillation baseline: using the PLM to generate the output label when provided with an unlabeled text input (from some corpus), and then using these pseudo labels to train a discriminative model. This would help shed light on what is the benefit of creating the synthetic data rather than directly transferring knowledge from the PLM to downstream models.

It would have been interesting to see if the generated synthetic data in a zero-shot manner could add to the labeled supervised data for the tasks via data augmentation, and help improve the performance of the fully supervised setting.
questions: "we include the average and standard deviation over 5 different random seeds."
Are these 5 runs performed on 5 different few-shot train/dev sets or on the same pre-sampled set over 5 different runs? 
limitations: In spite of excellent results on easy tasks such as simple text classification, the overall performance is still poor on tough tasks like CoLA. This limitation has to be discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes generating training data for certain NLU tasks using a pre-trained generative model and using this data to fine-tune a pre-trained MLM for the target task. The generated data is filtered based on likelihood and the classifier is regularized using label smoothing for robustness to noise. They generate on the order of a few thousand examples using prompts and the label space for 7 GLUE tasks and compare against zero-shot and few-shot baselines.
strengths_and_weaknesses: Strengths:  
The proposed approach to generate data using prompts is quite interesting and is generating data of good enough quality to achieve non-trivially good performance on the GLUE tasks. This has important implications for showing that using LLMs, zero- and few-shot settings don’t need to be as such by augmenting them with synthetic data, at least for the kinds of tasks explored in this paper. 

Weaknesses:  
Calling this entire setup “zero-shot” is a bit of a stretch. The generator can be seen as zero-shot for generating task-specific examples, but the fine-tuned classifier not so much. For this reason, the truly zero-shot baseline seems weak and it is unsurprising that the classifier fine-tuned on synthetic data outperforms it. Perhaps an interesting comparison that was left out was fine-tuning on few-shot data vs. fine-tuning on a combination of few-shot and the synthetic data to quantify how much value is added by the synthetic data. 

There’s nothing new and surprising about using synthetic data, it’s more the approach to generating the synthetic data that is the interesting contribution here. The story (and title) could use some revision to clarify this. 

questions: Have you considered generating synthetic data using the few-shot examples? Do you think that would make it possible to generate higher quality data with a greater value-add?
E.g. https://arxiv.org/pdf/2102.01335.pdf 
limitations: I think the limitations section doesn't really discuss potential societal impact. Perhaps the synthetic data generated by the model might contain unwanted biases that could be harmful.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: They study zero-shot learning of PLMs on NLU tasks without any task-specific or cross-task data.

They propose SuperGen approach: training data are created via unidirectional PLM. A bidirectional PLM is then finetuned on the generated texts to perform tasks.

Across seven classification tasks of the GLUE benchmark, SuperGen significantly outperforms the prompt-based zero-shot method. Their method for selecting quality training data based on their generated probability and using label smoothing and temporal ensembling demonstrates the strong performance of SuperGen.
strengths_and_weaknesses: Strengths

- Supervision automatically created by the generator eliminates the need for task-specific annotations.
- Rewarding and penalizing repetitions for sequence pair generation reduces the chance that the generated texts get stuck in repetition loops.
- Reasonable method for selecting quality training data to take the average log probability.
- Label smoothing forces the classifier to have less probability on training data. That improves robustness to label noise and prevent overfitting to the training set.

Weaknesses
- Since prompt would be a key factor for the performance, it would be nice to have prompt ablation studies for each task, not just three tasks(MNLI, RTE, MRPC).
- Even though SuperGen significantly outperforms zero-shot prompting and achieves a better result than all the few-shot methods, the performance cannot match to their SOTA results.
- The generated texts are not judged by humans.
questions: - In Table2, the performance of SuperGen without label smoothing for CoLA drops a lot. Is there a specific reason for that?
- Generator selection: Why did you not use large size PLMs (e.g., GPT3) from the beginning for the high-quality generation, if the model size matters for high-quality text generation like you mentioned in page 8, line 280?
- Are there other regularization methods other than label smoothing and temporal ensembling that you might have considered?
limitations: N/A
review_title: null

Meta-review: metareview: The paper introduces three theoretical analyses explaining the effectiveness of the transformer in long sequences. The reviewers raise three aspects that the paper can be further improved. First is the writing of the paper, it may be too dense and hard to follow. Second, the authors may want to elaborate a bit more about what deeper insights the proofs can bring to the community, except as a proof for the transformer’s effectiveness in long sequences. Third, the authors may consider adding empirical results from the experiments to support the theory. For those reasons, I think the paper is not ready to be accepted for now.
confidence: Certain
Decision: Accept