Language Models are Few-Shot Learners
OpenAI
NeurIPS 2020
Presenter: Shu Zhao
smz5505@psu.edu

Outline• Motivation• Related Work
• Method• Experiments• Limitations• Conclusion• Feature Work

Motivation• Remove the requirement oftask-specific datasets and taskspecific fine-tuning when
applying large language models
to downstream tasks

Why?

Motivation

Why?
• Expensive data collection and fine-tuning on each task
• Some works pointed out pre-training+fine-tuning paradigm may
have poor generalization ability• Simulate humans that don’t require large supervised datasetsHendrycks, Dan, et al. Pretrained transformers improve out-of-distribution robustness. ACL 2020.
Yogatama, Dani, et al. Learning and Evaluating General Linguistic Intelligence. 2019.
McCoy, Tom, et al. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. ACL 2019.

Related Work• GPT-1 (117M)• Unsupervised Generative Pre-Training + Supervised Fine-Tuning• Transformer Decoder Architecture• BooksCorpus + Word Benchmark for unsupervised pre-training, 5GB

Radford, Alec, et al. Improving Language Understanding by Generative Pre-Training. 2018.

Related Work• GPT-2 (1.5B)• Unsupervised Generative Pre-Training• GPT-1 + Modified layer normalization and initialization• WebText, 40GB

Image from https://www.lesswrong.com/posts/qxvihKpFMuc4tvuf4/recall-and-regurgitation-in-gpt2
Radford, Alec, et al. Language Models are Unsupervised Multitask Learners. 2019.

Method• Parameters
175B v.s. 1.5B (GPT-2)• Architecture
Sparse self-attention layer described in the Sparse Transformer
O(𝑁 ! ) -> O(N 𝑁)

Child, Rewon, et al. Generating Long Sequences with Sparse Transformers. 2019.

Method• The bigger, the better

Method• Dataset

Common Crawl (filtered) + WebText2 + Books1 + Books2 + Wikipedia (45TB)<- higher-quality are sampled more frequently

Experiments• Evaluation Settings• Fine-tuning• Few-shot• One-shot• Zero-shot• Tasks• Completion• Question Answering• Translation• Common Sense Reasoning• Reading Comprehension• Natural Language Inference• …

Experiments• Language Modeling, Cloze, and Completion Tasks• LAMBADA: Predict the last word of sentences• HellaSwag: Pick the best ending to a story or set of instructions• StoryCloze: Select the correct ending sentence for five-sentence long stories

Lack reasoning ability

Paperno, Denis, et al. The lambada dataset: Word prediction requiring a broad discourse context. 2016.
Zellers, Rowan, et al. Hellaswag: Can a machine really finish your sentence?. 2019.
Mostafazadeh, Nasrin, et al. A corpus and evaluation framework for deeper understanding of commonsense stories. 2016.

Experiments• Closed Book Question Answering• Compare to Open-Book: Answer the questions without conditioning on
auxilliary information• NaturalQS: Read and comprehend an entire Wikipedia article that may or may
not contain the answer to the question• WebQS: Answer questions based on Freebase• TriviaQA: Contain Question-answer-evidence triples and require more crosssentence reasoning to find answers

Kwiatkowski, Tom, et al. Natural questions: a benchmark for question answering research. TACL 2019.
Berant, Jonathan, et al. Semantic parsing on freebase from question-answer pairs. EMNLP 2013.
Joshi, Mandar, et al. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. 2017.

Experiments• Model capacity translates directly to more ‘knowledge’ absorbed in
the parameters of the modelWhy does GPT-3 have good generalization performance?

<- inner loop<- outer loop

Finn, Chelsea, et al. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. ICML 2017.

Limitations• Weaknesses in Text Synthesis• Repeat• Lose coherence• Contradict• Difficulty in Commonsense• Weight every token equally• Poor sample efficiency during pre-training• Lack of explainability• Closed source

West, Peter, et al. Symbolic Knowledge Distillation: from General Language Models to Commonsense Models. NAACL 2022.
Wei, Jason, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. 2022.

Conclusion• 175 billion parameter language model which shows strong
performance on many NLP tasks and benchmarks in the zero-shot,
one-shot, and few-shot settings• Still have some limitations and weaknesses to overcome• More data, larger model, more ability

Future Work• Inject knowledge into vision from LLM

Koh, Jing, et al. Grounding Language Models to Images for Multimodal Generation. 2023.
Hao, Liu, et al. Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment. 2023.

Thanks
Shu Zhaosmz5505@psu.edu