Language Models: Unsupervised
Multitask Learners
Presented by: Mahsa Sheikhi Karizaki

Problem Motivation
Natural language processing tasks, such as question answering, machinetranslation, reading comprehension, and summarization, are typically approachedwith supervised learning on task-specific datasets. This has served well to makeprogress on narrow experts. But the often erratic behavior of captioning models,reading comprehension systems, and image classifiers on the diversity and varietyof possible inputs highlights some of the short-comings of this approach.

Objective•The authors demonstrate that language models begin to learn tasks without any
explicit supervision when trained on a new dataset of millions of webpages calledWebText.

•They demonstrate language models can perform down-stream tasks in a zero-shotsetting without any parameter or architecture modification. This approach shows
potential by highlighting the ability of language models to perform a wide range oftasks in a zero-shot setting.

Related WorkThe dominant approach to creating ML systems is to collect a dataset of training
examples demonstrating correct behavior for a desired task, train a system toimitate these behaviors, and then test its performance on independent andidentically distributed (IID) held-out examples.
•Multitask learning is a promising framework for improving general performance.
Recent work reports modest performance improvements and the two mostambitious efforts to date have trained on a total of 10 and 17 (dataset,objective) pairs respectively.

•The current best performing systems on language tasks utilize a combination of
pre-training and supervised fine-tuning.

•another line of work has demonstrated the promise of language models toperform specific tasks, such as commonsense reasoning and sentiment analysis

Dataset
Most prior work trained language models on a single domain of text, such as news
articles or Wikipedia. This approach motivates building as large and diverse a dataset
as possible in order to collect natural language demonstrations of tasks in as varied ofdomains and contexts as possible.
Instead, they created a new web scrape which emphasizes document quality. To dothis they only scraped web pages which have been curated/filtered by humans.

Dataset

Table 1. Examples of naturally occurring demonstrations of English to French and French to
English translation found throughout the WebText training set

ModelA Transformer based architecture is used for the LMs. The model largely follows thedetails of the OpenAI GPT model with a few modifications. The authors trained andbenchmarked four LMs.

Table 2. Architecture hyperparameters for the 4 model sizesPerformance

Figure 1. Zero-shot task performance of WebText LMs as a function of model size on many NLP tasks.
Reading Comprehension results are on CoQA, translation on WMT-14 Fr-En, summarization on CNN
and Daily Mail, and Question Answering on Natural Questions.

Results
As an initial step towards zero-shot task transfer, we are interested in understanding how
WebText LM’s perform at zero-shot domain transfer on the primary task they are trainedfor: “language modeling”.

Table 3. Zero-shot results on many datasets. No training or fine-tuning was performed for any of theseresults.

Children’s Book Test

Figure 2. Performance on the Children’s Book Test as a function of model capacity.

Winograd Schema Challenge

Figure 3. Performance on the Winograd Schema Challenge as a function of model capacity.

Question Answering

Table 4. The 30 most confident answers generated by GPT-2 on the development set of Natural
Questions sorted by their probability according to GPT-2.

Results on Other Tasks
On reading comprehension, the performance of GPT-2 is competitive with supervised
baselines in a zero-shot setting. However, on other tasks such as summarization, while itis qualitatively performing the task, its performance is still only rudimentary according toquantitative metrics.

Generalization vs Memorization
Recent work in computer vision has shown that common image datasets contain a nontrivial amount of near-duplicate images.

Table 5. Percentage of test set 8 grams overlapping with training sets.

Thank you.

Contact Information:
mfs6614@psu.edu